Historical dataset showing Botswana literacy rate by year from 1991 to 2013.

Abstract

In 1991 the International Institute for Educational Planning (IIEP) and a number of Ministries of Education in Southern and Eastern Africa began to work together in order to address training and research needs in Education. The focus for this work was on establishing long-term strategies for building the capacity of educational planners to monitor and evaluate the quality of their basic education systems. The first two educational policy research projects undertaken by SACMEQ (widely known as "SACMEQ I" and "SACMEQ II") were designed to provide detailed information that could be used to guide planning decisions aimed at improving the quality of education in primary school systems.

During 1995-1998 seven Ministries of Education participated in the SACMEQ I Project. The SACMEQ II Project commenced in 1998 and the surveys of schools, involving 14 Ministries of Education, took place between 2000 and 2004. The survey was undertaken in schools in Botswana, Kenya, Lesotho, Malawi, Mauritius, Mozambique, Namibia, Seychelles, South Africa, Swaziland, Tanzania, Uganda, Zambia and Zanzibar.

Moving from the SACMEQ I Project (covering around 1100 schools and 20,000 pupils) to the SACMEQ II Project (covering around 2500 schools and 45,000 pupils) resulted in a major increase in the scale and complexity of SACMEQ's research and training programmes.

SACMEQ's mission is to: a) Expand opportunities for educational planners to gain the technical skills required to monitor and evaluate the quality of their education systems; and b) Generate information that can be used by decision-makers to plan and improve the quality of education.

Geographic coverage

National coverage

Analysis unit

• Pupils
• Teachers
• Schools

Universe

The target population for SACMEQ's Initial Project was defined as "all pupils at the Grade 6 level in 1995 who were attending registered government or non-government schools". Grade 6 was chosen because it was the grade level where the basics of reading literacy were expected to have been acquired.

Kind of data

Sample survey data [ssd]

Sampling procedure

Sampling Sample designs in the field of education are usually prepared amid a network of competing constraints. These designs need to adhere to established survey sampling theory and, at the same time, give due recognition to the financial, administrative, and socio-political settings in which they are to be applied. The "best" sample design for a particular project is one that provides levels of sampling accuracy that are acceptable in terms of the main aims of the project, while simultaneously limiting cost, logistic, and procedural demands to manageable levels. The major constraints that were established prior to the preparation of the sample designs for the SACMEQ II Project have been listed below.

Target Population: The target population definitions should focus on Grade 6 pupils attending registered mainstream government or non-government schools. In addition, the defined target population should be constructed by excluding no more than 5 percent of pupils from the desired target population.

Bias Control: The sampling should conform to the accepted rules of scientific probability sampling. That is, the members of the defined target population should have a known and non-zero probability of selection into the sample so that any potential for bias in sample estimates due to variations from "epsem sampling" (equal probability of selection method) may be addressed through the use of appropriate sampling weights.

Sampling Errors: The sample estimates for the main criterion variables should conform to the sampling accuracy requirements set down by the International Association for the Evaluation of Educational Achievement. That is, the standard error of sampling for the pupil tests should be of a magnitude that is equal to, or smaller than, what would be achieved by employing a simple random sample of 400 pupils.

Response Rates: Each SACMEQ country should aim to achieve an overall response rate for pupils of 80 percent. This figure was based on the wish to achieve or exceed a response rate of 90 percent for schools and a response rate of 90 percent for pupils within schools.

Administrative and Financial Costs: The number of schools selected in each country should recognize limitations in the administrative and financial resources available for data collection.

Other Constraints: The number of pupils selected to participate in the data collection in each selected school should be set at a level that will maximize validity of the within-school data collection for the pupil reading and mathematics tests.

The Specification of the Target Population The target population for both the SACMEQ I and SACMEQ II Projects was focussed on the Grade 6 level for three main reasons.

First, Grade 6 identified a point near the end of primary schooling where school participation rates were reasonably high for most of the seven countries that participated in the SACMEQ I data collection during 1995-1997, and also reasonably high for most of the fourteen countries that participated in the SACMEQ II collection during 2000-2002. For this reason, Grade 6 represented a point that was suitable for making an assessment of the contribution of primary schooling towards the literacy and numeracy levels of a broad cross-section of society.

Second, the NRCs considered that testing pupils at grade levels lower than Grade 6 was problematic - because in some SACMEQ countries the lower grades were too close to the transition point between the use of local and national languages by teachers in the classroom. This transition point generally occurred at around Grade 3 level - but in some rural areas of some countries it was thought to be as high as Grade 4 level.

Third, the NRCs were of the opinion that the collection of home background information from pupils at grade levels lower than Grade 6 was likely to lack validity for certain key "explanatory" variables. For example, the NRCs felt that children at lower grade levels did not know how many years of education that their parents had received, and they also had difficulty in accurately describing the socioeconomic environment of their own homes (for example, the number of books at home).

The Stratification Procedures The stratification procedures adopted for the study employed explicit and implicit strata. The explicit stratification variable, "Region", was applied by separating each sampling frame into separate regional lists of schools prior to undertaking the sampling. The implicit stratification variable was "School Size" - as measured by the number of Grade 6 pupils.

The main reason for choosing Region as the explicit stratification variable was that the SACMEQ Ministries of Education wanted to have education administration regions as "domains" for the study. That is, the Ministries wanted to have reasonably accurate sample estimates of population characteristics for each region.

There were two other reasons for selecting Region as the main stratification variable. First, this was expected to provide an increment in sampling precision due to known between-region differences in the educational achievement of pupils - especially between predominantly urban and predominantly rural regions. Second, this approach provided a broad geographical coverage for the sample - which was necessary in order to spread the fieldwork across each country in a manner that prevented the occurrence of excessive administrative demands in particular regions.

The use of School Size as an implicit stratification variable within regions also offered increased sampling precision because it provided a way of sorting the schools from "mostly rural" (small schools) to "mostly urban" (large schools). It was known that this kind of sorting was linked to the main criterion variables for the study - with urban schools likely to have higher resource levels and better pupil achievement scores than rural schools.

Sample Design Framework The SACMEQ II sample designs were prepared by using a specialized software system (SAMDEM) that enabled the high-speed generation of a range of sampling options which satisfied the statistical accuracy constraints set down for the project, and at the same time also addressed the logistical and financial realities of each country.

Note: Details of sampling design procedures are presented in the "Mauritius Working Report".

Mode of data collection

Face-to-face [f2f]

Research instrument

The data collection for SACMEQ’s Initial Project took place in October 1995 and involved the administration of questionnaires to pupils, teachers, and school heads. The pupil questionnaire contained questions about the pupils’ home backgrounds and their school life; the teacher questionnaire asked about classrooms, teaching practices, working conditions, and teacher housing; and the school head questionnaire collected information about teachers, enrolments, buildings, facilities, and management. A reading literacy test was also given to the pupils. The test was based on items that were selected after a trial-testing programme had been completed.

Cleaning operations

Data Checking and Data Entry Data preparation commenced soon after the main data collection was completed. The NRCs had to organize the safe return of all materials to the Ministry of Education where the data collection instruments could be checked, entered into computers, and then "cleaned" to remove errors prior to data analysis. The data-checking involved the "hand editing" of data collection instruments by a team of trained staff. They were required to check

Abstract

In 1991 the International Institute for Educational Planning (IIEP) and a number of Ministries of Education in Southern and Eastern Africa began to work together in order to address training and research needs in Education. The focus for this work was on establishing long-term strategies for building the capacity of educational planners to monitor and evaluate the quality of their basic education systems. The first two educational policy research projects undertaken by SACMEQ (widely known as "SACMEQ I" and "SACMEQ II") were designed to provide detailed information that could be used to guide planning decisions aimed at improving the quality of education in primary school systems.

During 1995-1998 seven Ministries of Education participated in the SACMEQ I Project. The SACMEQ II Project commenced in 1998 and the surveys of schools, involving 14 Ministries of Education, took place between 2000 and 2002. The survey was undertaken in schools in Botswana, Kenya, Lesotho, Malawi, Mauritius, Mozambique, Namibia, Seychelles, South Africa, Swaziland, Tanzania, Uganda, Zambia and Zanzibar.

Moving from the SACMEQ I Project (covering around 1100 schools and 20,000 pupils) to the SACMEQ II Project (covering around 2500 schools and 45,000 pupils) resulted in a major increase in the scale and complexity of SACMEQ's research and training programmes.

Geographic coverage

The surveys had national coverage of the countries participating in the project, including Botswana, Kenya, Lesotho, Malawi, Mauritius, Mozambique, Namibia, Seychelles, South Africa, Swaziland, Tanzania (including Zanzibar), Uganda, Zambia.

Analysis unit

Units of analysis in the survey included schools and individuals

Universe

The target population for SACMEQ's Initial Project was defined as "all pupils at the Grade 6 level in 1995 who were attending registered government or non-government schools". Grade 6 was chosen because it was the grade level where the basics of reading literacy were expected to have been acquired.

Kind of data

Sample survey data [ssd]

Sampling procedure

A stratified two-stage sample design was used to select around 150 schools in each country. Pupils were then selected within these schools by drawing simple random samples.

Mode of data collection

Face-to-face [f2f]

Research instrument

The data collection for SACMEQ’s Initial Project took place in October 1995 and involved the administration of questionnaires to pupils, teachers, and school heads. The pupil questionnaire contained questions about the pupils’ home backgrounds and their school life; the teacher questionnaire asked about classrooms, teaching practices, working conditions, and teacher housing; and the school head questionnaire collected information about teachers, enrolments, buildings, facilities, and management. A reading literacy test was also given to the pupils. The test was based on items that were selected after a trial-testing programme had been completed.

Abstract

In 1991 the International Institute for Educational Planning (IIEP) and a number of Ministries of Education in Southern and Eastern Africa began to work together in order to address training and research needs in Education. The focus for this work was on establishing long-term strategies for building the capacity of educational planners to monitor and evaluate the quality of their basic education systems. The first two educational policy research projects undertaken by SACMEQ (widely known as "SACMEQ I" and "SACMEQ II") were designed to provide detailed information that could be used to guide planning decisions aimed at improving the quality of education in primary school systems.

During 1995-1998 seven Ministries of Education participated in the SACMEQ I Project. The SACMEQ II Project commenced in 1998 and the surveys of schools, involving 14 Ministries of Education, took place between 2000 and 2004. The survey was undertaken in schools in Botswana, Kenya, Lesotho, Malawi, Mauritius, Mozambique, Namibia, Seychelles, South Africa, Swaziland, Tanzania, Uganda, Zambia and Zanzibar.

Moving from the SACMEQ I Project (covering around 1100 schools and 20,000 pupils) to the SACMEQ II Project (covering around 2500 schools and 45,000 pupils) resulted in a major increase in the scale and complexity of SACMEQ's research and training programmes.

SACMEQ's mission is to: a) Expand opportunities for educational planners to gain the technical skills required to monitor and evaluate the quality of their education systems; and b) Generate information that can be used by decision-makers to plan and improve the quality of education.

Geographic coverage

National coverage

Analysis unit

• Pupils
• Teachers
• Schools

Universe

The target population for SACMEQ's Initial Project was defined as "all pupils at the Grade 6 level in 1995 who were attending registered government or non-government schools". Grade 6 was chosen because it was the grade level where the basics of reading literacy were expected to have been acquired.

Kind of data

Sample survey data [ssd]

Sampling procedure

Sampling Sample designs in the field of education are usually prepared amid a network of competing constraints. These designs need to adhere to established survey sampling theory and, at the same time, give due recognition to the financial, administrative, and socio-political settings in which they are to be applied. The "best" sample design for a particular project is one that provides levels of sampling accuracy that are acceptable in terms of the main aims of the project, while simultaneously limiting cost, logistic, and procedural demands to manageable levels. The major constraints that were established prior to the preparation of the sample designs for the SACMEQ II Project have been listed below.

Target Population: The target population definitions should focus on Grade 6 pupils attending registered mainstream government or non-government schools. In addition, the defined target population should be constructed by excluding no more than 5 percent of pupils from the desired target population.

Bias Control: The sampling should conform to the accepted rules of scientific probability sampling. That is, the members of the defined target population should have a known and non-zero probability of selection into the sample so that any potential for bias in sample estimates due to variations from "epsem sampling" (equal probability of selection method) may be addressed through the use of appropriate sampling weights.

Sampling Errors: The sample estimates for the main criterion variables should conform to the sampling accuracy requirements that the standard error of sampling for the pupil tests should be of a magnitude that is equal to, or smaller than, what would be achieved by employing a simple random sample of 400 pupils.

Response Rates: Each SACMEQ country should aim to achieve an overall response rate for pupils of 80 percent. This figure was based on the wish to achieve or exceed a response rate of 90 percent for schools and a response rate of 90 percent for pupils within schools.

Administrative and Financial Costs: The number of schools selected in each country should recognize limitations in the administrative and financial resources available for data collection.

Other Constraints: The number of pupils selected to participate in the data collection in each selected school should be set at a level that will maximize validity of the within-school data collection for the pupil reading and mathematics tests.

The Specification of the Target Population For Lesotho, the desired target population was all pupils enrolled in Standard 6 in the ninth month of the school year (i.e., in September 2000). It was however, decided to exclude certain pupils. These were pupils in schools having fewer than 15 standard 6 pupils in them, pupils in 'inaccessible schools, and pupils in special schools.

The number of schools required in the sample is in part a function of the intra-class correlation (rho) which is an indicator of the proportion of variation (in achievement in this case) among schools of total variation. The following is the formula often used for estimating the value of rho in situations where two-stage cluster sampling is employed using (approximately) equal sized clusters).

estimated rho = (b. s(a)square - (s)square) / (b - 1)(s)square

where s(a)square is the variance of cluster means, (s)square is the variance of the element values, and b is the cluster size. The rho had been estimated at 0.30 in Lesotho; meaning that 30 percent of the variation was among schools while 70 percent of the variation was within schools.

The sample was stratified into districts.

Note: Details of sampling design procedures are presented in the "Lesotho Working Report".

Mode of data collection

Face-to-face [f2f]

Research instrument

The data collection for SACMEQ’s Initial Project took place in October 1995 and involved the administration of questionnaires to pupils, teachers, and school heads. The pupil questionnaire contained questions about the pupils’ home backgrounds and their school life; the teacher questionnaire asked about classrooms, teaching practices, working conditions, and teacher housing; and the school head questionnaire collected information about teachers, enrolments, buildings, facilities, and management. A reading literacy test was also given to the pupils. The test was based on items that were selected after a trial-testing programme had been completed.

Cleaning operations

Data entry and data cleaning were undertaken in the Ministry of Education using in-house facilities and the clerical staff of the Education Statistics Unit. The NRC and the Deputy NRC led and supervised this activity. The Deputy NRC's special skills in computer data processing made this activity easily manageable. As a result, Lesotho was one of the countries that submitted the well cleaned up data files to the unit of 'Monitoring Educational Quality' at the IIEP in Paris, relatively early, at the beginning of 2001. The IIEP team further worked on the data to perform consistency checks and other validity checks to ensure accuracy of the research findings.

Response rate

Response rates for pupils and schools respectively were 88 percent and 98 percent. The reason for the shortfall in pupil numbers was absenteeism by some learners in some of the schools on the day of data collection including the pupils in the schools that proved difficult to reach during the data collection exercise.

Sampling error estimates

The sample designs employed in the SACMEQ Projects departed markedly from the usual "textbook model" of simple random sampling. This departure demanded that special steps be taken in order to calculate "sampling errors" (that is, measures of the stability of sample estimates of population characteristics).

Abstract

In 1991 the International Institute for Educational Planning (IIEP) and a number of Ministries of Education in Southern and Eastern Africa began to work together in order to address training and research needs in Education. The focus for this work was on establishing long-term strategies for building the capacity of educational planners to monitor and evaluate the quality of their basic education systems. The first two educational policy research projects undertaken by SACMEQ (widely known as "SACMEQ I" and "SACMEQ II") were designed to provide detailed information that could be used to guide planning decisions aimed at improving the quality of education in primary school systems.

During 1995-1998 seven Ministries of Education participated in the SACMEQ I Project. The SACMEQ II Project commenced in 1998 and the surveys of schools, involving 14 Ministries of Education, took place between 2000 and 2004. The survey was undertaken in schools in Botswana, Kenya, Lesotho, Malawi, Mauritius, Mozambique, Namibia, Seychelles, South Africa, Swaziland, Tanzania, Uganda, Zambia and Zanzibar.

Moving from the SACMEQ I Project (covering around 1100 schools and 20,000 pupils) to the SACMEQ II Project (covering around 2500 schools and 45,000 pupils) resulted in a major increase in the scale and complexity of SACMEQ's research and training programmes.

SACMEQ's mission is to: a) Expand opportunities for educational planners to gain the technical skills required to monitor and evaluate the quality of their education systems; and b) Generate information that can be used by decision-makers to plan and improve the quality of education.

Geographic coverage

National coverage

Analysis unit

• Pupils
• Teachers
• Schools

Universe

The target population for SACMEQ's Initial Project was defined as "all pupils at the Grade 6 level in 1995 who were attending registered government or non-government schools". Grade 6 was chosen because it was the grade level where the basics of reading literacy were expected to have been acquired.

Kind of data

Sample survey data [ssd]

Sampling procedure

Two-stage sampling procedures were used. The first stage of sampling consists of the PPS selection of schools followed by the selection of a simple random sample of pupils in selected schools.

Note: Details of sampling design procedures are presented in the "Seychelles Working Report".

Mode of data collection

Face-to-face [f2f]

Research instrument

The data collection for SACMEQ’s Initial Project took place in October 1995 and involved the administration of questionnaires to pupils, teachers, and school heads. The pupil questionnaire contained questions about the pupils’ home backgrounds and their school life; the teacher questionnaire asked about classrooms, teaching practices, working conditions, and teacher housing; and the school head questionnaire collected information about teachers, enrolments, buildings, facilities, and management. A reading literacy test was also given to the pupils. The test was based on items that were selected after a trial-testing programme had been completed.

Cleaning operations

Data Checking and Data Entry Data preparation commenced soon after the main data collection was completed. The NRCs had to organize the safe return of all materials to the Ministry of Education where the data collection instruments could be checked, entered into computers, and then "cleaned" to remove errors prior to data analysis. The data-checking involved the "hand editing" of data collection instruments by a team of trained staff. They were required to check that: (i) all questionnaires, tests, and forms had arrived back from the sample schools, (ii) the identification numbers on all instruments were complete and accurate, and (iii) certain logical linkages between questions made sense (for example, the two questions to school heads concerning "Do you have a school library?" and "How many books do you have in your school library?").

The next step was the entry of data into computers using the WINDEM software. A team of 5-10 staff normally undertook this work. In some cases the data were "double entered" in order to monitor accuracy.

The numbers of keystrokes required to enter one copy of each data collection instrument were as follows: pupil questionnaire: 150; pupil reading test: 85; pupil mathematics test: 65; teacher questionnaire: 587; teacher reading test: 51; teacher mathematics test: 43; school head questionnaire: 319; school form: 58; and pupil name form: 51.

Data Cleaning The NRCs received written instructions and follow-up support from IIEP staff in the basic steps of data cleaning using the WINDEM software. This permitted the NRCs to (i) identify major errors in the sequence of identification numbers, (ii) cross-check identification numbers across files (for example, to ensure that all pupils were linked with their own reading and mathematics teachers), (iii) ensure that all schools listed on the original sampling frame also had valid data collection instruments and vice-versa, (iv) check for "wild codes" that occurred when some variables had values that fell outside pre-specified reasonable limits, and (v) validate that variables used as linkage devices in later file merges were available and accurate.

A second phase of data preparation directed efforts towards the identification and correction of "wild codes" (which refer to data values that that fall outside credible limits), and "inconsistencies" (which refer to different responses to the same, or related, questions). There were also some errors in the identification codes for teachers that needed to be corrected before data could be merged.

During 2002 a supplementary training programme was prepared and delivered to all countries via the Internet. This training led each SACMEQ Research Team step-by-step through the required data cleaning procedures - with the NRCs supervising "hands-on" data cleaning activities and IIEP staff occasionally using advanced software systems to validate the quality of the work involved in each data-cleaning step.

This resulted in a "cyclical" process whereby data files were cleaned by the NRC and then emailed to the IIEP for checking and then emailed back to the NRC for further cleaning.

The number of cycles required to complete all of the data cleaning was 9 cycles and it took 9 months to complete all the data cleaning.

Response rate

Response rates for pupils and schools respectively were %96 and %100.

Sampling error estimates

The sample designs employed in the SACMEQ Projects departed markedly from the usual "textbook model" of simple random sampling. This departure demanded that special steps be taken in order to calculate "sampling errors" (that is, measures of the stability of sample estimates of population characteristics).

In the report (Seychelles Working Report) a brief overview of various aspects of the general concept of "sampling error" has been presented. This has included a discussion of notions of "design effect", "the effective sample size", and the "Jackknife procedure" for estimating sampling errors.

Abstract

In 1991 the International Institute for Educational Planning (IIEP) and a number of Ministries of Education in Southern and Eastern Africa began to work together in order to address training and research needs in Education. The focus for this work was on establishing long-term strategies for building the capacity of educational planners to monitor and evaluate the quality of their basic education systems. The first two educational policy research projects undertaken by SACMEQ (widely known as "SACMEQ I" and "SACMEQ II") were designed to provide detailed information that could be used to guide planning decisions aimed at improving the quality of education in primary school systems.

During 1995-1998 seven Ministries of Education participated in the SACMEQ I Project. The SACMEQ II Project commenced in 1998 and the surveys of schools, involving 14 Ministries of Education, took place between 2000 and 2004. The survey was undertaken in schools in Botswana, Kenya, Lesotho, Malawi, Mauritius, Mozambique, Namibia, Seychelles, South Africa, Swaziland, Tanzania, Uganda, Zambia and Zanzibar.

Moving from the SACMEQ I Project (covering around 1100 schools and 20,000 pupils) to the SACMEQ II Project (covering around 2500 schools and 45,000 pupils) resulted in a major increase in the scale and complexity of SACMEQ's research and training programmes.

SACMEQ's mission is to: a) Expand opportunities for educational planners to gain the technical skills required to monitor and evaluate the quality of their education systems; and b) Generate information that can be used by decision-makers to plan and improve the quality of education.

Geographic coverage

National coverage

Analysis unit

• Pupils
• Teachers
• Schools

Universe

The target population for SACMEQ's Initial Project was defined as "all pupils at the Grade 6 level in 1995 who were attending registered government or non-government schools". Grade 6 was chosen because it was the grade level where the basics of reading literacy were expected to have been acquired.

Kind of data

Sample survey data [ssd]

Sampling procedure

The sample designs used in the SACMEQ II Project were selected so as to meet the standards set down by the International Association for the Evaluation of Educational Achievement. These standards required that sample estimates of important pupil population parameters should have sampling accuracy that was at least equivalent to a simple random sample of 400 pupils (thereby guaranteeing 95 percent confidence limits for sample means of plus or minus one tenth of a pupil standard deviation unit).

Some Constraints on Sample Design Sample designs in the field of education are usually prepared amid a network of competing constraints. These designs need to adhere to established survey sampling theory and, at the same time, give due recognition to the financial, administrative, and socio-political settings in which they are to be applied. The "best" sample design for a particular project is one that provides levels of sampling accuracy that are acceptable in terms of the main aims of the project, while simultaneously limiting cost, logistic, and procedural demands to manageable levels. The major constraints that were established prior to the preparation of the sample designs for the SACMEQ II Project have been listed below.

Target Population: The target population definitions should focus on Grade 6 pupils attending registered mainstream government or non-government schools. In addition, the defined target population should be constructed by excluding no more than 5 percent of pupils from the desired target population.

Bias Control: The sampling should conform to the accepted rules of scientific probability sampling. That is, the members of the defined target population should have a known and non-zero probability of selection into the sample so that any potential for bias in sample estimates due to variations from "epsem sampling" (equal probability of selection method) may be addressed through the use of appropriate sampling weights (Kish, 1965).

Sampling Errors: The sample estimates for the main criterion variables should conform to the sampling accuracy requirements set down by the International Association for the Evaluation of Educational Achievement (Ross, 1991). That is, the standard error of sampling for the pupil tests should be of a magnitude that is equal to, or smaller than, what would be achieved by employing a simple random sample of 400 pupils (Ross, 1985).

Response Rates: Each SACMEQ country should aim to achieve an overall response rate for pupils of 80 percent. This figure was based on the wish to achieve or exceed a response rate of 90 percent for schools and a response rate of 90 percent for pupils within schools.

Administrative and Financial Costs: The number of schools selected in each country should recognize limitations in the administrative and financial resources available for data collection.

Other Constraints: The number of pupils selected to participate in the data collection in each selected school should be set at a level that will maximize validity of the within-school data collection for the pupil reading and mathematics tests.

Note: Detailed descriptions of the sample design, sample selection, and sample evaluation procedures have been presented in the "Swaziland Working Report".

Mode of data collection

Face-to-face [f2f]

Research instrument

The data collection for SACMEQ’s Initial Project took place in October 1995 and involved the administration of questionnaires to pupils, teachers, and school heads. The pupil questionnaire contained questions about the pupils’ home backgrounds and their school life; the teacher questionnaire asked about classrooms, teaching practices, working conditions, and teacher housing; and the school head questionnaire collected information about teachers, enrolments, buildings, facilities, and management. A reading literacy test was also given to the pupils. The test was based on items that were selected after a trial-testing programme had been completed.

Cleaning operations

Data Checking and Data Entry Data preparation commenced soon after the main data collection was completed. The NRCs had to organize the safe return of all materials to the Ministry of Education where the data collection instruments could be checked, entered into computers, and then "cleaned" to remove errors prior to data analysis. The data-checking involved the "hand editing" of data collection instruments by a team of trained staff. They were required to check that: (i) all questionnaires, tests, and forms had arrived back from the sample schools, (ii) the identification numbers on all instruments were complete and accurate, and (iii) certain logical linkages between questions made sense (for example, the two questions to school heads concerning "Do you have a school library?" and "How many books do you have in your school library?").

Data Cleaning The NRCs received written instructions and follow-up support from IIEP staff in the basic steps of data cleaning using the WINDEM software. This permitted the NRCs to (i) identify major errors in the sequence of identification numbers, (ii) cross-check identification numbers across files (for example, to ensure that all pupils were linked with their own reading and mathematics teachers), (iii) ensure that all schools listed on the original sampling frame also had valid data collection instruments and vice-versa, (iv) check for "wild codes" that occurred when some variables had values that fell outside pre-specified reasonable limits, and (v) validate that variables used as linkage devices in later file merges were available and accurate.

A second phase of data preparation directed efforts towards the identification and correction of "wild codes" (which refer to data values that that fall outside credible limits), and "inconsistencies" (which refer to different responses to the same, or related, questions). There were also some errors in the identification codes for teachers that needed to be corrected before data could be merged.

During 2002 a supplementary training programme was prepared and delivered to all countries via the Internet. This training led each SACMEQ Research Team step-by-step through the required data cleaning procedures - with the NRCs supervising "hands-on" data cleaning activities and IIEP staff occasionally using advanced software systems to validate the quality of the work involved in each data-cleaning step.

This resulted in a "cyclical" process whereby data files were cleaned by the NRC and then emailed to the IIEP for checking and then emailed back to the NRC for further cleaning.

Response rate

Response rates for pupils and schools respectively were %92 and %99.

Sampling error estimates

The sample designs employed in the SACMEQ Projects departed markedly from the usual "textbook model" of simple random sampling. This departure demanded that special steps be taken in order to calculate "sampling errors" (that is, measures of the stability of sample estimates of population characteristics).

In the report (Swaziland Working Report) a brief overview of various aspects of the general concept of "sampling error" has been presented. This has included a discussion of notions of "design effect", "the effective sample size", and the "Jackknife procedure" for estimating sampling errors.