This performance evaluation uses a pre-post methodology to evaluate the functional literacy subactivity of the Artisan and Fez Medina Project. The evaluation followed the Kirkpatrick model trying to assemble a set of qualitative and quantitative evidence. To do so the evaluation team questioned 6 main criteria : relevance, coherence, efficiency, effectiveness, impact and sustainability and also assessed the program gender and environment approaches. 17 focus groups, 500 respondents to a survey and 70 semi-structured interviews contributed to collect primary and secondary sources of information so to triangulate findings.

Initial evaluation by TACC: This performance evaluation uses a pre-post methodology to evaluate the functional literacy subactivity of the Artisan and Fez Medina Project. The evaluation followed the Kirkpatrick model trying to assemble a set of qualitative and quantitative evidence. To do so the evaluation team questioned 6 main criteria : relevance, coherence, efficiency, effectiveness, impact and sustainability and also assessed the program gender and environment approaches. 17 focus groups, 500 respondents to a survey and 70 semi-structured interviews contributed to collect primary and secondary sources of information so to triangulate findings.

Follow-on evaluation by ISG: This performance evaluation uses a pre-post methodology to evaluate the functional literacy subactivity of the Artisan and Fez Medina Project. The evaluation followed the Kirkpatrick model trying to assemble a set of qualitative and quantitative evidence. To do so the evaluation team questioned 6 main criteria : relevance, coherence, efficiency, effectiveness, impact and sustainability and also assessed the program gender and environment approaches. 17 focus groups, 500 respondents to a survey and 70 semi-structured interviews contributed to collect primary and secondary sources of information so to triangulate findings

*** While the metadata for this evaluation lists "Pre-Post" as the methodology, only the evaluation by TACC used this methodology. The evaluation by ISG used an Ex-Post methodology

description: This performance evaluation uses a pre-post methodology to evaluate the functional literacy subactivity of the Artisan and Fez Medina Project. The evaluation followed the Kirkpatrick model trying to assemble a set of qualitative and quantitative evidence. To do so the evaluation team questioned 6 main criteria : relevance, coherence, efficiency, effectiveness, impact and sustainability and also assessed the program gender and environment approaches. 17 focus groups, 500 respondents to a survey and 70 semi-structured interviews contributed to collect primary and secondary sources of information so to triangulate findings.; abstract: This performance evaluation uses a pre-post methodology to evaluate the functional literacy subactivity of the Artisan and Fez Medina Project. The evaluation followed the Kirkpatrick model trying to assemble a set of qualitative and quantitative evidence. To do so the evaluation team questioned 6 main criteria : relevance, coherence, efficiency, effectiveness, impact and sustainability and also assessed the program gender and environment approaches. 17 focus groups, 500 respondents to a survey and 70 semi-structured interviews contributed to collect primary and secondary sources of information so to triangulate findings.

Abstract

This performance evaluation uses a pre-post methodology to evaluate the functional literacy subactivity of the Artisan and Fez Medina Project. The evaluation followed the Kirkpatrick model trying to assemble a set of qualitative and quantitative evidence. To do so the evaluation team questioned 6 main criteria : relevance, coherence, efficiency, effectiveness, impact and sustainability and also assessed the program gender and environment approaches. 17 focus groups, 500 respondents to a survey and 70 semi-structured interviews contributed to collect primary and secondary sources of information so to triangulate findings.

Geographic coverage

Marrakech, Souss Massa Draa, Tanger-Tétouan, and Fez Boulemane

Analysis unit

Individuals

Universe

There were 70 semi-structured interviews, 17 focus groups, and a survey of 500 beneficiaries in the four intervention regions. The objective was to include the maximum number of direct beneficiaries in each of the three data collection events. The overall size of the sample was kept to 1,297.

Kind of data

Sample survey data [ssd]

Sampling procedure

In order to meet the objectives and guarantee the quality of the survey results, the Consultant opted for stratification of the target populations to form homogeneous groups by variable, correlated with the variables of interest. Effective stratification not only yields better representativeness of the population as a whole but increases the accuracy of the results for homogeneous groups. Notwithstanding, this method will not consist of a uniform comparison of the stratification variables, but instead, differ by level, due to the small size of the sample to be surveyed. The aforementioned Methodology Note submitted at the conclusion of Phase 1 contains the definition of the sampling criteria (stratifications), which respect the sites indicated in the TOR for the functional literacy interventions. These criteria include a specification over and above the specifications in the initial proposal to better respond to the evaluation objectives, adapt to the concentration of beneficiaries in the selected regions, and guarantee the total number of interviews to conduct according to the TOR, while respecting the need to include the gender dimension in the evaluation.

In preparing the survey plan, contractual obligations were respected, along with the survey constraints, notably the available resources and the delays that had occurred. In choosing the sampling technique, both bias and variance had to be minimized. Given the rich sampling base that was available (exhaustiveness and reliability) and the evaluation context, the Consultant opted for a systematic survey with equal probability within the strata constructed. This type of survey is very common, as it offers better-quality results and requires no adjustments of any type; however, the sample is widely distributed geographically, which had direct implications for the cost of collecting data in the field. The beneficiaries will therefore be classified in a very specific order (by the different types of stratification variables), and the sample will be drawn systematically with a probability ranging from "equal" to "irregular."

The starting unit is drawn through simple random selection among the numbers from 1 to "not drawn." The units to survey are then drawn automatically using appropriate software.

Sampling deviation

The size of the total sample for the AFM and FLVT evaluations, originally set at 1297 individuals in the TOR, was cut to 770 following APP’s decision, communicated on 4 July 2013, to eliminate the Vocational Training and PEAQC activities from the Transtec-AC Consortium evaluation project. The final sample wovered 4 regions instead of 14.

Research instrument

Four data collection tools were used in the evaluation of the functional literacy subactivity: -A pre- and post-activity document review -Field surveys of beneficiaries -Three types of focus groups (with beneficiaries, literacy workers, and functional literacy service providers) -Semi-structured interviews with all categories of stakeholders in the functional literacy program

Cleaning operations

CSPro also enables users to set up a data entry control module to monitor proper entry of questionnaire data, automatically checking the consistency of the information inputted. The software allows users to create validation rules to ensure correct data entry through well-defined rules or error messages in the case of logical inconsistencies.