Source data assessment of statistical capacity (scale 0 - 100) in Comoros was reported at 40 in 2020, according to the World Bank collection of development indicators, compiled from officially recognized sources. Comoros - Source data assessment of statistical capacity (scale 0 - 100) - actual values, historical data, forecasts and projections were sourced from the World Bank on November of 2025.

Source data assessment of statistical capacity (scale 0 - 100) in Bolivia was reported at 60 in 2020, according to the World Bank collection of development indicators, compiled from officially recognized sources. Bolivia - Source data assessment of statistical capacity (scale 0 - 100) - actual values, historical data, forecasts and projections were sourced from the World Bank on November of 2025.

Men’s municipality index Life quality is a balance of all the themes that measure quality of life. Detailed indicators are normalised so that all municipal values are placed on a scale from 0 to 100 where 0 is the worst and 100 is best (for some indicators, inverted scale is used). In the next step, the standardised indicator values are weighed together into indices at aspect level. This is done with averages, all indicators weighed together with the same weight in each aspect. The values are also at this level in the range 0 to 100. Then the index at aspect level is weighed together to the thematic level according to the same principle and these values also fall between 0 and 100. Finally, the value of all themes is weighed together according to the same principle, with the same weight, into an overall quality of life index. Men’s municipality index Life quality is a balance of all the themes that measure quality of life. Detailed indicators are normalised so that all municipal values are placed on a scale from 0 to 100 where 0 is the worst and 100 is best (for some indicators, inverted scale is used). In the next step, the standardised indicator values are weighed together into indices at aspect level. This is done with averages, all indicators weighed together with the same weight in each aspect. The values are also at this level in the range 0 to 100. Then the index at aspect level is weighed together to the thematic level according to the same principle and these values also fall between 0 and 100. Finally, the value of all themes is weighed together according to the same principle, with the same weight, into an overall quality of life index.

Malawi MW: SPI: Pillar 4 Data Sources Score: Scale 0-100 data was reported at 49.008 NA in 2023. This stayed constant from the previous number of 49.008 NA for 2022. Malawi MW: SPI: Pillar 4 Data Sources Score: Scale 0-100 data is updated yearly, averaging 48.358 NA from Dec 2015 (Median) to 2023, with 9 observations. The data reached an all-time high of 49.008 NA in 2023 and a record low of 43.008 NA in 2017. Malawi MW: SPI: Pillar 4 Data Sources Score: Scale 0-100 data remains active status in CEIC and is reported by World Bank. The data is categorized under Global Database’s Malawi – Table MW.World Bank.WDI: Governance: Policy and Institutions. The data sources overall score is a composity measure of whether countries have data available from the following sources: Censuses and surveys, administrative data, geospatial data, and private sector/citizen generated data. The data sources (input) pillar is segmented by four types of sources generated by (i) the statistical office (censuses and surveys), and sources accessed from elsewhere such as (ii) administrative data, (iii) geospatial data, and (iv) private sector data and citizen generated data. The appropriate balance between these source types will vary depending on a country’s institutional setting and the maturity of its statistical system. High scores should reflect the extent to which the sources being utilized enable the necessary statistical indicators to be generated. For example, a low score on environment statistics (in the data production pillar) may reflect a lack of use of (and low score for) geospatial data (in the data sources pillar). This type of linkage is inherent in the data cycle approach and can help highlight areas for investment required if country needs are to be met.;Statistical Performance Indicators, The World Bank (https://datacatalog.worldbank.org/dataset/statistical-performance-indicators);Weighted average;

Niger NE: Source Data Assessment of Statistical Capacity: Scale 0 - 100 data was reported at 80.000 NA in 2017. This stayed constant from the previous number of 80.000 NA for 2016. Niger NE: Source Data Assessment of Statistical Capacity: Scale 0 - 100 data is updated yearly, averaging 80.000 NA from Dec 2004 (Median) to 2017, with 14 observations. The data reached an all-time high of 80.000 NA in 2017 and a record low of 60.000 NA in 2005. Niger NE: Source Data Assessment of Statistical Capacity: Scale 0 - 100 data remains active status in CEIC and is reported by World Bank. The data is categorized under Global Database’s Niger – Table NE.World Bank: Policy and Institutions. The source data indicator reflects whether a country conducts data collection activities in line with internationally recommended periodicity, and whether data from administrative systems are available. The source data score is calculated as the weighted average of 5 underlying indicator scores. The final source data score contributes 1/3 of the overall Statistical Capacity Indicator score.; ; World Bank, Bulletin Board on Statistical Capacity (http://bbsc.worldbank.org).; Unweighted average;

Virgin Islands (British) VG: SPI: Pillar 1 Data Use Score: Scale 0-100 data was reported at 60.000 NA in 2022. This stayed constant from the previous number of 60.000 NA for 2021. Virgin Islands (British) VG: SPI: Pillar 1 Data Use Score: Scale 0-100 data is updated yearly, averaging 20.000 NA from Dec 2004 (Median) to 2022, with 19 observations. The data reached an all-time high of 60.000 NA in 2022 and a record low of 20.000 NA in 2015. Virgin Islands (British) VG: SPI: Pillar 1 Data Use Score: Scale 0-100 data remains active status in CEIC and is reported by World Bank. The data is categorized under Global Database’s Virgin Islands (British) – Table VG.World Bank.WDI: Governance: Policy and Institutions. The data use overall score is a composite score measuring the demand side of the statistical system. The data use pillar is segmented by five types of users: (i) the legislature, (ii) the executive branch, (iii) civil society (including sub-national actors), (iv) academia and (v) international bodies. Each dimension would have associated indicators to measure performance. A mature system would score well across all dimensions whereas a less mature one would have weaker scores along certain dimensions. The gaps would give insights into prioritization among user groups and help answer questions as to why the existing services are not resulting in higher use of national statistics in a particular segment. Currently, the SPI only features indicators for one of the five dimensions of data use, which is data use by international organizations. Indicators on whether statistical systems are providing useful data to their national governments (legislature and executive branches), to civil society, and to academia are absent. Thus the dashboard does not yet assess if national statistical systems are meeting the data needs of a large swathe of users.;Statistical Performance Indicators, The World Bank (https://datacatalog.worldbank.org/dataset/statistical-performance-indicators);Weighted average;

Algeria DZ: SPI: Pillar 4 Data Sources Score: Scale 0-100 data was reported at 45.958 NA in 2022. This records a decrease from the previous number of 49.075 NA for 2021. Algeria DZ: SPI: Pillar 4 Data Sources Score: Scale 0-100 data is updated yearly, averaging 49.892 NA from Dec 2016 (Median) to 2022, with 7 observations. The data reached an all-time high of 52.417 NA in 2018 and a record low of 45.958 NA in 2022. Algeria DZ: SPI: Pillar 4 Data Sources Score: Scale 0-100 data remains active status in CEIC and is reported by World Bank. The data is categorized under Global Database’s Algeria – Table DZ.World Bank.WDI: Governance: Policy and Institutions. The data sources overall score is a composity measure of whether countries have data available from the following sources: Censuses and surveys, administrative data, geospatial data, and private sector/citizen generated data. The data sources (input) pillar is segmented by four types of sources generated by (i) the statistical office (censuses and surveys), and sources accessed from elsewhere such as (ii) administrative data, (iii) geospatial data, and (iv) private sector data and citizen generated data. The appropriate balance between these source types will vary depending on a country’s institutional setting and the maturity of its statistical system. High scores should reflect the extent to which the sources being utilized enable the necessary statistical indicators to be generated. For example, a low score on environment statistics (in the data production pillar) may reflect a lack of use of (and low score for) geospatial data (in the data sources pillar). This type of linkage is inherent in the data cycle approach and can help highlight areas for investment required if country needs are to be met.;Statistical Performance Indicators, The World Bank (https://datacatalog.worldbank.org/dataset/statistical-performance-indicators);Weighted average;

United States US: SPI: Pillar 1 Data Use Score: Scale 0-100 data was reported at 100.000 NA in 2019. This stayed constant from the previous number of 100.000 NA for 2018. United States US: SPI: Pillar 1 Data Use Score: Scale 0-100 data is updated yearly, averaging 60.000 NA from Dec 2004 (Median) to 2019, with 16 observations. The data reached an all-time high of 100.000 NA in 2019 and a record low of 40.000 NA in 2009. United States US: SPI: Pillar 1 Data Use Score: Scale 0-100 data remains active status in CEIC and is reported by World Bank. The data is categorized under Global Database’s United States – Table US.World Bank.WDI: Governance: Policy and Institutions. The data use overall score is a composite score measuring the demand side of the statistical system. The data use pillar is segmented by five types of users: (i) the legislature, (ii) the executive branch, (iii) civil society (including sub-national actors), (iv) academia and (v) international bodies. Each dimension would have associated indicators to measure performance. A mature system would score well across all dimensions whereas a less mature one would have weaker scores along certain dimensions. The gaps would give insights into prioritization among user groups and help answer questions as to why the existing services are not resulting in higher use of national statistics in a particular segment. Currently, the SPI only features indicators for one of the five dimensions of data use, which is data use by international organizations. Indicators on whether statistical systems are providing useful data to their national governments (legislature and executive branches), to civil society, and to academia are absent. Thus the dashboard does not yet assess if national statistical systems are meeting the data needs of a large swathe of users.;Statistical Performance Indicators, The World Bank (https://datacatalog.worldbank.org/dataset/statistical-performance-indicators);Weighted average;

Mexico MX: SPI: Pillar 4 Data Sources Score: Scale 0-100 data was reported at 86.900 NA in 2023. This stayed constant from the previous number of 86.900 NA for 2022. Mexico MX: SPI: Pillar 4 Data Sources Score: Scale 0-100 data is updated yearly, averaging 75.975 NA from Dec 2015 (Median) to 2023, with 9 observations. The data reached an all-time high of 86.900 NA in 2023 and a record low of 69.542 NA in 2016. Mexico MX: SPI: Pillar 4 Data Sources Score: Scale 0-100 data remains active status in CEIC and is reported by World Bank. The data is categorized under Global Database’s Mexico – Table MX.World Bank.WDI: Governance: Policy and Institutions. The data sources overall score is a composity measure of whether countries have data available from the following sources: Censuses and surveys, administrative data, geospatial data, and private sector/citizen generated data. The data sources (input) pillar is segmented by four types of sources generated by (i) the statistical office (censuses and surveys), and sources accessed from elsewhere such as (ii) administrative data, (iii) geospatial data, and (iv) private sector data and citizen generated data. The appropriate balance between these source types will vary depending on a country’s institutional setting and the maturity of its statistical system. High scores should reflect the extent to which the sources being utilized enable the necessary statistical indicators to be generated. For example, a low score on environment statistics (in the data production pillar) may reflect a lack of use of (and low score for) geospatial data (in the data sources pillar). This type of linkage is inherent in the data cycle approach and can help highlight areas for investment required if country needs are to be met.;Statistical Performance Indicators, The World Bank (https://datacatalog.worldbank.org/dataset/statistical-performance-indicators);Weighted average;

Liechtenstein LI: SPI: Pillar 1 Data Use Score: Scale 0-100 data was reported at 50.000 NA in 2023. This stayed constant from the previous number of 50.000 NA for 2022. Liechtenstein LI: SPI: Pillar 1 Data Use Score: Scale 0-100 data is updated yearly, averaging 10.000 NA from Dec 2004 (Median) to 2023, with 20 observations. The data reached an all-time high of 50.000 NA in 2023 and a record low of 0.000 NA in 2009. Liechtenstein LI: SPI: Pillar 1 Data Use Score: Scale 0-100 data remains active status in CEIC and is reported by World Bank. The data is categorized under Global Database’s Liechtenstein – Table LI.World Bank.WDI: Governance: Policy and Institutions. The data use overall score is a composite score measuring the demand side of the statistical system. The data use pillar is segmented by five types of users: (i) the legislature, (ii) the executive branch, (iii) civil society (including sub-national actors), (iv) academia and (v) international bodies. Each dimension would have associated indicators to measure performance. A mature system would score well across all dimensions whereas a less mature one would have weaker scores along certain dimensions. The gaps would give insights into prioritization among user groups and help answer questions as to why the existing services are not resulting in higher use of national statistics in a particular segment. Currently, the SPI only features indicators for one of the five dimensions of data use, which is data use by international organizations. Indicators on whether statistical systems are providing useful data to their national governments (legislature and executive branches), to civil society, and to academia are absent. Thus the dashboard does not yet assess if national statistical systems are meeting the data needs of a large swathe of users.;Statistical Performance Indicators, The World Bank (https://datacatalog.worldbank.org/dataset/statistical-performance-indicators);Weighted average;

Cameroon CM: SPI: Pillar 4 Data Sources Score: Scale 0-100 data was reported at 26.292 NA in 2019. This records a decrease from the previous number of 34.625 NA for 2018. Cameroon CM: SPI: Pillar 4 Data Sources Score: Scale 0-100 data is updated yearly, averaging 33.212 NA from Dec 2016 (Median) to 2019, with 4 observations. The data reached an all-time high of 34.625 NA in 2018 and a record low of 26.292 NA in 2019. Cameroon CM: SPI: Pillar 4 Data Sources Score: Scale 0-100 data remains active status in CEIC and is reported by World Bank. The data is categorized under Global Database’s Cameroon – Table CM.World Bank.WDI: Governance: Policy and Institutions. The data sources overall score is a composity measure of whether countries have data available from the following sources: Censuses and surveys, administrative data, geospatial data, and private sector/citizen generated data. The data sources (input) pillar is segmented by four types of sources generated by (i) the statistical office (censuses and surveys), and sources accessed from elsewhere such as (ii) administrative data, (iii) geospatial data, and (iv) private sector data and citizen generated data. The appropriate balance between these source types will vary depending on a country’s institutional setting and the maturity of its statistical system. High scores should reflect the extent to which the sources being utilized enable the necessary statistical indicators to be generated. For example, a low score on environment statistics (in the data production pillar) may reflect a lack of use of (and low score for) geospatial data (in the data sources pillar). This type of linkage is inherent in the data cycle approach and can help highlight areas for investment required if country needs are to be met.;Statistical Performance Indicators, The World Bank (https://datacatalog.worldbank.org/dataset/statistical-performance-indicators);Weighted average;

For detailed information, visit the Tucson Equity Priority Index StoryMap.Download the layer's data dictionaryWhat is the Tucson Equity Priority Index (TEPI)?The Tucson Equity Priority Index (TEPI) is a tool that describes the distribution of socially vulnerable demographics. It categorizes the dataset into 5 classes that represent the differing prioritization needs based on the presence of social vulnerability: Low (0-20), Low-Moderate (20-40), Moderate (40-60), Moderate-High (60-80) High (80-100). Each class represents 20% of the dataset’s features in order of their values. The features within the Low (0-20) classification represent the areas that, when compared to all other locations in the study area, have the lowest need for prioritization, as they tend to have less socially vulnerable demographics. The features that fall into the High (80-100) classification represent the 20% of locations in the dataset that have the greatest need for prioritization, as they tend to have the highest proportions of socially vulnerable demographics. How is social vulnerability measured?The Tucson Equity Priority Index (TEPI) examines the proportion of vulnerability per feature using 11 demographic indicators:Income Below Poverty: Households with income at or below the federal poverty level (FPL), which in 2023 was $14,500 for an individual and $30,000 for a family of fourUnemployment: Measured as the percentage of unemployed persons in the civilian labor forceHousing Cost Burdened: Homeowners who spend more than 30% of their income on housing expenses, including mortgage, maintenance, and taxesRenter Cost Burdened: Renters who spend more than 30% of their income on rentNo Health Insurance: Those without private health insurance, Medicare, Medicaid, or any other plan or programNo Vehicle Access: Households without automobile, van, or truck accessHigh School Education or Less: Those highest level of educational attainment is a High School diploma, equivalency, or lessLimited English Ability: Those whose ability to speak English is "Less Than Well."People of Color: Those who identify as anything other than Non-Hispanic White Disability: Households with one or more physical or cognitive disabilities Age: Groups that tend to have higher levels of vulnerability, including children (those below 18), and seniors (those 65 and older)An overall percentile value is calculated for each feature based on the total proportion of the above indicators in each area. How are the variables combined?These indicators are divided into two main categories that we call Thematic Indices: Economic and Personal Characteristics. The two thematic indices are further divided into five sub-indices called Tier-2 Sub-Indices. Each Tier-2 Sub-Index contains 2-3 indicators. Indicators are the datasets used to measure vulnerability within each sub-index. The variables for each feature are re-scaled using the percentile normalization method, which converts them to the same scale using values between 0 to 100. The variables are then combined first into each of the five Tier-2 Sub-Indices, then the Thematic Indices, then the overall TEPI using the mean aggregation method and equal weighting. The resulting dataset is then divided into the five classes, where:High Vulnerability (80-100%): Representing the top classification, this category includes the highest 20% of regions that are the most socially vulnerable. These areas require the most focused attention. Moderate-High Vulnerability (60-80%): This upper-middle classification includes areas with higher levels of vulnerability compared to the median. While not the highest, these areas are more vulnerable than a majority of the dataset and should be considered for targeted interventions. Moderate Vulnerability (40-60%): Representing the middle or median quintile, this category includes areas of average vulnerability. These areas may show a balanced mix of high and low vulnerability. Detailed examination of specific indicators is recommended to understand the nuanced needs of these areas. Low-Moderate Vulnerability (20-40%): Falling into the lower-middle classification, this range includes areas that are less vulnerable than most but may still exhibit certain vulnerable characteristics. These areas typically have a mix of lower and higher indicators, with the lower values predominating. Low Vulnerability (0-20%): This category represents the bottom classification, encompassing the lowest 20% of data points. Areas in this range are the least vulnerable, making them the most resilient compared to all other features in the dataset.

Japan JP: SPI: Pillar 4 Data Sources Score: Scale 0-100 data was reported at 84.050 NA in 2024. This stayed constant from the previous number of 84.050 NA for 2023. Japan JP: SPI: Pillar 4 Data Sources Score: Scale 0-100 data is updated yearly, averaging 78.317 NA from Mar 2017 (Median) to 2024, with 8 observations. The data reached an all-time high of 84.050 NA in 2024 and a record low of 71.542 NA in 2017. Japan JP: SPI: Pillar 4 Data Sources Score: Scale 0-100 data remains active status in CEIC and is reported by World Bank. The data is categorized under Global Database’s Japan – Table JP.World Bank.WDI: Governance: Policy and Institutions. The data sources overall score is a composity measure of whether countries have data available from the following sources: Censuses and surveys, administrative data, geospatial data, and private sector/citizen generated data. The data sources (input) pillar is segmented by four types of sources generated by (i) the statistical office (censuses and surveys), and sources accessed from elsewhere such as (ii) administrative data, (iii) geospatial data, and (iv) private sector data and citizen generated data. The appropriate balance between these source types will vary depending on a country’s institutional setting and the maturity of its statistical system. High scores should reflect the extent to which the sources being utilized enable the necessary statistical indicators to be generated. For example, a low score on environment statistics (in the data production pillar) may reflect a lack of use of (and low score for) geospatial data (in the data sources pillar). This type of linkage is inherent in the data cycle approach and can help highlight areas for investment required if country needs are to be met.;Statistical Performance Indicators, The World Bank (https://datacatalog.worldbank.org/dataset/statistical-performance-indicators);Weighted average;

This artifact accompanies the SEET@ICSE article "Assessing the impact of hints in learning formal specification", which reports on a user study to investigate the impact of different types of automated hints while learning a formal specification language, both in terms of immediate performance and learning retention, but also in the emotional response of the students. This research artifact provides all the material required to replicate this study (except for the proprietary questionnaires passed to assess the emotional response and user experience), as well as the collected data and data analysis scripts used for the discussion in the paper.

Dataset

The artifact contains the resources described below.

Experiment resources

The resources needed for replicating the experiment, namely in directory experiment:

alloy_sheet_pt.pdf: the 1-page Alloy sheet that participants had access to during the 2 sessions of the experiment. The sheet was passed in Portuguese due to the population of the experiment.

alloy_sheet_en.pdf: a version the 1-page Alloy sheet that participants had access to during the 2 sessions of the experiment translated into English.

docker-compose.yml: a Docker Compose configuration file to launch Alloy4Fun populated with the tasks in directory data/experiment for the 2 sessions of the experiment.

api and meteor: directories with source files for building and launching the Alloy4Fun platform for the study.

Experiment data

The task database used in our application of the experiment, namely in directory data/experiment:

Model.json, Instance.json, and Link.json: JSON files with to populate Alloy4Fun with the tasks for the 2 sessions of the experiment.

identifiers.txt: the list of all (104) available participant identifiers that can participate in the experiment.

Collected data

Data collected in the application of the experiment as a simple one-factor randomised experiment in 2 sessions involving 85 undergraduate students majoring in CSE. The experiment was validated by the Ethics Committee for Research in Social and Human Sciences of the Ethics Council of the University of Minho, where the experiment took place. Data is shared the shape of JSON and CSV files with a header row, namely in directory data/results:

data_sessions.json: data collected from task-solving in the 2 sessions of the experiment, used to calculate variables productivity (PROD1 and PROD2, between 0 and 12 solved tasks) and efficiency (EFF1 and EFF2, between 0 and 1).

data_socio.csv: data collected from socio-demographic questionnaire in the 1st session of the experiment, namely:

participant identification: participant's unique identifier (ID);

socio-demographic information: participant's age (AGE), sex (SEX, 1 through 4 for female, male, prefer not to disclosure, and other, respectively), and average academic grade (GRADE, from 0 to 20, NA denotes preference to not disclosure).

data_emo.csv: detailed data collected from the emotional questionnaire in the 2 sessions of the experiment, namely:

participant identification: participant's unique identifier (ID) and the assigned treatment (column HINT, either N, L, E or D);

detailed emotional response data: the differential in the 5-point Likert scale for each of the 14 measured emotions in the 2 sessions, ranging from -5 to -1 if decreased, 0 if maintained, from 1 to 5 if increased, or NA denoting failure to submit the questionnaire. Half of the emotions are positive (Admiration1 and Admiration2, Desire1 and Desire2, Hope1 and Hope2, Fascination1 and Fascination2, Joy1 and Joy2, Satisfaction1 and Satisfaction2, and Pride1 and Pride2), and half are negative (Anger1 and Anger2, Boredom1 and Boredom2, Contempt1 and Contempt2, Disgust1 and Disgust2, Fear1 and Fear2, Sadness1 and Sadness2, and Shame1 and Shame2). This detailed data was used to compute the aggregate data in data_emo_aggregate.csv and in the detailed discussion in Section 6 of the paper.

data_umux.csv: data collected from the user experience questionnaires in the 2 sessions of the experiment, namely:

participant identification: participant's unique identifier (ID);

user experience data: summarised user experience data from the UMUX surveys (UMUX1 and UMUX2, as a usability metric ranging from 0 to 100).

participants.txt: the list of participant identifiers that have registered for the experiment.

Analysis scripts

The analysis scripts required to replicate the analysis of the results of the experiment as reported in the paper, namely in directory analysis:

analysis.r: An R script to analyse the data in the provided CSV files; each performed analysis is documented within the file itself.

requirements.r: An R script to install the required libraries for the analysis script.

normalize_task.r: A Python script to normalize the task JSON data from file data_sessions.json into the CSV format required by the analysis script.

normalize_emo.r: A Python script to compute the aggregate emotional response in the CSV format required by the analysis script from the detailed emotional response data in the CSV format of data_emo.csv.

Dockerfile: Docker script to automate the analysis script from the collected data.

Setup

To replicate the experiment and the analysis of the results, only Docker is required.

If you wish to manually replicate the experiment and collect your own data, you'll need to install:

A modified version of the Alloy4Fun platform, which is built in the Meteor web framework. This version of Alloy4Fun is publicly available in branch study of its repository at https://github.com/haslab/Alloy4Fun/tree/study.

If you wish to manually replicate the analysis of the data collected in our experiment, you'll need to install:

Python to manipulate the JSON data collected in the experiment. Python is freely available for download at https://www.python.org/downloads/, with distributions for most platforms.

R software for the analysis scripts. R is freely available for download at https://cran.r-project.org/mirrors.html, with binary distributions available for Windows, Linux and Mac.

Usage

Experiment replication

This section describes how to replicate our user study experiment, and collect data about how different hints impact the performance of participants.

To launch the Alloy4Fun platform populated with tasks for each session, just run the following commands from the root directory of the artifact. The Meteor server may take a few minutes to launch, wait for the "Started your app" message to show.

cd experimentdocker-compose up

This will launch Alloy4Fun at http://localhost:3000. The tasks are accessed through permalinks assigned to each participant. The experiment allows for up to 104 participants, and the list of available identifiers is given in file identifiers.txt. The group of each participant is determined by the last character of the identifier, either N, L, E or D. The task database can be consulted in directory data/experiment, in Alloy4Fun JSON files.

In the 1st session, each participant was given one permalink that gives access to 12 sequential tasks. The permalink is simply the participant's identifier, so participant 0CAN would just access http://localhost:3000/0CAN. The next task is available after a correct submission to the current task or when a time-out occurs (5mins). Each participant was assigned to a different treatment group, so depending on the permalink different kinds of hints are provided. Below are 4 permalinks, each for each hint group:

Group N (no hints): http://localhost:3000/0CAN

Group L (error locations): http://localhost:3000/CA0L

Group E (counter-example): http://localhost:3000/350E

Group D (error description): http://localhost:3000/27AD

In the 2nd session, likewise the 1st session, each permalink gave access to 12 sequential tasks, and the next task is available after a correct submission or a time-out (5mins). The permalink is constructed by prepending the participant's identifier with P-. So participant 0CAN would just access http://localhost:3000/P-0CAN. In the 2nd sessions all participants were expected to solve the tasks without any hints provided, so the permalinks from different groups are undifferentiated.

Before the 1st session the participants should answer the socio-demographic questionnaire, that should ask the following information: unique identifier, age, sex, familiarity with the Alloy language, and average academic grade.

Before and after both sessions the participants should answer the standard PrEmo 2 questionnaire. PrEmo 2 is published under an Attribution-NonCommercial-NoDerivatives 4.0 International Creative Commons licence (CC BY-NC-ND 4.0). This means that you are free to use the tool for non-commercial purposes as long as you give appropriate credit, provide a link to the license, and do not modify the original material. The original material, namely the depictions of the diferent emotions, can be downloaded from https://diopd.org/premo/. The questionnaire should ask for the unique user identifier, and for the attachment with each of the depicted 14 emotions, expressed in a 5-point Likert scale.

After both sessions the participants should also answer the standard UMUX questionnaire. This questionnaire can be used freely, and should ask for the user unique identifier and answers for the standard 4 questions in a 7-point Likert scale. For information about the questions, how to implement the questionnaire, and how to compute the usability metric ranging from 0 to 100 score from the answers, please see the original paper:

Kraig Finstad. 2010. The usability metric for user experience. Interacting with computers 22, 5 (2010), 323–327.

Analysis of other applications of the experiment

This section describes how to replicate the analysis of the data collected in an application of the experiment described in Experiment replication.

The analysis script expects data in 4 CSV files,

For detailed information, visit the Tucson Equity Priority Index StoryMap.Download the Data DictionaryWhat is the Tucson Equity Priority Index (TEPI)?The Tucson Equity Priority Index (TEPI) is a tool that describes the distribution of socially vulnerable demographics. It categorizes the dataset into 5 classes that represent the differing prioritization needs based on the presence of social vulnerability: Low (0-20), Low-Moderate (20-40), Moderate (40-60), Moderate-High (60-80) High (80-100). Each class represents 20% of the dataset’s features in order of their values. The features within the Low (0-20) classification represent the areas that, when compared to all other locations in the study area, have the lowest need for prioritization, as they tend to have less socially vulnerable demographics. The features that fall into the High (80-100) classification represent the 20% of locations in the dataset that have the greatest need for prioritization, as they tend to have the highest proportions of socially vulnerable demographics. How is social vulnerability measured?The Tucson Equity Priority Index (TEPI) examines the proportion of vulnerability per feature using 11 demographic indicators:Income Below Poverty: Households with income at or below the federal poverty level (FPL), which in 2023 was $14,500 for an individual and $30,000 for a family of fourUnemployment: Measured as the percentage of unemployed persons in the civilian labor forceHousing Cost Burdened: Homeowners who spend more than 30% of their income on housing expenses, including mortgage, maintenance, and taxesRenter Cost Burdened: Renters who spend more than 30% of their income on rentNo Health Insurance: Those without private health insurance, Medicare, Medicaid, or any other plan or programNo Vehicle Access: Households without automobile, van, or truck accessHigh School Education or Less: Those highest level of educational attainment is a High School diploma, equivalency, or lessLimited English Ability: Those whose ability to speak English is "Less Than Well."People of Color: Those who identify as anything other than Non-Hispanic White Disability: Households with one or more physical or cognitive disabilities Age: Groups that tend to have higher levels of vulnerability, including children (those below 18), and seniors (those 65 and older)An overall percentile value is calculated for each feature based on the total proportion of the above indicators in each area. How are the variables combined?These indicators are divided into two main categories that we call Thematic Indices: Economic and Personal Characteristics. The two thematic indices are further divided into five sub-indices called Tier-2 Sub-Indices. Each Tier-2 Sub-Index contains 2-3 indicators. Indicators are the datasets used to measure vulnerability within each sub-index. The variables for each feature are re-scaled using the percentile normalization method, which converts them to the same scale using values between 0 to 100. The variables are then combined first into each of the five Tier-2 Sub-Indices, then the Thematic Indices, then the overall TEPI using the mean aggregation method and equal weighting. The resulting dataset is then divided into the five classes, where:High Vulnerability (80-100%): Representing the top classification, this category includes the highest 20% of regions that are the most socially vulnerable. These areas require the most focused attention. Moderate-High Vulnerability (60-80%): This upper-middle classification includes areas with higher levels of vulnerability compared to the median. While not the highest, these areas are more vulnerable than a majority of the dataset and should be considered for targeted interventions. Moderate Vulnerability (40-60%): Representing the middle or median quintile, this category includes areas of average vulnerability. These areas may show a balanced mix of high and low vulnerability. Detailed examination of specific indicators is recommended to understand the nuanced needs of these areas. Low-Moderate Vulnerability (20-40%): Falling into the lower-middle classification, this range includes areas that are less vulnerable than most but may still exhibit certain vulnerable characteristics. These areas typically have a mix of lower and higher indicators, with the lower values predominating. Low Vulnerability (0-20%): This category represents the bottom classification, encompassing the lowest 20% of data points. Areas in this range are the least vulnerable, making them the most resilient compared to all other features in the dataset.

Source data assessment of statistical capacity (scale 0 - 100) in Belarus was reported at 70 in 2020, according to the World Bank collection of development indicators, compiled from officially recognized sources. Belarus - Source data assessment of statistical capacity (scale 0 - 100) - actual values, historical data, forecasts and projections were sourced from the World Bank on November of 2025.

This dataset simulates the academic, behavioral, and skill development data of 658 secondary vocational school students. Each record represents a student’s multi-dimensional profile including academic performance, skill proficiency, engagement metrics, behavioral traits, and learning preferences.

1. student_id – A unique identifier assigned to each student (e.g., S001–S1600) to ensure individual-level traceability without revealing personal identity.

2. age – Represents the student’s age in years, typically ranging from 15 to 19, reflecting the secondary vocational school population.

3. gender – Indicates the student’s gender identity (Male, Female, or Other), useful for demographic analysis.

4. department – Specifies the vocational stream or academic department such as Mechanical, Electrical, Information Technology, or Design.

5. academic_performance – A composite score (0–100) based on students’ academic achievements, exams, and coursework performance.

6. skill_proficiency_score – Measures practical and technical skill mastery, combining both theoretical and hands-on assessments (0–100 scale).

7. behavioral_score – Reflects classroom behavior, participation, and discipline levels, represented as a 0–10 scale derived from teacher observations.

8. learning_style – Indicates each student’s dominant mode of learning—Visual, Auditory, or Kinesthetic—determined through surveys or observation.

9. engagement_level – A normalized value (0–1) capturing the student’s overall participation rate in learning management systems, assignments, and class interactions.

10. project_completion_rate – Expressed as a percentage (0–100), showing how many projects or assignments a student successfully completed relative to those assigned.

11. peer_collaboration_index – Evaluates teamwork and cooperative learning abilities within group activities, measured on a 0–10 scale.

12. teacher_feedback_score – A numeric value (1–5) representing overall instructor evaluation, including attitude, effort, and growth potential.

13. attendance_rate – The percentage of attended classes out of the total scheduled sessions, typically ranging between 50 and 100 percent.

14. learning_motivation_score – Reflects the student’s intrinsic and extrinsic motivation toward learning, rated between 0 and 10 based on self-assessment or observation.

15. time_management_score – Indicates how efficiently a student plans and manages study time, assignments, and deadlines, rated from 0 to 10.

16. cognitive_flexibility – Measures the student’s ability to adapt to new learning content or shift between different subjects effectively, scored between 0 and 10.

17. learning_module_selected – Shows the currently active learning module or course (Module A–H) chosen by the student or assigned by the RL model.

18. recommended_intervention – Suggests personalized learning support such as Tutoring, Peer Study, Skill Lab, or E-Learning, recommended by the RL system.

19. reinforcement_reward – A numeric reward signal between -1.0 and +1.0 used by the reinforcement learning algorithm to measure improvement or stagnation.

20. growth_path_score (Target 1) – Represents the predicted long-term growth potential of each student, measured on a 0–100 scale based on model outputs.

21. performance_class (Target 2) – A categorical label (Low, Medium, High) derived from the growth_path_score, representing overall student development level.

Kyrgyzstan KG: SPI: Pillar 4 Data Sources Score: Scale 0-100 data was reported at 53.058 NA in 2019. This stayed constant from the previous number of 53.058 NA for 2018. Kyrgyzstan KG: SPI: Pillar 4 Data Sources Score: Scale 0-100 data is updated yearly, averaging 52.271 NA from Dec 2016 (Median) to 2019, with 4 observations. The data reached an all-time high of 53.058 NA in 2019 and a record low of 49.483 NA in 2017. Kyrgyzstan KG: SPI: Pillar 4 Data Sources Score: Scale 0-100 data remains active status in CEIC and is reported by World Bank. The data is categorized under Global Database’s Kyrgyzstan – Table KG.World Bank.WDI: Governance: Policy and Institutions. The data sources overall score is a composity measure of whether countries have data available from the following sources: Censuses and surveys, administrative data, geospatial data, and private sector/citizen generated data. The data sources (input) pillar is segmented by four types of sources generated by (i) the statistical office (censuses and surveys), and sources accessed from elsewhere such as (ii) administrative data, (iii) geospatial data, and (iv) private sector data and citizen generated data. The appropriate balance between these source types will vary depending on a country’s institutional setting and the maturity of its statistical system. High scores should reflect the extent to which the sources being utilized enable the necessary statistical indicators to be generated. For example, a low score on environment statistics (in the data production pillar) may reflect a lack of use of (and low score for) geospatial data (in the data sources pillar). This type of linkage is inherent in the data cycle approach and can help highlight areas for investment required if country needs are to be met.;Statistical Performance Indicators, The World Bank (https://datacatalog.worldbank.org/dataset/statistical-performance-indicators);Weighted average;

Directory content: This directory contains six .CSV files with screening (i.e., IDs ScreeningData) and subjective (i.e., IDs SubjectiveData; MEQr, SSS, BORG, NASA-TLX, MSAQ, for the acronyms see below) data. Data descriptions are reported in an .xlsx file (Legend_DataSubjective).Method and instruments: Through the experiment, we asked drivers to fill in seven questionnaires (digital format). First, we asked drivers to fill in a questionnaire aiming to collect sociodemographic data (e.g., age, handiness). Then, we used the reduced version of the Morningness – Eveningness Questionnaire (MEQr; Adan and Almirall, 1991) and the Spanish Driving Behavior Questionnaire (SDBQ; López de Cózar et al., 2006). The MEQr is a 5-item questionnaire assessing preferences in sleep-wake and activity schedule and allowing the classification of individuals into one of the following subtypes: definitely morning type (22–25 points); moderately morning type (18–21); neither type (12–17); moderately evening type (8–11); and definitely evening type (4–7). The SDBQ is a 34-item questionnaire that allows the identification of adaptive and maladaptive driving styles. Drivers answered the SDBQ items on a Likert scale ranging from 0 (never) to 10 (always).To assess the drivers’ perceived sleepiness and fatigue in three separate measuring times (i.e., pre-driving session, after 90-min of driving, and at the end of the session [after ∼ 180-min]), we administered the Standford Sleepiness Scale (SSS; Hoddes et al., 1973) and the Borg Scale of Perceived Exertion (BORG; Borg, 1998). The SSS provides a global measure of how alert a person is feeling, ranging between 1 and 7. The BORG indicates the level of fatigue, and consists of a numerical scale (ranging from 6 to 20) anchored by “not exertion at all” (score 6) to “maximal exertion” (score 20). To fill both questionnaires after 90 minutes of driving, the participants used the dedicated tablet inside the simulator (for further details, see Driving simulator indices directory). If the vehicle was set in manual driving modality, drivers were instructed to temporarily stop the vehicle.At the end of the driving session, to assess the degree of task complexity and the level of motion sickness experienced, we used the NASA-Task Load Index (NASA-TLX; Hart, 2006) and the Motion Sickness Assessment Scale (MSAQ; Gianaros et al., 2001). The NASA-TLX assesses the task load through six bipolar dimensions: mental, physical, and temporal demand, own performance, effort, and frustration, using a total score between 0 and 100 (higher values indicate higher perceived task load). The MSAQ includes 16 brief statements describing the most common motion sickness symptoms (e.g., “I felt sick to my stomach”) using a Likert scale ranging from 1 (“not at all”) to 9 (“severely”).

Source data assessment of statistical capacity (scale 0 - 100) in Latvia was reported at 60 in 2020, according to the World Bank collection of development indicators, compiled from officially recognized sources. Latvia - Source data assessment of statistical capacity (scale 0 - 100) - actual values, historical data, forecasts and projections were sourced from the World Bank on November of 2025.