Post by Caine Rolleston, Padmini Iyer, Rhiannon Moore, Jack Rossiter & Bridget Azubuike
Educational attainment is as much about where a child goes to school as her home advantage. School systems vary widely in effectiveness – yet there is more nuance in the picture when we examine the overlap between attainment distributions. Despite the large differences in resources and average attainment levels, there are students in Ethiopia whose attainment is as high as in Vietnam. This week, Young Lives has calibrated an internationally comparable test scale which will allow us to examine how school system effectiveness shapes learning across three countries.
Last year, Young Lives went back to school to conduct school effectiveness surveys in Ethiopia, India (Andhra Pradesh and Telangana) and Vietnam. The first round of data collection, carried out at the beginning of the school year, is now complete in all three countries, and data collection at the end of the school year will be complete by July 2017. In each country, we went to schools in the Young Lives sites.
Maths and English tests were administered to students in Grades 7 and 8 in Ethiopia, Grade 9 in India, and Grade 10 in Vietnam in each country. These are the grades in which we expected to find the majority of the Young Lives Younger Cohort children, who were born in 2001-2. On average, we found that students in the Ethiopia survey were 14.4 years old, 14 in India, and 15.4 in Vietnam. The slightly older age of the Vietnamese students in the survey is important to note when looking at findings from the survey, and we will control for this age gap in future analyses.
This week, the Young Lives education team have been working together to examine preliminary trends from the Maths tests results from this unique cross-country education dataset. This blog post outlines some of our early findings.
Maths Performance by Country
A 40-item, multiple-choice Maths test was administered to students at the beginning of the school year[i]; 12 items on these tests were common to all three countries. This allowed us to link the Maths tests across the three countries; we put them on a common IRT scale, and fixed the mean at 500 and the standard deviation at 100 (we’ll be sharing country reports with details of methodology soon). The graph below shows the distribution of Maths test scores in the three countries.
Firstly, this shows that the average performance in each country is largely as we would expect, with the lowest average score at 418 in Ethiopia (82 points below the cross-country mean); India in the middle with 489 (11 points below the cross-country mean), and the highest average score at 586 in Vietnam (86 points above the cross-country mean).
What does this mean in a wider context? Imagine that 600 on our scale is approximately equivalent to the OECD mean on the PISA scale[ii]. Based on this estimation, 2% of students in Ethiopia, 10% in India and 45% in Vietnam are performing at a level comparable to average OECD performance in Maths. This gives us a sense of how students in our sample are performing on an international scale – but the aim of our cross-country analysis is not to rank Ethiopia, India and Vietnam in relation to each other, or to other countries. Instead, our aim is to learn more about potential similarities and differences within and across school systems – this graph highlights the differences between the countries, but importantly also indicates the overlaps in student performance across these diverse contexts.
Maths performance by location
Who are the students whose test scores overlap between the three countries? To answer this question, we began by identifying the most and least advantaged areas across the Young Lives sites, based on existing Young Lives data. The most advantaged area in our Ethiopia sample is Addis Ababa; in our India sample, the least advantaged sites are in rural areas and the most advantaged sites are in urban areas[iii]; and in our Vietnam sample, the least advantaged sites are in Lao Cai province. The graph below shows more information on the overlaps between the countries by presenting the distribution of student Maths performance in each of these areas.
We can see that students in Addis Ababa perform as well as students in rural YL India sites, and students in urban YL India sites perform as well as students in Lao Cai. This reveals that students in the most advantaged area in Ethiopia are performing at the same level as students in the least advantaged area in AP and Telangana, while students in the most advantaged area in AP and Telangana are performing at the same level as students in the least advantaged area in Vietnam.
What is it about students in Lao Cai that means they perform as well as students in our urban sites in India? Similarly, what is it about students in Addis Ababa that means they perform at the level of students in our rural sites in India? These are not questions that can be answered by this preliminary analysis, but these graphs point us towards key areas for further investigation.
Once we have student data from the end of year tests, we’ll be able to consider the amount of progress made by students and the value-added by schools across the three countries, and in these overlapping areas. We will be able to consider whether advantaged students access schools with higher value-added within and across the three countries, and the implications for equity within these school systems – for example, do the most disadvantaged students in Lao Cai make more progress over one year than the most disadvantaged students in rural AP and Telangana?
There are many further areas that the school survey data will allow us to explore – such as the school, teacher, class and student factors associated with levels of progress across different geographical areas and different school types. In addition to these insights into school effectiveness across Ethiopia, India and Vietnam, the 2016-17 Young Lives school surveys provide an important contribution to ongoing debates about how to assess quality learning on a global scale, which is particularly important in light of the emphasis on ensuring quality learning outcomes at primary and secondary level in SDG 4.
[i] 64 schools, approx. 12,000 students in Ethiopia; 202 schools, approx. 10,000 students in India; and 52 schools, approx. 9,000 students in Vietnam.
[ii] This is a rough approximation based on the performance of Vietnamese students in our survey and in PISA; 15 year olds in Vietnam in our survey have a mean score of around 600, while the mean score of Vietnamese students in PISA 201[5] was [500]. Given that the Young Lives sample is pro-poor, we would expect students in our sample to perform at a lower level on the PISA scale. We can therefore infer that 600 on our scale is roughly equivalent to 500 on the PISA scale.
[iii] There are also important factors related to school types in our India sample, which are not reflected here. For example, students at rural private aided schools on average perform better than students at urban state government schools. We will explore performance according to school type in future analyses.
Post by Caine Rolleston, Padmini Iyer, Rhiannon Moore, Jack Rossiter & Bridget Azubuike
Educational attainment is as much about where a child goes to school as her home advantage. School systems vary widely in effectiveness – yet there is more nuance in the picture when we examine the overlap between attainment distributions. Despite the large differences in resources and average attainment levels, there are students in Ethiopia whose attainment is as high as in Vietnam. This week, Young Lives has calibrated an internationally comparable test scale which will allow us to examine how school system effectiveness shapes learning across three countries.
Last year, Young Lives went back to school to conduct school effectiveness surveys in Ethiopia, India (Andhra Pradesh and Telangana) and Vietnam. The first round of data collection, carried out at the beginning of the school year, is now complete in all three countries, and data collection at the end of the school year will be complete by July 2017. In each country, we went to schools in the Young Lives sites.
Maths and English tests were administered to students in Grades 7 and 8 in Ethiopia, Grade 9 in India, and Grade 10 in Vietnam in each country. These are the grades in which we expected to find the majority of the Young Lives Younger Cohort children, who were born in 2001-2. On average, we found that students in the Ethiopia survey were 14.4 years old, 14 in India, and 15.4 in Vietnam. The slightly older age of the Vietnamese students in the survey is important to note when looking at findings from the survey, and we will control for this age gap in future analyses.
This week, the Young Lives education team have been working together to examine preliminary trends from the Maths tests results from this unique cross-country education dataset. This blog post outlines some of our early findings.
Maths Performance by Country
A 40-item, multiple-choice Maths test was administered to students at the beginning of the school year[i]; 12 items on these tests were common to all three countries. This allowed us to link the Maths tests across the three countries; we put them on a common IRT scale, and fixed the mean at 500 and the standard deviation at 100 (we’ll be sharing country reports with details of methodology soon). The graph below shows the distribution of Maths test scores in the three countries.
Firstly, this shows that the average performance in each country is largely as we would expect, with the lowest average score at 418 in Ethiopia (82 points below the cross-country mean); India in the middle with 489 (11 points below the cross-country mean), and the highest average score at 586 in Vietnam (86 points above the cross-country mean).
What does this mean in a wider context? Imagine that 600 on our scale is approximately equivalent to the OECD mean on the PISA scale[ii]. Based on this estimation, 2% of students in Ethiopia, 10% in India and 45% in Vietnam are performing at a level comparable to average OECD performance in Maths. This gives us a sense of how students in our sample are performing on an international scale – but the aim of our cross-country analysis is not to rank Ethiopia, India and Vietnam in relation to each other, or to other countries. Instead, our aim is to learn more about potential similarities and differences within and across school systems – this graph highlights the differences between the countries, but importantly also indicates the overlaps in student performance across these diverse contexts.
Maths performance by location
Who are the students whose test scores overlap between the three countries? To answer this question, we began by identifying the most and least advantaged areas across the Young Lives sites, based on existing Young Lives data. The most advantaged area in our Ethiopia sample is Addis Ababa; in our India sample, the least advantaged sites are in rural areas and the most advantaged sites are in urban areas[iii]; and in our Vietnam sample, the least advantaged sites are in Lao Cai province. The graph below shows more information on the overlaps between the countries by presenting the distribution of student Maths performance in each of these areas.
We can see that students in Addis Ababa perform as well as students in rural YL India sites, and students in urban YL India sites perform as well as students in Lao Cai. This reveals that students in the most advantaged area in Ethiopia are performing at the same level as students in the least advantaged area in AP and Telangana, while students in the most advantaged area in AP and Telangana are performing at the same level as students in the least advantaged area in Vietnam.
What is it about students in Lao Cai that means they perform as well as students in our urban sites in India? Similarly, what is it about students in Addis Ababa that means they perform at the level of students in our rural sites in India? These are not questions that can be answered by this preliminary analysis, but these graphs point us towards key areas for further investigation.
Once we have student data from the end of year tests, we’ll be able to consider the amount of progress made by students and the value-added by schools across the three countries, and in these overlapping areas. We will be able to consider whether advantaged students access schools with higher value-added within and across the three countries, and the implications for equity within these school systems – for example, do the most disadvantaged students in Lao Cai make more progress over one year than the most disadvantaged students in rural AP and Telangana?
There are many further areas that the school survey data will allow us to explore – such as the school, teacher, class and student factors associated with levels of progress across different geographical areas and different school types. In addition to these insights into school effectiveness across Ethiopia, India and Vietnam, the 2016-17 Young Lives school surveys provide an important contribution to ongoing debates about how to assess quality learning on a global scale, which is particularly important in light of the emphasis on ensuring quality learning outcomes at primary and secondary level in SDG 4.
[i] 64 schools, approx. 12,000 students in Ethiopia; 202 schools, approx. 10,000 students in India; and 52 schools, approx. 9,000 students in Vietnam.
[ii] This is a rough approximation based on the performance of Vietnamese students in our survey and in PISA; 15 year olds in Vietnam in our survey have a mean score of around 600, while the mean score of Vietnamese students in PISA 201[5] was [500]. Given that the Young Lives sample is pro-poor, we would expect students in our sample to perform at a lower level on the PISA scale. We can therefore infer that 600 on our scale is roughly equivalent to 500 on the PISA scale.
[iii] There are also important factors related to school types in our India sample, which are not reflected here. For example, students at rural private aided schools on average perform better than students at urban state government schools. We will explore performance according to school type in future analyses.