Why global education rankings don't reveal the whole picture

Image 20170214 25977 k3uo90.jpg?ixlib=rb 1.1
Is east Asia really the top dog when it comes to education? shutterstock
Daniel Caro, University of Oxford and Jenny Lenkeit, University of Oxford

Country rankings in international education tests – such as PISA and TIMSS – are often used to compare and contrast education systems across a range of countries. But it isn’t always an even playing field.

This is because countries with very different social and economic realities participate, so countries such as Norway, Russia, Chile, Lebanon and Thailand are all being compared against each other. And this is without the difference in socio-economic backgrounds of these different countries being taken into account.

If the latest world education rankings are anything to go by, Turkey and Thailand perform poorly when it comes to their students’ achievement in science. But our analysis shows that if you look at the rankings differently (from an even starting point), both Thailand and Turkey may in fact be just as good as some of the high performing Asian countries.

Our analysis is a much fairer comparison, as it allows for the differences in wealth and social development in which students learn and teachers teach. It builds upon our previous work, where we produced and analysed an indicator of “effectiveness”. The effectiveness indicator ranks performance of countries as if they all had similar socio-economic conditions – thus levelling the playing field.

This makes it easier to see which countries are actually the most effective at educating their students, with social economic factors like wealth taken into account.

New style rankings

The graph below shows how countries are ranked in their effectiveness. At the top of the effectiveness ranking, we find education systems such as Singapore and Japan, which are also generally high performing in PISA and TIMSS.

TIMSS-PISA 2015. Values higher than zero (towards the right-hand side) indicate that students in those education systems perform above expectations, meaning the education system is effective. Those values below zero, to the left, indicate ineffective education systems. Author provided

But our analysis also revealed that countries such as Turkey and Thailand are actually highly effective and perform above expectations in terms of education. This is despite both countries having an overall lower performance score in the global education rankings.

As the graph shows, the performance of education systems in Turkey and Thailand is underestimated if guided by country rankings alone. This is because although these countries perform below average and rather poorly in PISA, they are as effective as high performing Asian countries.

This means that Turkey and Thailand would be ranked among the highest performing countries in the world – if there was no socio-economic differences between countries.

TIMSS-PISA 2015. Effectiveness vs performance according to rankings. The horizontal line at 0 separates effective from ineffective systems and the vertical line at 500 indicates average performance for TIMSS and OECD education systems. Author provided

Our analysis also shows education systems in Norway are ineffective, and the same was found to be true of Australia. So while these countries are ranked highest in human development in the world, they are not among the highest performing in these international tests when we level the playing field.

On the lower end of the effectiveness ranking, we find Qatar and the United Arab Emirates. These countries are also among the lowest performing in PISA rankings and could be doing much better for their high income per capita levels.

Fair comparison

It is clear then that overall performance rankings alone do not make a fair comparison when it comes to judging the quality of education in different countries. And our analysis shows how the socioeconomic conditions of a country are vitally important when comparing global performance in education rankings.

The ConversationUsing our data, there would certainly be a case for countries like Chile, Malta, Georgia, the United Arab Emirates and Qatar to look to Turkey and Thailand to work out how to improve their education systems. And as our analysis shows, global education rankings are probably not the best measure of educational performance after all.

Daniel Caro, Research Fellow, Oxford University Centre for Educational Assessment, University of Oxford and Jenny Lenkeit, Research Fellow, Oxford University Centre for Educational Assessment, University of Oxford

This article was originally published on The Conversation. Read the original article.

Which countries punch above their weight in education rankings?

Image 20151123 18227 1xcbdhz.jpg?ixlib=rb 1.1
Thinking differently about comparing education systems.
Daniel Caro, University of Oxford and Jenny Lenkeit, University of Oxford

Rankings of countries based on how well their students perform in the Programme for International Student Assessment (PISA) always receive a great deal of attention from the media and politicians. But PISA rankings, produced by the Organisation for Economic Cooperation and Development, are limited when it comes to evaluating the quality of education systems and their efforts to improve children’s lives.

The rankings tell us, for example, that 15-year-olds in Germany and the US perform better in mathematics than students in Peru or Indonesia. Yet we also know that income per capita in Peru and Indonesia is way less than that in Germany and the US.

So is it fair to compare education systems operating in such different socio-economic conditions? How would these education systems perform if they served students and operated in countries with more or less similar socio-economic characteristics?

In studies of how well individual children do and the “effectiveness” of individual schools, pupils’ socio-economic characteristics are now ritually taken into account. The same reasoning can be extended when comparing countries’ education systems.

Our research has measured the effectiveness of education systems by adjusting the performance of students who take the PISA tests for the socio-economic context. The effectiveness measure is obtained by calculating the difference between how well students in the education systems ranked by the OECD actually perform, and how they should be expected to perform due to the socio-economic characteristics of students, schools, and countries.

In the graph below, those countries with values higher than zero – towards the right-hand side – have 15-year-olds who perform above expectations in mathematics, meaning the education system is effective. Those below zero, to the left, indicate ineffective performance.

PISA 2012, effectiveness of education systems in mathematics performance. Note: Albania and Lichtenstein were removed for missing data in the economic, social and cultural status index and sample size restrictions. Author provided

The results show a different configuration of countries’ performance once the socio-economic context is considered. For example, Turkey, Thailand, and Indonesia are effective systems, once results are adjusted in this way, although their absolute performance in the PISA tests is below average. Conversely, the US, Sweden, and Norway are among the least effective systems if socio-economic context is taken into account, but exhibit higher absolute performance scores.

There are also systems that perform highly according to both absolute scores and for effectiveness once their performance is adjusted for socio-economic context – such as Hong Kong, Korea, and Chinese Taipei. Others have both low absolute performance scores and effectiveness measures, such as Argentina, Jordan, and Qatar. And there are also systems which perform within their expected range, such as Mexico, Spain, Finland, and New Zealand.

Taking wealth into account

The second graph below shows the relationship between absolute mathematics scores in PISA 2012 and effectiveness measures, adjusted for socio-economic context. Those education systems that perform well in absolute terms do also tend to be more effective, but the relationship is not perfect and there are considerable differences between the two.

For example, absolute mathematics performance in Norway and the US is similar to performance in Portugal – but Portugal is effective and Norway and the US are not. For their socio-economic context, Norway and the US perform below what can be expected, whereas Portugal exceeds its expected performance. You could argue that for their advantaged socio-economic conditions, the US and Norway could perform considerably higher than they do.

Which countries are most effective? PISA 2012, math absolute scores versus effectiveness scores, adjusted for socio-economic context. The horizontal line at 0 separates effective from ineffective systems and the vertical line at 500 score points indicates the OECD average. Albania and Lichtenstein removed for missing data. PISA 2012 data., Author provided

It’s clear that some countries perform very differently when their socio-economic situation is taken into account, whereas others perform more or less the same. For example, performance in Mexico, Spain, Finland, and New Zealand practically does not change after the socio-economic context is considered.

Better than expected

But performance is much higher than expected in Thailand, Turkey, and Portugal. Put differently, these systems would score higher in PISA rankings if it took into account the socio-economic context of the countries.

There are, on the other hand, systems such as Norway, Sweden, the US, Israel, Greece, Jordan, and Qatar that perform worse than expected. It seems that comparatively, these countries could perform better considering the available economic resources and the socio-economic characteristics of the student populations they serve.

But there might be other cultural and economic factors, such as the distribution of economic resources, that condition the results of effectiveness and are difficult to change through reforms to education policy.

The PISA results tell us how countries perform in absolute terms. But looking at how effective education systems are according to their socio-economic context offers a complementary perspective.

The ConversationViewed this way, it is possible for an education system operating in relatively disadvantaged conditions, such as Indonesia, Thailand, Turkey, and Vietnam, to perform better than one with higher performance according to PISA rankings but operating under more favourable socioeconomic conditions, such as Iceland, Norway, the UK, the US, and Sweden.

Daniel Caro, Research Fellow, University of Oxford and Jenny Lenkeit, Research Fellow, Oxford University Centre for Educational Assessment, University of Oxford

This article was originally published on The Conversation. Read the original article.