Source: CNN

The World Economic Forum’s latest Global Competitiveness Report, which assesses the economies of 144 countries and ranks them based on more than 100 indicators, had something to say about education. This is hardly surprising since education is a main determinant of economic competitiveness. The question, though, is whether the index – which relies heavily on business executive assessments for the ranking – really gets education right.

After looking at the country rankings, it’s difficult not to have doubts.

For example, Lebanon and Qatar were among the top 10 countries in terms of quality of math and science education, quality of primary education, and more generally quality of the educational system. Though no doubt a pleasant surprise for Lebanese and Qataris, this assessment is a significant departure from the results of student achievement tests in every international test Lebanon and Qatar have participated in. In the international student achievement tests in math and sciences (TIMSS) in 2007 and 2011 for grade 8, Lebanese and Qatari students scored well below the international average of 500. In math, Lebanon scored 449 in both 2007 and 2011, while Qatar scored 307 in 2007, but 410 in 2011.

As for the quality of primary education, available data on student achievement for grade 4 in math and science from TIMSS 2011, and in reading literacy from PIRLS (Progress in International Reading Literacy Study) 2011, show Qatar’s scores were significantly below average. Lebanon did not participate in these tests, but the performance of grade 8 students in international tests does not suggest that they had a high quality primary education.

And there are other problems with the WEF assessment of education quality. For example, the percentage of students at the advanced and high levels of achievement in math and science at a certain grade surely reflects the quality of education in those subjects. But in Lebanon, only 1 percent of students in grade 8 performed at the advanced level in math and 9 percent at the high level. In Qatar, the comparable figures were 2 percent and 10 percent, respectively. In contrast, the percentages of students performing at the advanced level in grade 8 math ranged between 8 percent and 49 percent in the top ten countries in TIMSS 2011.

The shortcomings in WEF’s estimation of the quality of education are not restricted to Lebanon and Qatar. Russia, which ranked among the top 10 countries in student achievement in both math and science for grade 8, ranked 52nd in the quality of math and science education in the WEF data. Russia also ranked 62nd in the WEF report on the quality of primary education; the average scores of its grade 4 students in math and science in TIMSS 2011 and in reading in PIRLS 2011 placed Russia among the top 10 of participating countries.

In terms of primary education, Hong Kong was ranked 29th in the WEF report despite its grade 4 students receiving the top rank in reading (PIRLS 2011), third place in math and eighth in science. The United States, meanwhile, was ranked 38th although it was among the top 10 countries in reading, math and science for grade 4.

So why these shortcomings in assessing education? The problem is largely the result of the subjective nature of the “executive opinion survey” data. Respondents were not education experts offering informed opinions, and are therefore more likely to base their opinions on their own personal experiences, perhaps at quality private schools.

All this is not to argue that the shortcomings on education would have radically altered the overall results of the competitiveness rankings. But the WEF would do well to rethink its dependence on such a potentially subjective source of information.

This article was originally published in CNN.