Perspectives

Taking issue with PISA

By Bernie Froese-Germain
March 7, 2014

In less than two decades the OECD’s Programme for International Student Assessment (PISA) has become the widely used proxy for the performance of education systems around the world. No other international study of education commands as much attention as PISA. The number of participating countries, and increasingly “economies”, grows with each testing cycle – countries feel they cannot be left out of this influential test.

The OECD’s interest in education in the service of human capital development and economic growth is not surprising given the organization’s focus. From the OECD’s perspective, in an age of intense global competition among information-based economies, education is increasingly viewed as an important (if not the most important) national “competitive advantage” – hence comparing education systems is key to improving them. PISA 2012 results were released in early December.

There are a number of things we find problematic about PISA in general, and PISA 2012 in particular.

The obsessive focus on the PISA rankings leads to countries being praised or shamed for their relative performance. Educators have been raising concerns about test score rankings and league tables for years, with good reason – it bears repeating that the quality and complexity of a country’s education system cannot be reduced to a single number (there are parallels here with the limitations of using narrow indicators such as the Gross Domestic Product (GDP) per capita to assess national economic health in a broad sense). A poor PISA ranking not only neglects the positive developments occurring in a school system, it could potentially undo them as the focus shifts to improving test results in a narrow range of subjects to improve one’s ranking for the next testing cycle. Education International (EI) notes that "education systems which are improving often occupy the lower part of the tables. Also OECD itself acknowledges that because of sampling a single league table position is not accurate.”

The narrow focus on numeracy, reading and science is another problematic aspect of PISA (and the addition in 2012 of financial literacy is not our idea of broadening PISA’s focus). While we recognize the importance of these core subjects for student learning, the assumption that these literacies are a proxy for the quality of our education systems is highly misleading. And yet every three years PISA reinforces the importance of the core subjects at the expense of everything else that goes on in schools, essentially hijacking the conversation about the broad goals of public education.

Continuing a trend from PISA 2009, the Asian countries and economies were the top performers in math on PISA 2012 – Shanghai-China, Singapore, Hong Kong-China, Chinese Taipei, Korea, Macau-China and Japan. According to EI, this “raises a question about how scientifically valid and fair it is to compare geographically small regions such as Shanghai, Macau and Hong Kong in China with whole large and diverse countries in other continents.” Is the OECD in essence comparing apples and oranges? Other countries including Canada are assessed on the basis of country-wide performance. Irrespective of how representative the Chinese economies are of the country as a whole (arguably not very representative), they will undoubtedly be incorrectly used as a proxy for China’s performance on PISA.

While PISA may not be a high-stakes test with respect to individual students and schools, the stakes are indeed high at a national policy level. In his analysis of the policy impact of PISA, Breakspear states that “PISA is becoming an influential element of education policy-making processes at the national level. Furthermore, the findings provide preliminary evidence that PISA is being used and integrated within national/federal policies and practices of assessment and evaluation, curriculum standards and performance targets.” Through PISA, the OECD wields enormous influence over education policy in participating PISA jurisdictions (and beyond) to create uniform policies that have the effect of reducing schooling to occupational training (Weiner 2013).

The moderate decline in Canada’s PISA 2012 math scores prompted some alarmist rhetoric from CEO and President of the Canadian Council of Chief Executives, John Manley, who stated: “This is on the scale of a national emergency” (Alphonso 2013). Declining test scores are no more a threat to our economic prosperity than rising scores are a guarantee of that prosperity; our economic prosperity has less to do with minor changes in test scores than with broader economic and social policies and trends.

PISA appears to be here to stay as EI General Secretary Fred Van Leeuwen remarks: “Ministers’ careers can be undermined or enhanced by PISA’s results. It is now seen as the authoritative judgement on education. Indeed, whatever anyone might think of it, it will be a key part of the educational landscape for the foreseeable future.” Our considerable challenge as Dunleavy notes is to identify what we can, and can't, learn from PISA as well as continue to raise concerns over the use and misuse of the results.

Ultimately however we cannot lose sight of this fact: for the OECD, PISA is a powerful tool in its growing education arsenal that governments and corporate interests can use to shape education systems to serve narrow economic objectives.

Is this really what good schools and quality public education are all about?

References

Alphonso, Caroline (Dec. 3, 2013). “Canada's fall in math-education ranking sets off alarm bells.” Globe & Mail.
www.theglobeandmail.com/news/national/education/canadas-fall-in-math-education-ranking-sets-off-red-flags/article15730663/

Breakspear, Simon (2012). “The Policy Impact of PISA: An Exploration of the Normative Effects of
International Benchmarking in School System Performance.” OECD Education Working Papers, No. 71,
OECD Publishing.

Dunleavy, Jodene (Summer 2011). “Ranking our responses to PISA 2009.” Education Canada.
www.cea-ace.ca/education-canada/article/ranking-our-responses-pisa-2009

Education International (Dec. 2, 2013). “Key messages from PISA 2012. An analysis and briefing from Education International.”
http://download.ei-ie.org/Docs/WebDepot/Circular_PISA2012_E.pdf
www.ei-ie.org/en/news/news_details/2788

Van Leeuwen, Fred (Dec. 3, 2012). “‘Pisa makes it clear that no education system can be successful without confident teachers’.” TES News (Opinion blog).
http://news.tes.co.uk/opinion_blog/b/weblog/archive/2013/12/03/39-pisa-makes-it-clear-that-no-education-system-can-be-successful-without-a-confident-teachers-39.aspx

Weiner, Lois (Dec. 4, 2013). “How teachers unions should respond to the PISA test results.” New Politics blog.
http://newpol.org/content/how-teachers-unions-should-respond-pisa-test-results

Bernie Froese-Germain is a Researcher with the Canadian Teachers’ Federation.

Publication:
Canadian Teachers’ Federation
2490 Don Reid Drive
Ottawa, ON K1H 1E1
Tel: 613-232-1505
Fax: 613-232-1886
Toll Free: 1-866-283-1505

Perspectives web magazine is published by the Canadian Teachers’ Federation (CTF), a national alliance of provincial and territorial teacher organizations that represent over 238,000 elementary and secondary school teachers across Canada.

Editor In Chief: Francine Filion | Translation and Editing: Marie‑Caroline Uhel and Marie‑Hélène Larrue
Proofreading: Denise Léger
Graphic Design: Nathalie Hardy and Jean-Louis Lauriol | Web Design: Greg Edwards

Permission:
Requests for permission to reproduce any part of this publication for academic, professional, or commercial purposes should be sent to info@ctf-fce.ca. Articles and advertisements do not necessarily reflect the views of the CTF.

Comments: info@ctf-fce.ca