Since 2002, when the federal No Child Left Behind Act was signed into law, American public schools, and later their teachers, have been evaluated by the standardized test scores of their students.  States have been required to punish the schools with the lowest scores—firing their principals or some of their teachers, closing the schools, or turning them over to charter schools.  But the idea that we can judge schools and judge teachers by metrics—by the aggregate test scores of their students—evolved long before the passage of No Child Left Behind—even prior to the publication in 1983 of the A Nation At Risk report that is said to have begun the wave of standards-based school reform. Perhaps it has been part of growing enchantment with our society’s advancing capacity to collect and analyze data.

Today it is becoming widely acknowledged, however, that the strategy of test-and-punish didn’t improve public schools, didn’t identify the best or the worst teachers, and didn’t help students who had been left far behind.  Even so, the narrative that we should judge schools by their test scores has become what everybody just believes—the conventional wisdom.

In a new commentary, Education Reforms: Everything Important Cannot Be Measured, (also published here), William Mathis, the managing director of the National Education Policy Center at the University of Colorado at Boulder, traces the long history of our belief that raising test scores will be the solution to our society’s problems. And he identifies two primary flaws in this thinking.

First, Mathis reminds us that the standardized test scores which carry so much weight these days measure more about what happens in children’s lives outside of school than the tests measure the quality of the school.  As a society, Mathis explains, we misread test scores: “That is, the low scores are strongly affected by circumstances outside the schools. Children coming from violent, economically challenged and drug addicted homes, as a group, are not going to do as well as their more fortunate classmates. As the family income gap between children has widened, the achievement gap has also widened.  A Stanford professor compared all the school districts in the nation using six different measures of socio-economic well-being and found that a stunning 70% of test scores could be predicted by these six factors.  When the PARCC tests, which are used to test “college and career readiness” were compared with freshman grade point average, the tests only predicted between one and 16% of the GPA. What this means is that the tests do a better job of measuring socio-economic status than measuring schools. This pattern has been solidly and consistently confirmed by a mountain of research since the famous Coleman report in 1966.  It pointed to family and social problems rather than schools.”

Second, much of the most important contribution of schools to children’s development can’t be measured at all by standardized tests. Mathis concludes: “In focusing on what is easily measured, rather than what is important, we fail to grasp the real problem. To be sure, tests measure reading and math reasonably well and we need to keep tests for that purpose. But that’s only one part of education. Schools also teach children to get along with others, prepare young people for citizenship, encourage creativity, teach job and human skills, integrate communities, teach tolerance and co-operation, and generally prepare students to be contributing members of society. These things are not so easily measured.” “By concentrating only on the easily measurable, we squeeze the life out of schools. We devalue, de-emphasize and defund things that lead to a better life, better schools and a better civilization…  Parents want their children to grow and lead productive, happy lives and contribute to society. They want their children to practice civic virtue and have loving relationships. But these things are not easily measured by a test.”

As a public school educator and an academic, Mathis brings a unique point of view to this topic.  Under his leadership, the National Education Policy Center (NEPC) publishes expert research on education by academics at major universities.  NEPC also publishes expert, third-party reviews of selected non-peer-reviewed publications—many of them reviews of so-called research from ideological think tanks.  NEPC’s website declares: “Using academic peer review standards, reviewers consider the quality and defensibility of a report’s assumptions, methods, findings, and recommendations.”

But Mathis is also a lifelong public school educator.  He has spent his working life in public schools and he knows from experience what matters in schools. He is the former superintendent of schools of the Rutland Northeast Supervisory Union in Brandon, Vermont. He was a National Superintendent of the Year finalist and a Vermont Superintendent of the Year.  Today Mathis serves on the Vermont State Board of Education.

Baffled by the test-and-punish policy our society has been actively pursuing for decades—despite massive evidence that we ought to be addressing poverty, inequality and segregation along with improving our public schools—Mathis ponders the path we’ve chosen to follow: “We collected more data. We now have ‘data dash-boards.’ Countless ads on the web tout this lucrative market and proclaim how people can ‘drill down,’ create interactive charts and visuals to provide ‘deep learning.’ They display all manner of things such as differences by ethnic group, technical education, graduation rate and a myriad of exotic esoterica.  By all means, we need to continue to collect this important data. The problem is that we already know what the dash-board tells us.”

However, “What it doesn’t tell us is the nature of the real problems and how to correct them. First, we must look to those things outside the school that affect school performance. Second, in addition to hard data, we must use on-the-ground observations to see whether we provide legitimate opportunities to all children, whether the school is warm and inviting, and whether the curriculum is up to date and well-delivered.”