Here’s the (incendiary) headline: Test Scores Show Dramatic Declines!

Here’s the truth: this set of test scores tells us nothing for certain. The data are apples-to-oranges-to bowling balls muddled. If anything, if you still believe test scores give us valuable information, the data might be mildly encouraging, considering what students have encountered over the past 18 months.

More about the numbers later. First, let me share with you the moment I stopped believing that standardized test data had any valid role in determining whether students or schools were successful.

I was attending a State Board of Education meeting in Lansing. These are monthly day-long affairs where education policy and affairs are discussed and instituted. (Sometimes, the legislature passes different laws, in an attempt to undermine the State Board, but that’s not relevant in this example.) The Board, on this occasion, was setting cut scores from a round of new testing data.

I can’t tell you what year this occurred, exactly, but it was after NCLB was passed, and the Board was doing what they were supposed to do: managing the data generated by federally imposed standardized testing, grades 3-8. 

Until that meeting, I assumed that there was a hard, established science to setting cut scores. I thought scores were reasonably reliable, valid measures of learning and there were pre-determined, universal clusters of students who would be labeled proficient, advanced, below basic or whatever descriptors were used. I assumed there were standard, proven psychometric protocols—percentage of correct answers, verified difficulty of questions, and so on. I was familiar with bell curves and skewed distributions and standard deviations.

What surprised me was how fluid—and even biased– the whole process seemed. There was, indeed, a highly qualified psychometrician leading the discussion, but a lot of the conversation centered on issues like: If we set the Advanced bar too low, we’ll have a quarter of the students in Michigan labeled Advanced and we can’t have that! If we move the cutoff for Basic to XX, about 40% of our students will be Below Basic—does that give us enough room for growth and enough reason to put schools under state control?

The phrase “set the bar high” was used repeatedly. The word “proficient” became meaningless. The Board spent hours moving cut bars up and down, labeling groups of students to support their own well-meant theories about whether certain schools were “good” and others needed to be shut down. So much for science.

The problem is this: You can’t talk about good schools or good teachers or even “lost learning” any more, without a mountain of numbers. Which can be inscrutable to nearly everyone, including those making policies impacting millions of children. When it comes to standardized test score analysis, we are collectively illiterate. And this year’s data? It’s meaningless.

Bridge Magazine (headline: Test Scores Slump) provides up/down testing data for every school district in Michigan. The accompanying article includes plenty of expert opinion on how suspect and incomplete the numbers are, but starts out with sky-is-falling paragraphs:  In English, the share of third-graders considered “proficient” or higher dropped from 45.1 percent to 42.8 percent; in sixth-grade math, from 35.1 percent to 28.6 percent; in eighth-grade social studies, from 28 percent to 25.9 percent.

These are, of course, aggregated statewide numbers. Down a few percent, pretty much across the board. Unsurprising, given the conditions under which most elementary and middle school students were learning. Down the most for students of color and those in poverty—again, unsurprising. Still, there’s also immense score variance, school to school, even grade to grade. The aggregate numbers don’t tell the whole story–or even the right story.

The media seemed to prefer a bad-news advertising campaign for the alarming idea that our kids are falling further behind. Behind whom, is what I want to know? Aren’t we all in this together? Is a two-point-something score drop while a virus rages reason to clutch your academic pearls?

Furthermore: what does ‘proficient’ even mean? It’s a word which appears repeatedly, with absolutely no precise definition. Everybody (including media) seems to think they understand it, however.

The really interesting thing was looking at district-by-district data. There were places where pretty much everybody took the tests, and schools where almost nobody did. Districts where the third grade scores dropped twenty percent while the fourth grade, in the same school, went up eight percent. What happened there—was it teachers? curriculum? It was also clear that charters, including virtual charters, were not the shining solution to pandemic learning.

What I took away from the data is that public education held up pretty well in Michigan, under some dire and ever-shifting conditions. In some places, kids and teachers did very well, indeed, amidst disruption. Kids without resources—broadband, devices, privacy, constant adult supervision, or even breakfast and lunch—had the hardest time. They’re the ones who need the most attention now. And good luck hiring qualified, experienced teachers to do that.

There’s probably a lot that can be learned from a close look at the 2020-21 data, but most of it isn’t about quantified student learning gains. And please—stop with the “acceleration” crapola. The pace of learning will improve when our students feel safe and part of a community, the exact conditions we’ve been striving for in perpetuity, and aren’t present anywhere, in September 2021.

Stu Bloom said, last week: I’m seriously tired of the politicians, pundits (looking at you, NYT Editorial Board), and policy-makers telling teachers and public schools to single-handedly solve the problems of racism and poverty by increasing test scores. Public schools and public school teachers are not the only ones who have anything to contribute to growing our society!

He then goes on to point out the value of actually investing in public education, in evidence-based policies and practices, designed to improve life and learning for all school-aged children. We know what to do, he says. And he’s right.

It’s time to end our national love affair with testing, to make all Americans understand that educational testing is a sham that’s harmed many children. Testing hasn’t ever worked to improve public education outcomes, and it’s especially wasteful and subject to misinterpretation right now.