This is the 10th of 20 posts I will be writing during the PARCC testing window of March 9 – April 10.  If you’re interested in the growing master list of reasons we opted out of standardized tests in 2015, you can find it here.

Reason 10:  Perpetuating the Myth of American Failure

My last post discussed the fact that American students have been improving on the National Assessment of Educational Progress (NAEP) exam over the past 40 years despite the common belief perpetuated by the media, politicians, and education reformers that they have not.  Today I discuss how this manipulation of understanding possible and it involves a little-known but easy to understand statistical phenomenon known as “Simpson’s Paradox”.

To put it somewhat simply, Simpson’s paradox is what occurs when the “observed variable” (in this case it’s test scores) is affected by “lurking variables” (in this case it’s race and class) so that the observed variable doesn’t tell the whole story.  That’s an important point to remember.  Observed variables (overall test scores) don’t tell the whole story.

Let’s look at the 4th Grade NAEP Reading scores as an example.

The overall scores on this test have improved 11 points since 1975.  That doesn’t seem like a lot, but that belies the progress our schools have made with African American and Hispanic students since that time.   As you can see in the graph, both African American and Hispanic students have improved 25 points over that same time period.  So why haven’t overall scores improved more than they have?  It’s because the demographics of American schools are changing and the percentage of non-white test takers is much higher than it was in the past.  Here’s a look at how much the demographics have changed since 1990.

So the lurking variable of our changing demographics is affecting the overall average score.  The percentage of white test takers has decreased by 22% in 23 years, while the percentage of African American and Hispanic test takers has increased by 3% and 19% respectively.  All of these groups of students are improving, especially the African American and Hispanic students. But the overall numbers appear somewhat flat because we have more students who are non-white than ever before.  They are becoming a larger part of the pool of test takers so they are making up a larger proportion of the overall score.  To be clear, these students are not “dragging our average down”.  It’s simply an unfortunate fact that students who are not white have scored lower than white students over time.  The good news is that our white and non-white students are improving even if the overall average obscures that progress by washing out the details.  We still have a great deal of work to do though.

I’m willing to bet that 9 out of 10 readers of this post did not know this (based on our research), and that’s why it is possible for politicians, pundits and education reformers to manipulate you with oversimplified numbers.

For next time (March 30th): How “Closing the Gap” is all about the wrong “Gap”