Every year, around this time, the College Board publicizes its SAT results, and hundreds of newspapers, blogs, and television stations run stories suggesting that trends in the aggregate scores are, by themselves, a meaningful indicator of U.S. school quality. They’re not.

Everyone knows that the vast majority of the students who take the SAT in a given year didn’t take the test the previous year – i.e., the data are cross-sectional. Everyone also knows that participation is voluntary (as is participation in the ACT test), and that the number of students taking the test has been increasing for many years and current test-takers have different measurable characteristics from their predecessors. That means we cannot use the raw results to draw strong conclusions about changes in the performance of the typical student, and certainly not about the effectiveness of schools, whether nationally or in a given state or district. This is common sense.

Unfortunately, the College Board plays a role in stoking the apparent confusion – or, at least, they could do much more to prevent it. Consider the headline of this year’s press release:

SAT Report: Only 43 Percent of 2012 College-Bound Seniors Are College Ready

This finding was reprinted in dozens of news headlines and ledes. But it is, at best, highly misleading, and this is evident in the very first sentence of the same College Board press release:

The SAT Report on College and Career Readiness released today revealed that only 43 percent of SAT takers in the class of 2012 graduated from high school with the level of academic preparedness associated with a high likelihood of college success. [my emphasis]

The difference between “43 percent of college-bound seniors” and “43 percent of SAT takers” is not just semantic. It is critical. The former suggests that the SAT sample is representative of college-bound seniors, while the latter properly limits itself to students who happened to take the SAT this year. Without evidence that test takers are representative – which is a very complicated issue, especially given that SAT participation varies so widely by state - average U.S. SAT scores don’t reflect how “college ready” the typical college-bound senior might be.

But the situation is even worse when talking about trends. Due to the cross-sectional data, coupled with the changing demographic composition of test-takers, the trends in aggregate SAT scores are mostly a function of who’s taking the test, not the quality of U.S. schools. This is especially salient when changes are small, as was the case this year.

The College Board, to their credit, does not report trends in their release, but might very well implicitly encourage them by presenting aggregate scores as representative of the college-bound senior population, and failing to warn against the problems inherent in looking at changes in scores over time. Actually, the press release itself doesn’t provide any warnings at all.

And, predictably, story after story noted the miniscule decrease in the average score (with trumped-up phrases such as “a 40-year low”), and many even used this to gently or directly suggest that overall student performance is on the decline and our education system is failing.

Reporters or others who wish to read up on correct interpretation can either refer to the selections included in the full report or, better yet, see the SAT “proper uses” documentation. There they will find warnings against the very same types of conclusions that made their way into news stories throughout the nation (though, again, direct warnings regarding trends are tough to find in College Board documentation).*

The annual confusion about SAT results (which has been lamented by researchers for a few decades) is, for me, exceedingly disheartening. If we as a public don’t always seem to understand that the SAT – a completely voluntary test, with ever-changing test-taker demographics – cannot be compared across years, then there may be little hope for proper interpretation of state test data, which suffer similar, but far less obvious limitations.

- Matt Di Carlo

*****

* These provide warnings such as: “… it is important to note that many College Board tests are taken only by particular groups of self selected students. Therefore, aggregate results of their performance on these tests usually do not reflect the educational attainment of all students in a school, district or state.”