It’s hard to know whether to laugh or cry when an “All-Purpose-Pundit” at The New York Times takes it upon him/herself to write a commentary about education.

“Thomas Friedman is infamous for his uninformed pieces on education,” Larry Ferlazzo, a retired schoolteacher and ubiquitous education commentator on the Internet recently observed. And there’s “David Brooks, who is equally off-base.”

Diane Ravitch, lamenting a recent column by Times editorialist Bill Keller, who lazily blamed widespread problems with education performance on university teacher preparation programs (without mustering a shred of evidence to support his claim), concluded, “It would be wonderful if the New York Times elevated someone to the op-ed page as a columnist who actually knew something about education.”

Staying true to form last week was Times opinionator Nicholas Kritsof. Prompted by the latest results of the National Assessment of Education Progress, aka “the Nation’s Report Card,” Kristof observed on Twitter, “Latest NAEP school test scores suggest that school reform helps. Big improvements in DC & Tennessee, both centers of reform.”

Since 1990, the “Main NAEP,” given every other year in grades 4 and 8 to measure national and state achievement in reading and math, has led to all sorts of overblown claims. This year’s results have been subjected to the same tendencies – despite the fact that the results were described as “stubbornly mixed” by reliable news outlets, with stagnation in 4th grade reading and math and slight gains in 8th grade reading only.

Unfortunately, delusions suffered by Kristof and his cohort are not limited to pages of The New York Times.

Cue Up The Conservative Messaging Machine

It’s perhaps understandable why Kristof would make the claims he does about the NAEP results, given the level of propaganda pumped into the airwaves by those who claim to be “reforming” education.

Hard at work in the messaging mill every day are a plethora of highly paid operatives in conservative “think tanks” whose missions are to impose their policy prescriptions on public schools.

These outlets were quick to jump on just the sort of statistic Krstof did. Because some states that have implemented the favored policy reforms had gains in their NAEP scores that outpaced the national average, these “analysts” made the case that “reforms” – such as using high-stakes test scores to rate teachers and schools, and increasing the numbers of unregulated, privately operated charter schools – are “working.”

To amplify this message, Michelle Rhee went on camera at MSNBC to claim score gains produced by the District of Columbia, the school system she once led, and Tennessee, a school system she has had considerable influence in, were vindication of very controversial policies that led to her ouster from the D.C. system.

Such claims are truly silly.

As sociologist and professor Aaron Pallas at the Hechinger Institute pointed out, the difference between the NAEP scores from this year and those in the previous year can’t be viewed as “indicator[s] of growth or decline.” To do so would prove only how little you understand about statistics.

To ascertain any real growth or decline in students’ academic achievement, NAEP would need to measure the very same individual students it did in previous years. “But NAEP does not measure the same individual at two points in time. Instead, it measures different individuals at each assessment,” Pallas explained (emphasis original).

The 4th-graders and 8th-graders that NAEP assessed for this year in Tennessee and the District of Columbia aren’t the same ones that were assessed in previous years. Their demographics are different. Their experiences are different. “These changes in demography could account for differences in the average performance.”

Still No Proof Rhee-form Works

Claims that D.C.’s NAEP scores are proof of the effectiveness of reform policies implemented by Rhee are a particularly big stretch.

Writing at the site Public School Shakedown, edu-blogger Guy Brandenberg looked deeper into the data and concluded that the District’s NAEP gains “are a continuation of a trend that has been going on since about 2000 … You will have to strain your imagination to see any huge differences between the trends pre-Rhee and post-Rhee.”

Brandenburg, and others, point out that D.C.’s NAEP gains this year shouldn’t obscure the fact that the District still scores “dead last in the nation.”

Statistical sense aside, the editorial board of The Washington Post nevertheless fell in line with reform propagandists to trumpet, “School reform in the District is working.”

This prompted Bob Somerby, a former teacher who now blogs at The Daily Howler, to note, “Score gains were actually larger before 2007″when Rhee’s tenure began.

“The board should stop playing cheerleader in this arena,” Somerby admonished, “and start behaving like journalists.”

Reform “Works,” Whatever The Results

Yes, behaving like journalists would be good. So would behaving like public policy leaders – something that U.S. Secretary of Education Arne Duncan seemed inept at when presented with NAEP results.

Duncan was first out the gate with false claims about what NAEP results proved in terms of education policy.

Quoted in Education Week, Duncan pointed out that the two states, Tennessee and Hawaii, and the District of Columbia that made strong gains in NAEP scores were also early adopters of his pet policy, Race to the Top.

“Tennessee, D.C. and Hawaii have done some really tough, hard work, and it’s showing some pretty remarkable dividends,” he boasted.

Education Week reporter Catherine Gewertz also noted, “The secretary also linked adoption of the common core – expectations his administration has trumpeted from coast to coast – to NAEP gains in eight states.”

Isn’t it convenient how Duncan seems to see in NAEP results confirmations of the biases he had before results were ever announced?

That tendency toward confirmation bias is the norm in regards to NAEP. Education Week’s veteran reporter Stephen Sawchuk warned of just this sort of thing before the NAEP scores were released.

NAEP results, Sawchuk explained, “are frequently pressed into service to bolster claims about the effect that policies [have had].” But, “while those assertions are compelling, provocative, and possibly even correct, they are also mostly speculative.”

The Huffington Post’s education reporter Joy Resmovits, noted a similar problem with NAEP commentary, quoting education researcher Mat Di Carlo, who said NAEP “is a measure of student performance, not school or policy effectiveness, and this valuable information is too often lost in a barrage of advocacy and unwarranted causal inference.”

Di Carlo has contended for years that NAEP results are used improperly to bolster arguments about policy effectiveness. “People on all ‘sides’ will interpret the results favorably no matter how they turn out.”

There may be some validity to Duncan’s claims that states that have implemented his policy preferences for teacher evaluations, charter school proliferation, and Common Core – all essential elements in Race to the Top – are producing gains in achievement. But for every state Duncan can identify as a “top performer” on the NAEP assessment, one can point to at least one state or more, as Rutgers University professor Bruce Baker did, that has excelled in NAEP score gains over the years when reform measures were not being implemented.

Does this mean that results on the NAEP are meaningless measures? Of course not.

NAEP. What Is It Good For?

One of the most significant revelations from NAEP is the continued disparity in academic achievement among the nation’s children based on their ethnicity and income level.

Results from NAEP have alerted policy makers and journalists to the “achievement gap” in America long before policy ideas like No Child Left Behind, which has too often been erroneously credited with revealing those gaps, came into being.

What this recent NAEP assessment revealed was very little if any progress on closing these gaps. As Te Huffington Post’s Resmovits reported, some black-white score gaps have narrowed since 1990, and the same for some Hispanic-white gaps. But since 2011, score gaps haven’t narrowed at all.

In its analysis, the Education Trust, an organization devoted to decreasing education disparities, cherry-picked a few states that  exhibited some progress in “gap closing” on NAEP results, but noted that D.C., who reformers identified as a star, actually saw its achievement gap widen.

States that the analysis cited for their “improvement” are truly a mixed bag whose policies can’t be easily generalized. And the analysis confined its scope to the past decade alone, when the record of advancement pales in comparison to the gains African-American and Hispanic students made on NAEP national assessments in years prior to No Child Left Behind.

“Lots of work remains,” the Education Trust concluded. But work on what?

Reform Harder

Among the education reform community, results from the NAEP assessment, despite any clear signs of significant progress in the nation’s education achievement, have led to the resounding conclusion that there is a deep and urgent need for – you guessed it – more reform.

What else would they have concluded? Had NAEP scores increased or decreased, either way, there would be justification in the reform movement to demand, “We must continue the reform.”

In the meantime, the urgent matter NAEP results really do reveal – that deep, longstanding racial and economic inequities still characterize America’s system of education – get barely any purchase at all.

It’s understandable that reform proponents like Michelle Rhee would make huge overreaches using NAEP results to advance their policy demands. Education reform, after all, is a big business, and the NAEP results appear to provide reform merchants with something to spin for marketing their cause.

But for journalists and policy leaders to parrot these specious claims is inexcusable. The American public deserves better.