Mike Petrilli appears to have guzzled a jug of the Kool-Aid being served up nationwide by Jeb Bush and Matthew Ladner. He has now sent out an email blast two days in a row promoting his blog post on Fordham’s Flypaper blog and the EdNext blog, trumpeting the Florida reform mix and attacking NEPC’s skepticism of that advocacy.

The basic problem with the Bush/Ladner argument is that they have confused correlation with causation—but behind this basic mistake are a lot of documents:

1.      A Heritage Foundation report by Matthew Ladner, then associated with the Phoenix-based Goldwater Institute, and now leading the effort at Jeb Bush’s Foundation for Excellence in Education.

2.      An NEPC-published review of the Ladner Heritage Report by Columbia University professor Madhabi Chatterji.

3.      A presentation by Jeb Bush of the so-called “Florida Formula” (a similar presentation is available here).

4.      An NEPC-published review of that Bush presentation by William Mathis, Managing Director of NEPC.

Here’s what Mike Petrilli is now saying, “Skeptics, including the union-funded ‘National Education Policy Center’ at the University of Colorado, wonder if Florida’s grade-retention policies explain the impressive results. (Florida students can’t move onto the fourth-grade until they are reading proficiently.) Sunshine State kids are scoring better, goes the argument, because they are older. But Ladner dismisses that argument thoroughly in the pages of Education Next—something the NEPC refuses to acknowledge.”

Initially, it is important to stress that nobody associated with NEPC has argued that grade-retention policies explain impressive learning results. In fact, the research on this point is pretty clear: simply flunking a student does nothing to improve learning and, in fact, is highly correlated with a greater likelihood of later dropping out. What professor Chatterji demonstrated is that the grade retention policies likely lead to substantial increases in test scores due to the extra year in school prior to taking the test.

It should also be noted that Ladner’s contentions in the EdNext article are in response to an older challenge by Boston College professor Walt Haney, not to the Chatterji review of Ladner’s Heritage report. Moreover, the response in EdNext is, as a matter of research and causal inference, as weak as his work in the Heritage report itself.  In a nutshell, he contends that fourth grade test scores in Florida were increasing before the grade retention policy went into effect, ergo the retention policy didn’t cause increases.

This all takes us into the weeds, but the weeds are actually very interesting. Please do read the Heritage report and the Chatterji review (as well as the Bush/Mathis exchange) and make a decision for yourself about the strength of the Florida evidence.

As for the Ladner histogram that Petrilli includes in his own blog post, the trouble with resting one’s arguments on asking readers to confuse correlation with causation is that it then puts one in all sorts of untenable positions.  So, Mike, will you acknowledge that school funding levels are extraordinarily important, since I see Utah (low spending) doing poorly and New Jersey (high spending) doing very well?  And since Michigan and Arizona (two big charter/choice states) are doing poorly, I’m sure you’ll be demanding that these policies be scaled back pronto.

All of that is nonsense of course, and the Mike Petrilli that I know should be smart enough to recognize that. Any sensible causal inferences require much better data and a much stronger research design. The same is very much true of the Ladner/Bush advocacy.

Allow me to re-print here something I wrote to Matthew Ladner as part of an earlier email exchange:

The study you are marketing here, Matthew, is misleading, incomplete, and lacks the data or design to make the causal inferences you'd like to (i.e., "any benefits seen here are [due] to policies x, y, and z"). We can't tell if there are actual benefits, and if there are, we can't tell what caused them.

By way of example, what if FL [had], in 2001, adopted an absurd policy of making all students [wear] green on Tuesdays. That would then become part of the FL package of education reform.

I might think that the FL writing intervention I mentioned earlier is the true cause of any benefits seen.  Madhabi might point to the supports that accompanied the grade retention policy.  [A third person] might point to the class size reduction initiative. Matthew might point to the school choice and alternative teacher certification. [Hopefully, nobody would point to the green clothes reform.]

The truth might be: (a) there are not actual improvements (the current study is too weak to say whether or not there are), (b) there are improvements, and they're caused by a combination of all these things, (c) there are improvements, and they're caused by something none of us pointed to (perhaps the green shirts??), or (d) there are improvements, and they're caused by some of the things we're pointing to BUT some of the other things we're pointing to are actually harming students (just not enough harm to overcome the benefits of the other things).

In other words, when it comes to understanding the FL package of reforms, we are flying blind.

That doesn't however mean that we're flying blind more generally about best practices and how to improve schools. There is a great deal of [high]-quality research about the importance of  mentoring/induction for teachers, early childhood education, class size reduction at early grades, detracking (universally accelerated curriculum) and supports, and culturally responsive and otherwise engaging teaching that's connected to students' lives. Here in CO, I see many or all of these things in successful charter schools and other public and private schools -- and imagine the same is true in [other states]. No miraculous packages from FL are needed.

Finally, I have to wonder why Mike Petrilli’s blog post identifies NEPC as “union funded” (unions are among our funders, but not our largest) but mentions nothing in his post about the right-wing funders backing Fordham, Hoover (EdNext), or Bush’s Foundation for Excellence in Education? I can’t speak to how these other organizations run, but all of NEPC’s funders are completely hands-off, playing zero role in determining editorial content. While NEPC believes that it’s important to disclose funders, our reviews have always relied on substantive critiques instead of name-calling and innuendo. I imagine that Heritage’s funders are not particularly middle of the road, but readers will find no mention of funders in Prof. Chatterji’s review—just a clear explanation of why policy makers should turn elsewhere to understand what has happened in Florida.