Has "reform" helped the students of New Orleans? Does the data show that school effectiveness has improved following Katrina? Is the conversion of the city's school district into a "portfolio" system a model for the rest of the nation?

If you listen to the reformsters, the answer is an unequivocal "Of course!" Not only that: anyone who questions the narrative of undeniable success in New Orleans is guilty, in the words of Campbell Brown, of "denigrat[ing] hard work & progress of LA teachers, parents & KIDS."

As if the constant droning about the "failure" of American education found at Brown's website isn't the same level of denigration. Way to elevate the conversation, Campbell...

The certainty of the reformy side in the rightness of their beliefs is, of course, a core feature of their movement. I will concede that there are those on the opposite side who rush to condemn any evidence that favors things like charter schools or merit pay or school reconstitution.

But the reformsters always seem to forget that the burden of proof is on them. That doesn't mean that every argument against reformy policies has to be rebutted beyond any doubt; it does mean, however, that reasonable critiques should be engaged beyond simple mockery.

Take Peter Cook, for example, striking back at Andrea Gabor's NY Times op-ed, which questioned the awesomeness of reform in NOLA:

While conceding that proficiency, high school graduation, and college entry rates have all risen in New Orleans over the past ten years (no small matters), Gabor attempts to diminish these accomplishments by claiming:

“But the New Orleans miracle is not all it seems. Louisiana state standards are among the         lowest in the nation.”

Apparently, Professor Gabor has somehow missed the extensive national media coverage [for example, in the New York Times, Politico, Washington Post, Talking Points Memo, and even Gawker] of the battle between State Superintendent John White and Governor Bobby Jindal over the Common Core State Standards, which Louisiana adopted in 2012.

In short, the contention that Louisiana state standards are among the lowest in the nation is, to borrow Gabor’s own words, “simply false.”

Um, no.

It's quite clear to those of us who study these things that Gabor was talking about how the Louisiana's tests compare to other states' in terms of their difficulty. And Gabor is right: when mapped against the NAEP, the national standardized test, Louisiana's definition of proficiency is, in fact, quite low. It's perfectly appropriate to point this out when presenting a critique of the research that purports to show NOLA's reforms have led to better student outcomes. Why, then, does Cook snidely dismiss Gabor's point?

The reliably reformy Jon Chait has put together a collection of research that supports the NOLA success narrative. I'll admit that some of it is well worth considering. Doug Harris, for example, is a well-regarded and highly competent economist, and the work of the Education Alliance for New Orleans should be seriously considered in any conversation about school reform in New Orleans.

But no one should unquestioningly jump to the conclusion that we need to replicate NOLA in cities around the country on the basis of a brief Harris wrote for EducationNext. First of all, even he admits it doesn't document his full methods (Harris very graciously replied to an email I sent asking him when the "long form" of his research will be out; he said sometime within the next few months, hopefully).

More importantly, and as Harris says in a surprisingly level-headed interview with Matt Barnum for The Seventy Four*, there is all the world of difference between acknowledging some outcome gains in NOLA and imposing some of the reforms that have been implemented there on the rest of the country:

Imagine a hypothetical school district that's an urban district that's struggling — I understand that “hypothetical school district” might not be the best way to think about this — but a school district that's struggling, does not have any charter schools or has a limited charter sector. And they come to you and they say, “We want to replicate the New Orleans reform. Should we do it?” What are the factors you would tell them to consider and how likely would you be to recommend the New Orleans reforms?

Let me answer that by adding one more thing to what I was saying earlier about the conditions of New Orleans. One of the conditions in New Orleans was it was a very low performing district. It was a disaster by almost any measure. When I first got here I thought some of that was just hyperbole because everyone kept saying it, but now I've seen more and more objective evidence of that. The FBI had an office within the school board because there was so much corruption to investigate that they needed to have offices within the school board offices. I just got my hands on a report after many years of trying that was the Council of Great City Schools evaluating the human resources management of the district. The litany of problems in the district was just appalling, the way it operated. The outcomes were also poor as a result. It was the second lowest performing district in the second lowest performing state in the country.

If you're in a district that already functions pretty well, maybe this is not necessarily a good idea. Here you had a district that was not functioning well by any measure. I think people tend to assume that all urban districts are terrible because the scores are low but it really is true that a lot of that is driven by the demographics and the socioeconomics of the districts. That's real. There are some very good schools in urban areas and there are some districts that probably do a pretty good job. You don't want to throw that out just because of a probable misinterpretation, if they're not taking into account where students are starting.

I would tell a district or somebody advocating it in a district to look very carefully at how the traditional district is operating before going down that path.

I would also tell them all the conditions, all the things they have to do to make it work. I think people tend to think, oh we just need more charter schools. But Detroit is a case study in how that goes wrong. Detroit has a lot of charter schools — I think it's second or third in market share — but it's terrible. The system has so little coordination. [emphasis mine]

That's a point that needs to be stressed repeatedly: New Orleans is merely one example of education "reform." It doesn't necessarily "prove" the reform agenda is unfailingly successful, any more than Ohio "proves" charter schools are always doomed to failure, or Newark "proves" that school reconstitution is always a train wreck.

Social science is a messy business, and results, even in tightly controlled randomized experiments, are hard to replicate. Which means that caution is always warranted when moving from research findings to policy formation -- especially when there is substantial evidence that contradicts the portrait of New Orleans as an unbridled success:

Dropout and Graduation

The RSD graduation rate is last in the state of Louisiana at 61.1% out of 72 school systems.[xiv]The RSD is still the biggest dropout and “push out”[xv] factory in the state, with many low performing students leaving school as early as the 7th and 8th grades.[xvi]

Advanced Placement

Even though many RSD schools tout themselves as college prep in the media and public discourse, only 5.5% of their students who take Advanced Placement courses in the RSD score high enough on the AP tests to get credit. RSD Advanced Placement test results are some of the lowest in the state.[xvii]

That's Julian Vasquez-Heilig**, another well-regarded education researcher. Even if there have been significant improvements in New Orleans, by many metrics the Recovery School District and the State of Louisiana are low performing systems.

Further, as this brief from the National Education Policy Center points out, the cost of these test score gains may be unacceptably high:

“Recently, the Education Research Alliance for New Orleans, an organization at Tulane University dedicated to objective analysis of the New Orleans reforms, shared preliminary results from a study that reported meaningfully higher test scores following the post-Katrina reforms, even after accounting for population changes,” said Professor Huriya Jabbar, who has studied the New Orleans reforms. “However, the authors also report that the gains were not equal across groups: white students gained more than black students from the reforms.

“Furthermore, researchers have not yet determined which features of the reforms were successful (e.g., autonomy, teacher labor market reforms, and increased resources, including the influx of private philanthropic funds), as well as the role of other citywide changes in housing and employment.”

Moreover, groups of students, parents, and community members remain skeptical of the reform movement and have raised concerns that the new school system remains inequitable. For example, students and parents have raised concerns with some charter schools that have been unresponsive to students and too harsh in their disciplinary policies. After years of complaints lodged by parents about the treatment of students with special needs in the charter system, including physical and emotional abuse and “counseling out,” the parties settled a lawsuit brought by the Southern Poverty Law Center, acknowledging these grievances and requiring independent monitoring and auditing of charter schools’ special education services. According to news reports, the decentralized, fragmented school system in New Orleans has also been particularly unprepared to serve the growing percentages of English Language Learners in the city.

Further, within the choice system family income exerts a strong influence. A recent study found that low-income families make schooling decisions differently than affluent families. Low-income families are much more constrained in their choices because of practical considerations such as after-school care and distance, and therefore measured academic outcomes play a smaller role their decisions.

Schools have also engaged in behaviors that constrain parents’ choices. A recent study revealed that school leaders in New Orleans, facing the combined pressure of recruiting more students and raising test scores, cream-skimmed the more affluent or high-achieving students through, for example, selective advertising and recruitment.

Who's right: NEPC and Vasquez Heilig, or Harris and ERA? The answer, of course, is that they both are -- but the question is flawed. It's not a matter of the data "proving" which policy course we must pursue; it's a matter of weighing the evidence within context to inform decisions about how to improve our education systems.

So what's the context of New Orleans?

Prior to Katrina, NOLA was one of the lowest-performing and most poorly managed school systems in the nation, suffering from the effects of segregation, inadequate funding, and chronic poverty. Then an unprecedented (at least in the modern Unites States) natural disaster displaced an enormous number of people, changing the demographics of the city:

Afterward, a large amount of public funds poured into the schools. The teacher workforce changed dramatically, as did the governance of the schools. Charter schools flourished, but arguably at the expense of special education students, and with great variation in their spending.

There appear to be some significant test score increases in New Orleans' schools -- although there continue to be questions as to whether or not researchers have adequately controlled for changes in the city's student population. However, the district still lags far behind the rest of the nation.

Further, while charterization was a significant change in New Orleans education policy, the expansion of charters was concurrent with increased spending on schools, large changes in social services spending and delivery, and, again, substantial changes in the demographics of the student and city population.

I think that's a fair reading of what's been happening in New Orleans -- and I think it isn't nearly enough evidence to charge into a wholesale dismantling of our urban school systems.If there is a lesson to be learned from New Orleans, it's that too many in the education policy arena are ready to jump on a study or two that confirm their ideological predilections without stopping to consider all of the ramifications of what they propose.

In other words: No one has come close to showing that New Orleans has "proved" anything about how to improve America's urban school districts.

So relax, reformsters: it's called The Big Easy for a reason.

God bless NOLA.

* Just to be clear: I'm not surprised that either Harris or Barnum is level-headed; rather, I'm surprised such a nuanced view showed up in The Seventy Four.

** So I see a lot of criticism out there of this brief. I won't defend it, but I will point out it shows Louisiana is a low performing state on the NAEP, which counts for something if the measure of success in NOLA is how the city does compared to the rest of the state. Also, I haven't read Lopez and Olson's NAEP study, so I'll reserve judgment on that until I do...

Hmm, looks like that hasn't been published. Which is the problem with this whole conversation, isn't it? I wish NPE had waited and given us some real depth, much as I wish EducationNext had given us the long-form results of Harris's work.