In 2009, I assumed that President Obama appointed Arne Duncan to split the differences between data-driven and traditional reformers. I sure hoped the President did not think that a non-educator who became the CEO of Chicago schools at the age of 37 would have had the opportunity to learn how schools actually work. I hoped that Duncan understood that the briefings he received in that position were based on numbers that often said little about teaching and learning.

Duncan served at the end of a twenty year experiment in "reform." I doubt that he had much direct knowledge about the reasons why standardized metrics are so flawed, but Duncan had access to the Consortium of Chicago School Research. Surely he could have been influenced by one of the greatest research organizations in education or any other field.

So, the latest study by the Consortium on Chicago School Research (download here) could play an essential role in informing a dialogue between Duncan, and practitioners and social scientists. The Consortium analyzed the results of "reform" over twenty years in the city that pioneered data-driven accountability. It found Chicago's publicly reported data, which supposedly showed tremendous progress in elementary math and reading tests, was inaccurate. In fact, a generation of reforms only produced "incremental gains in math and almost no growth in reading." The Consortium documented discrepancies that are due to myriad issues with publicaly reported data ... that make year-to-year comparisons nearly impossible."

Researchers found improvements in overall school organization, although they "did not translate into better overall instructional quality in classrooms." Era One, 1988 to 1995, actually got a bad rap, despite showing modest improvements in both elementary and high school. The greatest increases in test scores occurred during Era Two, under Paul Vallas, but the Consortium explains persuasively why they did not signify "real" increases in learning.

Ironically, Duncan had his era's successes backwards, which raises the question of how many of the statistics that staff used to brief him were backwards. Contrary to his claims, it was high schools, not elementary schools that improved more during his term, with some of their gains being large. In other words, Duncan's numbers were misleading about both the failure of his policies and their successes. Duncan's prescriptions were not as successful in the real world as he thought, but during his term, as in the 1990s, many educators were doing better than he realized.

On the other hand, Duncan's main targets were schools that failed the most disadvantaged students. In contrast to his claims, equity declined under his term, as did progress in closing the achievement gap or improving the most challenging schools. Also, the saddest results occurred under Duncan, with students reporting declines in relationships with teachers and in a trusting environment.

Of course, these results aren't about Duncan or about teachers. The Consortium used Chicago as a huge laboratory that evaluated whether it makes sense to use test score growth to fire educators and drive policy. Combine the Consortium's findings with those of the National Academies of Sciences, and other scientific explanations why standardized test-driven "reform" has failed, and it is hard to explain how the unintended effects of Duncan's approaches are worth their risks. If there are two factors that transcend all others in improving schools it is teaching children to read for comprehension and building trusting relationships, and Duncan did not do either.

I doubt that the Secretary of Education has had the time to read, reread, and wrestle with the Consortium's new findings. Were I in his shoes, I would use the research as an opportunity to revisit pivotal decisions based on bad information. I would want to know when, how, and why my staff briefed me with untrustworthy numbers. I would sure want to understand the mistakes made in the city that I know best before imposing those unsuccessful policies on the rest of the nation.

I am sure Duncan knows that the Consortium has long argued that it is impossible to turn around troubled schools without trusting relationships. Since the Secretary can only have a limited personal contact with the nation's diverse schools, the question for Duncan is who and what does he trust, and mistrust, as he seeks evidence about relationships in schools. Should Duncan continue to trust the accountability hawks who surround him, while mistrusting the social science that traditional reformers find persuasive?

I have often exasperated my friends by trying to give Duncan, and the President, the benefit of the doubt. I have often sounded like a proverbial peasant before the French Revolution who would rationalize, "If only the King knew ..."

At least I was not naïve enough to believe that Duncan could craft a workable reauthorization of No Child Left Behind. ... But lo and behold a deal was crafted by Senators Tom Harkin and Wayne Enzi. All that the new NCLB compromise would cost Duncan would be his claim that the federal government should force states and localities to use standardized tests to fire teachers. Sure, the teacher bashers would complain, and Duncan would have to give them some rhetorical support, but this sort of win win opportunity should be a no-brainer for both Duncan and teachers.

I would think that anyone, liberal or conservative, Democrat or Republican, regardless of their educational policy, would have some trepidation about the federal government telling local districts how they should evaluate their employees. Keeping with my desire (illusion?) that we could devise a "big tent" coalition for reform, I will gladly acknowledge a federal role in promoting teacher quality efforts for poor schools. It would be dumbfounding, however, if Duncan stuck with his position that the federal government, through NCLB, must force states to adopt dangerous and untested evaluation systems that are "autopsies" of student work.

And that gets us back to the Consortium's key finding, "the publicly reported statistics used to hold schools and districts accountable for making academic progress are not accurate measures of progress." (Emphasis by the Consortium) The challenge of making assessments valid for schools and districts is far easier than using them for high stakes decisions for individuals. I cannot understand why Duncan would thus take the risk of incentivizing educational malpractice due to teaching to primitive high-stakes assessments by imposing not-ready-for-prime-time evaluations on the entire nation.

Duncan often says we must act on "imperfect" information. After a generation of failed experiments, perhaps Duncan now needs to go on record about how he parses imperfect evidence. Duncan loves to praise "great teachers," as he rejects our accounts of the unintended damage done by high-stakes testing. Duncan, personally, shows no public animosity towards teachers or our representatives, but neither does he show any skepticism about advisors who scapegoat us, as they double down on their one-size-fits-all gambles.

So, perhaps we should turn the question around and ask Duncan who he mistrusts more. Why does he want to fire the players without first questioning his own flawed playbook? Why is he more skeptical of the professional judgments of educators and social scientists than the theories of non-educators? Why does he doubt us more than he doubts the statistical engineering that has such a long history of failure? For more than a generation, his "imperfect" metrics have been saying that up is flat, and flat is up. Why do Duncan and Obama continue to demonstrate more respect for number crunchers who can't estimate straight than he does educators?

What do you think? Am I being naïve in hoping that the Secretary of Education's reading and discussion of research could influence his policies? If Duncan would back off from his support of standardized testing, would his other policies be salvageable? If so, how could we persuade him to reconsider that issue?