This marks our eighth year of handing out the Bunkum Award, recognizing the lowlights in educational research over the past year. As long as the bunk keeps flowing, the awards will keep coming. It’s the least we can do. This year’s deserving awardees join a pantheon of divine purveyors of weak data, shoddy analyses, and overblown recommendations from years past. Congratulations, we guess—to whatever extent congratulations are due.
Watch the 2013 Bunkum Award Ceremony:
The “Do You Believe in Miracles?” Award goes to the Public Agenda Foundation for Failure is Not an Option: How Principals, Teachers, Students and Parents from Ohio’s High-Achieving, High-Poverty Schools Explain Their Success
A particularly egregious disservice is done by reports designed to convince readers that investment in disadvantaged communities can be ignored. In this increasingly common mythology, students’ substandard outcomes are blamed on teachers and schools that don’t follow the miracle-laden path of exceptional schools.
Early in 2013, we sent a report of this genre out for review. The authors of this report, from Public Agenda, identified nine Ohio schools where “failure is not an option.” The report’s basic claim was that certain school-based policies and programs can by themselves overcome the impact of poverty on student performance. Among the earth-shaking recommendations were: “Engage teachers,” “Leverage a great reputation,” “Be careful about burnout,” and “Celebrate success.”
While these seem like good practices and have indeed been pursued since the time when the report’s authors were in kindergarten, it’s hard to see how they will lead to miracles. Miracles are hard to come by and even harder to sustain. In fact, notwithstanding the report’s title, four of the nine selected schools had poverty rates at the state average and thus not particularly high-poverty schools.
While it may be easy to laugh at the idea that the recommended approaches will somehow overcome the effects of unemployment, bad health care, sub-standard living conditions and the like, it is also an outrageous neglect of the fundamental social needs and problems of neighborhoods, families and children. The truth that these reports hide is that school failure will almost always prevail in a society that will not invest in disadvantaged communities and the families who live there.
The “We’re Pretty Sure We Could Have Done More with $45 Million” Award goes to the Gates Foundation and its Measures of Effective Teaching Project.
We think it important to recognize whenever so little is produced at such great cost. The MET researchers gathered a huge data base reporting on thousands of teachers in six cities. Part of the study’s purpose was to address teacher evaluation methods using randomly assigned students. Unfortunately, the students did not remain randomly assigned and some teachers and students did not even participate. This had deleterious effects on the study–limitations that somehow got overlooked in the infinite retelling and exaggeration of the findings.
When the MET researchers studied the separate and combined effects of teacher observations, value-added test scores, and student surveys, they found correlations so weak that no common attribute or characteristic of teacher-quality could be found. Even with 45 million dollars and a crackerjack team of researchers, they could not define an “effective teacher.” In fact, none of the three types of performance measures captured much of the variation in teachers’ impacts on conceptually demanding tests. But that didn’t stop the Gates folks, in a reprise from their 2011 Bunkum-winning ways, from announcing that they’d found a way to measure effective teaching nor did it deter the federal government from strong-arming states into adoption of policies tying teacher evaluation to measures of students’ growth.
The “It’s Just Not Fair to Expect PowerPoints to Be Based on Evidence” Award goes to Elliot Smalley of Tennessee’s Achievement School District and Patrick Dobard of the Louisiana Recovery School District.
For years, Jeb Bush’s “Florida Miracle” has been unmatched as the most bountiful wellspring of misleading education reform information. But Florida and Jeb have now been overtaken by the Louisiana Recovery School District, which serves the nation as the premier incubator of spurious claims about education reform and, in particular, the performance of “recovery school districts,” take-overs, portfolio districts, and charter schools.
Superintendent Patrick Dobard has taken his suitcase of PowerPoints on the road, touting the Recovery School District’s performance. Nothing has stood in his way. Not the dramatic post-Katrina change in student composition. Not the manipulation of student achievement standards in ways that inflate performance outcomes. Not the unique influx of major funds from foundations, the federal government and billionaires. And not the unaccounted-for effects of a plethora of other relevant factors.
But Dobard is not alone. Elliot Smalley, the chief of staff for the Achievement School District in Memphis, flexed his PowerPoints to show his school district’s “Level 5 Growth.” This certainly sounds impressive—substantially more impressive, for instance, than, say, Level 3 Growth. But this growth scale is unfortunately not explained in the PowerPoint itself. What we can say is that a particular school picked by Smalley to demonstrate the district’s positive reform effects may not have been a good choice, since the overall reading and math scores at that school went down. Picky researchers might also argue that more than seven schools should be studied for more than two years before shouting “Hosannah!”
As was the case with the Florida Miracle, the Bunkum Award here is not for the policy itself—serious researchers are very interested in understanding the reform processes and outcomes in these places. Rather, the Bunkum is found in the slick sales jobs being perpetrated with only a veneer of evidence and little substance backing the claims made.
Back in the old days, when people thought they had a good idea, they would go through the trouble of carefully explaining the notion, pointing to evidence that it worked to accomplish desired goals, demonstrating that it was cost effective, and even applying the scientific method! But that was then, and this is now. And some of the coolest kids have apparently decided to take a bit of a shortcut: They simply announce that all their ideas are fantastic, and then decorate them in a way that suggests an evidence-based judgment. Witness the fact that we are now swimming in an ocean of report cards and grades whereby A’s are reserved for those who adopt the unproven ideas of the cool kids. Those who resist adopting these unproven ideas incur the wrath of the F-grade.
It’s apparently quite a fun little game. The challenge is to create a grading system that reflects the unsubstantiated policy biases of the rater while getting as many people as possible to believe that it’s legitimately based on social science. The author of the rating scheme that dupes the most policy makers wins!
This year, there are triple-winners of the “Look Mom! I gave myself an ‘A’ on my report card!” award, including our Grand Prize Winner for 2013!
Second Runner-up goes to StudentsFirst, which came up with 24 measures based on the organization’s advocacy for school choice, test-based accountability and governance changes. Unfortunately, the think tank’s “State Policy Report Card” never quite gets around to justifying these measures with research evidence linking them to desired student outcomes. Apparently, they are grounded in revealed truth unseen or unseeable to lesser mortals. Evidence, though, has never been a requirement for these report card grades. And naturally the award-winning states embrace the raters’ subjective values. In a delightful expose, our reviewers demonstrated that the 50 states received dramatically different grades from a variety of recent report cards: a given state often received a grade of “A” on one group’s list and an “F” on another group’s list.
First Runner-up goes to the American Legislative Exchange Council (ALEC), which almost took the top honors as the most shameful of a bad lot. What makes the ALEC report card particularly laughable is the Emperor’s-clothes claim that its grades are “research-based.” Yes, evidence-based or research-based report card grades would be most welcome, but all ALEC offers is a compilation of cherry-picked contentions from other advocacy think tanks. Thus, what is put forth as scientifically based school choice research is actually selective quotations from school-choice advocacy organizations such as Fordham, Friedman and the Alliance for School Choice. Similarly, the report’s claims about the benefits of alternate teacher certification in attracting higher quality candidates are based on only one paper showing higher value-added scores. Unfortunately, that paper was unpublished—and the report’s reference section led to a dead link.
This year’s Grand Prize Winner is the Brookings Institution and its Brown Center on Education Policy. Brookings has worked hard over the years to build a reputation for sound policy work. But, at least in terms of its education work, it is well on its way to trashing that standing with an onslaught of publications such as their breathtakingly fatuous choice and competition rating scale that can best be described as political drivel. It is based on 13 indicators that favor a deregulated, scaled-up school choice system, and the indicators are devoid of any empirical foundation suggesting these attributes might produce better education.
Since the mere construction of this jaundiced and unsupported scale would leave us all feeling shortchanged, Brookings has also obliged its audience with an application of its index to provide an “evaluation” of New York City’s choice system. Where an informative literature review would conventionally be presented, the authors of this NYC report touchingly extoll the virtues of school choice. They then claim that “gains” in NYC were due to school choice while presenting absolutely nothing to support this causal claim. And, following from this claim and from their exquisite choice and competition rating scale, they offer the expected recommendations. They almost literally give themselves an “A.”
Seldom do we see such a confluence of self-assured hubris and unsupported assertions. It’s hard to find words that capture this spectacular display except to say, “Congratulations, Brookings! You just won the Bunkum’s Grand Prize for shoddiest educational research for 2013.”