High-Production Values and Eye-Catching Charts and Graphs Can Never Replace Strong Methodology and Sound Research Practices

BOULDER, Colo. and TEMPE, Ariz. (February 15, 2010) -- State education agencies and local school districts are increasingly asked to make evidence-based decisions about school reform initiatives, often assuming that all evidentiary claims are the result of high-quality research.  Unfortunately, much of the evidence offered in policy debates is based on research reports that have bypassed the quality control mechanisms of academic research.

In an effort to help education policy makers separate the wheat from the chaff, expert third party reviews are provided by the Think Tank Review Project, a collaboration of the Education and Public Interest Center (EPIC) at the University of Colorado at Boulder and the Education Policy Research Unit (EPRU) at Arizona State University. Each year the reports identified by experts as the worst of the worst are awarded a "Bunkum." The Think Tank Review Project today announced five "honorees" for 2009.

While the social science of the winning reports was sub-par, they typically had very high production values, glossy paper, multi-color printing, and artful layouts.  "Given the bibliographies, footnotes, charts and tables, policymakers or laypeople may be forgiven for thinking that these honoree reports are based on the highest quality research. We hope that our expert reviews have helped to correct that impression," said EPIC director Kevin Welner.

The 2009 Bunkum Award honorees:

The Time Machine Award
Weighted Student Formula Yearbook 2009
Authored by Lisa Snell, and published by the Reason Foundation.

In a truly breathtaking innovation, the report enters its time machine and attributes positive reform outcomes to policy changes that had not yet been implemented.

The Data Dodger Award
How New York City's Charter Schools Affect Achievement
Authored by Caroline M. Hoxby, Sonali Murarka & Jenny Kang, and published by the National Bureau of Economic Research.

New York City's charter schools might genuinely be improving student outcomes; however, this study -- because of the information it withheld and its methodological shortcomings -- does not and cannot resolve the issue.

The Misdirection Award: Keep Our Eyes Off What Works
Reroute the Preschool Juggernaut
Authored by Checker Finn and published by the Hoover Institution.

The report misdirects readers from a mountain of empirical, peer-reviewed and widely accepted evidence, and instead cherry-picks a few weak studies to critique proposals for universal preschool.

The Innovations in Promoting Alternative Certification Award
An Evaluation of Teachers Trained Through Different Routes to Certification: Final Report
Authored by Jill Constantine, Daniel Player, Tim Silva, Kristin Hallgren, Mary Grider & John Deke, and published by the U.S. Department of Education, Institute of Education Sciences.

The authors report 'no evidence' that traditionally trained teachers provided better student scores than alternatively trained teachers. The report does not bother to set forth caveats to the 'no evidence' conclusion, but there should, in fact, have been many, many caveats -- including small sample size, sampling methods, and a failure to distinguish the treatments. Also interesting: the study actually included many analyses that found traditionally trained teachers outperformed their alternative route counterparts. It's just that the authors chose not to fully report and acknowledge these findings in the report's conclusions.

The Annual Friedman Foundation Johnny One-Note Award
Multiple Reports by the Friedman Foundation
Multiple authors, all published by the Friedman Foundation.

The Friedman Foundation has, over the past three years, cloned the same study on the cost of drop-outs in at least seven states, a tax credit voucher report in at least six states, and opinion polls on school choice in 15 states. Amazingly, all these reports lead to the same conclusion: vouchers and other forms of school choice will save money and improve student outcomes.  The basic technique used by Friedman researchers is to take the same report, change the name of the state, plug in some state-specific data, vary the title a bit, and come up with the predetermined conclusion.

This year's honorees were selected following expert third-party reviews of research reports published by think tanks and other research organizations.  Reports reviewed by the Think Tank Review Project are carefully selected.  Every day the web sites of prominent think tanks are visited to identify new research publications for possible review.  If a report is deemed of sufficient importance, it is then assigned for review to an independent scholar with expertise in the area of inquiry.

A complete analysis of this year's Bunkum Award winners can be found at: http://epicpolicy.org/think-tank/bunkum-awards.

About the Bunkum Awards

The term 'bunkum,' meaning essentially 'nonsense,' came about because of a long-winded and pointless speech given in 1820 on the House floor by Congressman Felix Walker of Buncombe County, North Carolina. The Bunkum Awards help to highlight nonsensical, confusing, and disingenuous reports produced by education think tanks.

CONTACT:
Nikki Rashada McCord
Associate Director
Education and the Public Interest Center (EPIC)
University of Colorado at Boulder
(303) 735-5290
Nikki.McCord@colorado.edu

**********
###