Earlier review found data were misused, conclusions untenable

Contact:
William Mathis, NEPC
(802) 383-0058
William.Mathis@colorado.edu

BOULDER, CO (February 22, 2011) – Just two-and-a-half months ago, the National Education Policy Center released a Think Twice review of a report published by the Heritage Foundation called, “Closing the Racial Achievement Gap: Learning from Florida’s Reforms.” The review, authored by Teachers College, Columbia University Professor Madhabi Chatterji, identified and explained major flaws in the Heritage report.

However, Matthew Ladner, the co-author of that Heritage report, just re-issued his flawed findings from Florida, in the form of a Friedman Foundation report titled, “Lessons for Tennessee from Florida's Education Revolution.”

In both the Heritage report and the Friedman report, Ladner is able to reach his desired conclusion by:
• Cherry-picking achievement results. Dr. Ladner’s analyses all used only fourth grade reading results from NAEP. Fourth grade math and eighth grade reading and math results did not show as favorable results and were omitted from his analyses.
• Failing to account for flunking effects. The report ignores group differences resulting from the state’s mandatory grade retention policy for the weakest readers in grade 3. This policy-driven increase in grade retention rates spuriously inflates the average scores of grade 4 students on state and national assessments, making racial achievement gaps narrower.
• Asserting the results he cites are the result of his favored reform efforts (school choice, performance pay charters, tax credits, mandatory grade retention, virtual schools, alternative certification), even though neither the data nor the analyses used are even close to sufficient to make causal claims. Florida in fact has several additional school-reform initiatives, including summer interventions, an early reading initiative, and class size reduction that are not listed or considered as possible contributors.
• Ignoring important research on these topics. Almost all his references are to publications by advocacy think tanks or media reports.

Dr. Ladner has also written state-specific versions of this report for Arizona, Colorado, Indiana, New Mexico, Oklahoma, Utah and Wisconsin, as well as versions for the Hoover Institution and the Pacific Research Institute.

“It is both surprising and troubling that Dr. Ladner would re-issue essentially the same study after Professor Chatterji explained the serious problems,” said NEPC Director Kevin Welner. “It is also disappointing that the Friedman Foundation would condone such work, given the preface of each Friedman report, which extols the think tank’s ‘Commitment to Methods & Transparency.’” This commitment reads as follows:

"The Foundation for Educational Choice is committed to research that adheres to high scientific standards, and matters of methodology and transparency are taken seriously at all levels of our organization. We are dedicated to providing high-quality information in a transparent and efficient manner.

"All individuals have opinions, and many organizations (like our own) have specific missions or philosophical orientations. Scientific methods, if used correctly and followed closely in well-designed studies, should neutralize these opinions and orientations. Research rules and methods minimize bias. We believe rigorous procedural rules of science prevent a researcher’s motives, and an organization’s particular orientation, from pre-determining results.

"If research adheres to proper scientific and methodological standards, its findings can be relied upon no matter who has conducted it. If rules and methods are neither specified nor followed, then the biases of the researcher or an organization may become relevant, because a lack of rigor opens the door for those biases to affect the results.

"The author welcomes any and all questions related to methods and findings."

For the think tank review of Ladner’s Florida work, see
http://nepc.colorado.edu/sites/default/files/TTR-HeritageFla-Chatterji…

For the 2010 Bunkum awards, go to
http://nepc.colorado.edu/think-tank/bunkum-awards/2010

For the Tennessee adaptation of the Florida results, click on
http://www.edchoice.org/Research/Reports/Lessons-for-Tennessee-from-Flo…

For the 2009 Friedman Bunkum award,
http://nepc.colorado.edu/think-tank/bunkum-awards/2009

The Think Twice think tank review project (http://thinktankreview.org), a project of the National Education Policy Center, provides the public, policy makers, and the press with timely, academically sound, reviews of selected think tank publications. The project is made possible in part by the generous support of the Great Lakes Center for Education Research and Practice.

The mission of the National Education Policy Center is to produce and disseminate high-quality, peer-reviewed research to inform education policy discussions. We are guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence. For more information on NEPC, please visit http://nepc.colorado.edu/.