Reviewer of Milwaukee report raises key questions but praises contribution

Contact: Robert Bifulco, (315) 443-3114; (email) rbifulco@syr.edu
Kevin Welner, (303) 492-8370; (email) kevin.welner@gmail.com

TEMPE, Ariz. and BOULDER, Colo. (May 5, 2009) -- A report released last month on charter schools in Milwaukee found no significant reading achievement gains but small gains in math among a segment of students in charter schools, in comparison to students in conventional public schools. The report also explored whether competition or some other charter effect could be seen in nearby public schools, but found no such effect. A new review of the report raises validity questions but notes strengths in the methods used. When the results are considered together with the large body of research on charters schools, the conclusion that charter schools should not be expected to have large effects on achievement in urban schools is found by the reviewer to be reasonable.

The report "The Impact of Milwaukee Charter Schools on Student Achievement" was written by Stephane Lavertu and John Witte and published by the Brookings Institution. It was reviewed for the Think Tank Review Project by Robert Bifulco, Associate Professor of Public Administration at the Maxwell School of Syracuse University.

Probably the most noteworthy finding from the report is that, even though no significant benefit in reading scores was associated with charter attendance, the report did find a small but statistically significantly higher math test score improvement among charter school students compared with conventional public school students. Yet it also found that those gains were largely limited to earlier years of the charter school program and to students in their first year in a charter school.

In his review, Bifulco observes that the statistical methods used in the Brookings report have been used in many other charter school studies and have been endorsed by a panel of experts on school choice research. Despite their "widely recognized strengths," however, these same methods "have come under sharp criticism," he notes, and should probably not be used by themselves. For instance, Bifulco points out that the study's charter school sample excludes students who have attended only charter schools, who may be different than in important, unmeasured ways from those (included) students who transferred in or out of charters during their schooling years.

Bifulco points out that the study assumes that all types of transfers between schools--whether in the normal course of school promotion (as from elementary to middle school) or because of special circumstances (such as changing residence)--"have the same average effect on achievement gains." He notes that the math score benefit is largely isolated to that first, post-transfer year, and he questions whether all or part of this apparent benefit is in fact an artifact of this transfer assumption.

A further concern, acknowledged but not fully resolved by the Brookings authors, is that Wisconsin changed its achievement test midway through the study. Given these and other "threats to the validity of the estimates presented in this report, robustness checks and alternative estimates are important."

The study results, Bifulco concludes, "are most informative when considered in the context of the larger body of research on charter schools"-namely, that charters produce "at best modest effects on student performance." In that light, he says, the authors are wise to point out that charters offer "no silver bullet" in the school-reform arsenal.

Find Robert Bifulco's review on the web at:
http://epicpolicy.org/thinktank/review-impact-Milwaukee-charter

CONTACT:
Robert Bifulco
Syracuse University
(315) 443-3114
rbifulco@syr.edu

Kevin Welner, Professor and Director
Education and the Public Interest Center
University of Colorado at Boulder
(303) 492-8370
kevin.welner@gmail.com

About the Think Tank Review Project

The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the ASU Education Policy Research Unit (EPRU) and CU-Boulder's Education and the Public Interest Center (EPIC), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.

Kevin Welner, the project co-director, explains that the project is needed because, "despite their garnering of media attention and their influence with many policy makers, reports released by private think tanks vary tremendously in their quality. Many think tank reports are little more than ideological argumentation dressed up as research. Many others include flaws that would likely have been identified and addressed through the peer review process. We believe that the media, policy makers, and the public will greatly benefit from having qualified social scientists provide reviews of these documents in a timely fashion." He adds, "we don't consider our reviews to be the final word, nor is our goal to stop think tanks' contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate."

**********
###

The Education and the Public Interest Center (EPIC) at the University of Colorado at Boulder and the Education Policy Research Unit (EPRU) at Arizona State University collaborate to produce policy briefs and think tank reviews. Our goal is to promote well-informed democratic deliberation about education policy by providing academic as well as non-academic audiences with useful information and high quality analyses.

Visit EPIC and EPRU at http://www.educationanalysis.org/

EPIC and EPRU are members of the Education Policy Alliance
(http://educationpolicyalliance.org).

###
**********