Review concludes that positive test-score effects of voucher competition on Milwaukee public schools are very small at best

Contact: Gregory Camilli, (732) 932-7496; camilli@rutgers.edu

TEMPE, Ariz. and BOULDER, Colo. (May 18, 2009) -- A recent report contends that competition from Milwaukee's private school voucher program for low-income families has benefited Milwaukee public schools. A new review of that report raises a number of questions about its statistical methods and concludes that any positive effect of competition is very small, if it exists at all.

The report is The Effect of Milwaukee's Parental Choice Program on Student Achievement in Milwaukee Public Schools. It was written by Jay P. Greene and Ryan H. Marsh for the School Choice Demonstration Project at the University of Arkansas. It was reviewed for the Think Tank Review Project by Gregory Camilli of Rutgers University, an expert in the use of statistics and measurement in social science research.

Greene and Marsh used a simple ("ordinary least squares") linear regression model to examine the performance on standardized achievement tests of students in the Milwaukee public schools. They had initially proposed a novel method, geocoding, to gauge whether the relative proximity public schools and voucher schools had a demonstrable effect on public school performance. However, their analyses using geocoded indicators showed no statistically significant competition effects. The authors instead focused their analysis on the general effect of voucher schools in the district as a whole.

Camilli points out that the largest competition benefit emphasized in the report was obtained with no controls, but also that these benefits were small from a practical perspective. Moreover, when statistical controls are used, no statistically or practically significant competition benefits were found. Those controls essentially account for whether there is in fact private school competition for a public school student (that is, the effects of grade and year are distinguished from the effect of competition).

Regarding the results of this study, Camilli observes that ultimately, "A great deal of faith is thereby required in standard linear regression without controls as well as substantial forbearance for the puzzling finding that competition effects were unrelated to voucher-school proximity."

Despite the questions Camilli raises about the report, he commends it for clearly and thoroughly presenting its findings and for presenting its methods and statistical models clearly.

He concludes that before the Greene and Marsh study can be used effectively to chart a course for expanding or reducing Milwaukee's voucher program, they will need to justify their use of uncontrolled estimates. He also recommends a more comprehensive approach to statistical modeling, particularly a multilevel approach in which students are nested within schools. And he points out that if the report had annualized results instead of announcing total gains over the seven-year span of the study, it would be "clear they are very modest in absolute size." He also urges the authors to make their data set freely available for secondary analyses by other researchers.

Find Gregory Camilli's review on the web at:
http://epicpolicy.org/thinktank/review-Effect-Milwaukee-Parental-Choice

CONTACT:
Gregory Camilli, Professor
Rutgers University
(732) 932-7496
camilli@rutgers.edu

About the Think Tank Review Project

The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the ASU Education Policy Research Unit (EPRU) and CU-Boulder's Education and the Public Interest Center (EPIC), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.

Kevin Welner, the project co-director, explains that the project is needed because, "despite their garnering of media attention and their influence with many policy makers, reports released by private think tanks vary tremendously in their quality. Many think tank reports are little more than ideological argumentation dressed up as research. Many others include flaws that would likely have been identified and addressed through the peer review process. We believe that the media, policy makers, and the public will greatly benefit from having qualified social scientists provide reviews of these documents in a timely fashion." He adds, "we don't consider our reviews to be the final word, nor is our goal to stop think tanks' contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate."

**********
###

The Education and the Public Interest Center (EPIC) at the University of Colorado at Boulder and the Education Policy Research Unit (EPRU) at Arizona State University collaborate to produce policy briefs and think tank reviews. Our goal is to promote well-informed democratic deliberation about education policy by providing academic as well as non-academic audiences with useful information and high quality analyses.

Visit EPIC and EPRU at http://www.educationanalysis.org/

EPIC and EPRU are members of the Education Policy Alliance
(http://educationpolicyalliance.org).

###
**********