Further examination of Milwaukee voucher program should be improved and clarified

Contact: Casey Cobb, (860) 486-0253; casey.cobb@uconn.edu
Kevin Welner, kevin.welner@gmail.com

TEMPE, Ariz. and BOULDER, Colo. (May 28, 2009) -- A recent report on the second year of a five-year evaluation of the Milwaukee school voucher program finds no achievement growth differences between voucher schools and the Milwaukee Public Schools over a one-year period. A new review of the report finds its design and methods sound, but its usefulness is limited by several factors, including a lack of clarity about the representativeness of the included voucher schools.

The MPCP Longitudinal Educational Growth Study Second Year Report was written for the University of Arkansas School Choice Demonstration Project by a team led by John Witte, as part of a five-year evaluation of the Milwaukee Parental Choice Program (MPCP).

The report was reviewed for the Think Tank Review Project by Casey Cobb of the University of Connecticut, an expert in policy analysis and social science research methods.

The Second Year Report's primary finding was that there were no overall statistically significant differences in achievement growth between a group of students in the voucher schools included in the study and a matched group of students in Milwaukee Public Schools (MPS).

Cobb compliments the report's authors for being "careful not to overstate the findings of a single-year study on achievement effects." At the same time, however, he points to some concerns regarding a gender-disaggregated analyses (urging the authors to explain their reasoning in future reports if they continue to study such differences) and with the idea that students' residence location is enough to overcome possible self-selection bias among voucher students and their families.

The most substantial concern Cobb raises, however, involves the sample of schools used for this evaluation -- and likely for future evaluation reports as part of this project. An act of the Wisconsin legislature requires schools accepting vouchered students to administer a nationally normed standardized test in reading, math, and science in grades 4, 8, and 10. These are the test scores that are analyzed in the new report. But the law doesn't require any particular exam, and different schools administer different tests. The sample of voucher schools included in the report reflects only those using one particular standardized test, the Wisconsin Knowledge and Concepts Examination (WKCE). As a result, Cobb points out in his review, the results "are only generalizable to MPCP schools that administer the WKCE." The key question, and one that a reader of the report cannot answer, is whether this sample differs in any meaningful way from those schools that use a different exam. For instance, might a given subgroup of schools, such as those run by the Milwaukee Catholic Archdiocese, be excluded from the study because they use a different test?

In the end, Cobb praises the report for its transparent writing and analysis and notes that, despite the concerns he raises, the methodology "adheres to rigorous principles of social science." However, because of questions such as those described above, this second-year report is unlikely to be very useful to policymakers. It is hoped these concerns can be addressed with changes and explanations included in the third-year report.

Find Casey Cobb's review on the web at:
http://epicpolicy.org/thinktank/review-MPCP-longitudinal

This is the third Think Tank Review in recent weeks to examine the School Choice Demonstration Project's Milwaukee reports. Previously, a review by Professor Clive Belfield found that a report about the voucher program's fiscal impact offers valuable insights but also questioned the report's findings fiscal benefits and noted that the report omits some important factors essential to a more complete analysis. And a review by Professor Gregory Camilli of a report about the voucher program's competitive effects raised a number of questions about the report's statistical methods, ultimately concluding that any positive effect of competition is very small, if it exists at all.

* Belfield's review can be found at:
http://epicpolicy.org/thinktank/review-fiscal-impact-Milwaukee.

* Camilli's review can be found at:
http://epicpolicy.org/thinktank/review-Effect-Milwaukee-Parental-Choice

CONTACT:
Casey Cobb, Associate Professor
Director, Center for Education Policy Analysis
University of Connecticut
(860) 486-0253
casey.cobb@uconn.edu

Kevin Welner
Director, Education and the Public Interest Center
University of Colorado at Boulder
kevin.welner@gmail.com

About the Think Tank Review Project

The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the ASU Education Policy Research Unit (EPRU) and CU-Boulder's Education and the Public Interest Center (EPIC), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.

Kevin Welner, the project co-director, explains that the project is needed because, "despite their garnering of media attention and their influence with many policy makers, reports released by private think tanks vary tremendously in their quality. Many think tank reports are little more than ideological argumentation dressed up as research. Many others include flaws that would likely have been identified and addressed through the peer review process. We believe that the media, policy makers, and the public will greatly benefit from having qualified social scientists provide reviews of these documents in a timely fashion." He adds, "we don't consider our reviews to be the final word, nor is our goal to stop think tanks' contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate."

**********
###

The Education and the Public Interest Center (EPIC) at the University of Colorado at Boulder and the Education Policy Research Unit (EPRU) at Arizona State University collaborate to produce policy briefs and think tank reviews. Our goal is to promote well-informed democratic deliberation about education policy by providing academic as well as non-academic audiences with useful information and high quality analyses.

Visit EPIC and EPRU at http://www.educationanalysis.org/

EPIC and EPRU are members of the Education Policy Alliance
(http://educationpolicyalliance.org).

###
**********