Effects are modest and presentation of results incomplete, reviewer concludes

Contact: Martin Carnoy, (650) 856-7722; carnoy@stanford.edu
Kevin Welner, kevin.welner@gmail.com

TEMPE, Ariz. and BOULDER, Colo. (May 21, 2009) -- A recent evaluation report on the District of Columbia's voucher program finds that participants saw a statistically significant improvement in reading scores, but not math, compared with a control group of students who didn't get vouchers. A new review of the report, however, points out that any such effect is modest and provides little support for vouchers as a solution to the District of Columbia's educational problems.

The report is titled Evaluation of the DC Opportunity Scholarship Program: Impacts After Three Years. Its lead author is Patrick Wolf and it was co-published by the University of Arkansas' School Choice Demonstration Project and by the Institute of Educational Sciences at the National Center for Educational Evaluation and Regional Assistance. It was reviewed for the Think Tank Review Project by Professor Martin Carnoy of Stanford University, an economist and expert in voucher research.

The report and Carnoy's review come in the midst of a heated discussion about whether to continue the program. President Obama generally opposes the program but has agreed to maintain funding for students already participating -- those who received vouchers last year.

The D.C. program offers vouchers to low-income students in public schools, helping them attend one of about 60 participating private schools. The study analyzes reading and math achievement test scores of randomly selected applicants who were offered a $7,500 voucher during the first two years of the program, comparing these students with a control group of other applicants who were not offered vouchers.

The study concluded that, after three years, students who used vouchers scored 5.27 points, or one-sixth of a standard deviation, higher than those who stayed in public schools on achievement tests in reading -- a statistically significant difference -- and less than one point higher in math, which was not statistically significant. However, Carnoy points out that analyses of subgroups of these students show that any positive voucher effect on reading was concentrated among students with higher initial reading scores, middle school students, and females. Moreover, while Congress created the voucher program primarily to benefit students attending D.C. schools designated as "needing improvement" -- the study found that the voucher effect was focused on students who had not attended those schools.

According to the evaluation report, parents of voucher students felt their children were safer and that they were significantly more satisfied with their children's schools. Among students themselves, however, there was no significant difference in school satisfaction or perceived safety.

There are two sets of findings to be gleaned from the report. One is what the report itself emphasizes, which Carnoy summarizes as follows: "Sending low-income students to existing, predominantly religious (and even predominantly Catholic), small (average size, 265 students) private schools with small class sizes (average student-teacher ratio, 10.3 students) can modestly increase these students' achievement (in reading but not mathematics) and results in greater parent satisfaction with their children's school."

Yet, Carnoy observes, embedded in the report is another set of findings, reported but not emphasized by the authors. They include:
• Capacity for enrolling new voucher students has been shrinking dramatically, apparently because private schools have too few available seats.
• Voucher recipients vary widely in their usage of the program.
• Substantial majorities of both voucher recipients and the non-recipients (controls) switched schools at least once over the three-year study.
• Positive voucher effects appear to have been concentrated among students who already were doing better in school.

In fact, closer inspection of the results, Carnoy writes, suggests that the results showing a positive effect from vouchers are primarily due to the first cohort of students enrolled in the program. By the second year, available private school seating was reduced. And, due to research design limitations, the first cohort did not include elementary-level students -- a group that, in the second cohort, did not appear to have benefited. Carnoy faults the study for not offering any detailed analysis of differences among the two cohorts of students enrolled in the program, which might shed light on factors other than vouchers influencing the results.

Carnoy concludes that the report could have done far better in analyzing the results of the experiment by presenting them in a more nuanced fashion that focused on the possibility of varied effects with different populations and in different contexts and discussed the limitations of the results for more generalized large-scale applications.

Find Martin Carnoy's review on the web at:
http://epicpolicy.org/thinktank/review-evaluation-DC-opportunity

CONTACT:
Martin Carnoy, Professor
Stanford University
(650) 856-7722
carnoy@stanford.edu

Kevin Welner, Professor and Director
Education and the Public Interest Center
University of Colorado at Boulder
kevin.welner@gmail.com

About the Think Tank Review Project

The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the ASU Education Policy Research Unit (EPRU) and CU-Boulder's Education and the Public Interest Center (EPIC), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.

Kevin Welner, the project co-director, explains that the project is needed because, "despite their garnering of media attention and their influence with many policy makers, reports released by private think tanks vary tremendously in their quality. Many think tank reports are little more than ideological argumentation dressed up as research. Many others include flaws that would likely have been identified and addressed through the peer review process. We believe that the media, policy makers, and the public will greatly benefit from having qualified social scientists provide reviews of these documents in a timely fashion." He adds, "we don't consider our reviews to be the final word, nor is our goal to stop think tanks' contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate."

**********
###

The Education and the Public Interest Center (EPIC) at the University of Colorado at Boulder and the Education Policy Research Unit (EPRU) at Arizona State University collaborate to produce policy briefs and think tank reviews. Our goal is to promote well-informed democratic deliberation about education policy by providing academic as well as non-academic audiences with useful information and high quality analyses.

Visit EPIC and EPRU at http://www.educationanalysis.org/

EPIC and EPRU are members of the Education Policy Alliance
(http://educationpolicyalliance.org).

###
**********