New review explains CREDO charter school research flaws,
raises concerns about misunderstandings of effect sizes
Contact:
William J. Mathis, (802) 383-0058, wmathis@sover.net
Andrew Maul, (805) 893-7770, amaul@education.ucsb.edu
URL for this press release: http://tinyurl.com/mbse6m7
BOULDER, CO (April 27, 2015) – A recent report contends charter schools generally helped students increase reading and math scores and that urban charters had an even stronger positive effect. But a new review released today questions the strong reliance that has been placed on this and similar reports.
Andrew Maul reviewed Urban Charter School Study Report on 41 Regions 2015 for the Think Twice think tank review project. The review is published by the National Education Policy Center, housed at the University of Colorado Boulder School of Education. Maul, an assistant professor in the Graduate School of Education at the University of California-Santa Barbara, focuses his research on measurement theory, validity, and research design.
Urban Charter School Study Report on 41 Regions 2015 was produced and published by the Center for Research on Education Outcomes (CREDO) at Stanford University’s Hoover Institution. It is a follow-up report to CREDO’s 2013 National Charter School Study.
The new report analyzes the differences in student performance at charter schools and traditional public schools in 41 urban areas in 22 states. Researchers sought to establish whether being in an urban charter school, as opposed to a non-urban one, had a different effect on reading and math scores, and if so, why. The report found a small positive effect of being in a charter school overall on both math and reading scores, and a slightly stronger effect in urban environments.
Maul’s review, however, explains “significant reasons to exercise caution.”
For its analysis, CREDO again used its own, unusual research technique that attempts to simulate a controlled experiment: constructing “virtual twins” for each charter student. The “twins” were derived by averaging the performance of up to seven other students, chosen to match the charter students by demographics, poverty and special education status, grade level, and a prior year’s standardized test score.
Maul points out that the technique isn’t adequately documented. He adds: “It remains unclear and puzzling why the researchers use this approach rather than the more accepted approach of propensity score matching.” The CREDO technique, he warns, might not adequately control for differences between families who select a charter school and those who do not.
CREDO also fails to justify choices such as the estimation of growth and the use of “days of learning” as a metric.
But regardless of concerns over methodology, Maul points out, “the actual effect sizes reported are very small, explaining well under a tenth of one percent of the variance in test scores.” The effect size reported, for example, may simply reflect the researchers’ exclusion of some lower-scoring students from their analysis.
“To call such an effect ‘substantial’ strains credulity,” Maul concludes. Overall, the report fails to provide compelling evidence that charter schools are more effective than traditional public schools, whether or not they are located in urban districts.
Find Andrew Maul’s review on the NEPC website at:
http://nepc.colorado.edu/thinktank/review-urban-charter-school
Find CREDO’s Urban Charter School Study Report on 41 Regions 2015 on the web at:
http://urbancharters.stanford.edu/index.php
The Think Twice think tank review project (http://thinktankreview.org) of the National Education Policy Center (NEPC) provides the public, policymakers, and the press with timely, academically sound reviews of selected publications. NEPC is housed at the University of Colorado Boulder School of Education. The Think Twice think tank review project is made possible in part by support provided by the Great Lakes Center for Education Research and Practice.
The mission of the National Education Policy Center is to produce and disseminate high-quality, peer-reviewed research to inform education policy discussions. We are guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence. For more information on the NEPC, please visit http://nepc.colorado.edu/.
This review is also found on the GLC website at http://www.greatlakescenter.org/