BOULDER, CO (May 26, 2020) – A recent report from EdChoice, working with Hanover Research, identifies and reviews studies that use Randomized Control Trials (RCTs) to determine student achievement or educational attainment outcomes of nine broad “education reform” areas.
Bruce Baker of Rutgers University reviewed Comparing Ed Reforms: Assessing the Experimental Research on Nine K-12 Education Reforms. He concludes that the report accomplishes what it set out to do, but he points to several issues that minimize its usefulness in guiding policy, practice, or research.
The report simply presents counts of studies with positive, negative, and neutral findings from RCT studies. RCTs are presented in the report as “gold standard” studies for determining effects of specific treatments on measured outcomes, and the approach has clear strengths. But Professor Baker explains that the approach also has notable limitations. In fact, the actual randomization in education studies generally applies to only part of natural experiment; the randomization operates in a largely non-random context. Similarly, the studies are not fully controlled.
Each of the report’s specific education-reform-area counts also has notable limitations. RCT studies of charter schools tend to be limited to specific contexts, models, programs and services. Private school voucher policies are similarly varied and difficult to classify as a single “treatment.” Moreover, as the report notes, studies categorized under the reform “open enrollment” are actually magnet school programs. In fact, the report found no RCT studies that could be squeezed into three of the reform categories. All six of the reform areas that were in fact tallied—from smaller class sizes and schools to pre-k programs to school choice—score well, with positives outweighing negatives by large margins. But that truly tells us very little, Professor Baker explains, and the study authors are in fact cautious in explaining their modest goals—merely counting the quantity and superficial results of research done in those six specific areas.
The main concern with this report, therefore, is that the casual reader will take the table presenting the tallies out of context and use it to argue that charter schools and vouchers for private schools are most important or worthwhile because they have been studied most and thus have the highest counts of positive effects. If, however, the report is not misused in such a way, it offers a limited contribution for readers wanting to get an initial feel for the RCT research in these areas.
Find the review, by Bruce D. Baker, at:
https://nepc.colorado.edu/thinktank/charter-research
Find Comparing Ed Reforms: Assessing the Experimental Research on Nine K-12 Education Reforms, written by Paul DiPerna, with reviews conducted by Hanover Research and published by EdChoice, at:
https://www.edchoice.org/wp-content/uploads/2020/04/comparing-ed-reforms.pdf