NEPC Review: Measuring and Understanding Education Advocacy (Brookings Institution, March 2015)
This report uses two methodological innovations to measure the impact of advocacy groups on education reform policy, Surveys with Placebo (SwP) and Critical Path Analysis (CPA). SwP is designed to measure the influence of advocacy groups and CPA to identify which tactics were successful in influencing reform. The report finds them to be effective methods for achieving these goals. It finds that coordination of advocacy groups strengthens their impact and that the perceived impact of advocacy groups tracks closely with policy outcomes. While we agree that the SwP and CPA may be useful in education policy research, these methods are more limited than the report acknowledges. Moreover, the research is a small case study of three states, with a low response rate for the SwP and CPA based on advocacy groups’ self-reported tactics. The report also fails to give sufficient information on the responses to the SwP and the selection of the advocacy groups studied to assess the usefulness of methods and validity of the report’s conclusions. Finally, there is not a strong connection between the evidence presented in the report and its conclusions. We therefore caution against adoption of the methods or reliance on the conclusions presented in this report without significant further research.