Miller’s post notes two concerns:  An issue with using peer-reviewed research (which is the same issue addressed in his first post) and an issue with our having made policy recommendations.

I’ll reiterate, the peer review process is far from flawless but it’s a standard whose goal is to provide some amount of veracity and rigor (We noted this past week in Science Magazine when the same fake article about a cancer “cure” was easily published in 157 for-profit “peer reviewed” journals).  We used this standard as a way to reduce accusations of cherry-picking.  Yes, we or another set of researchers could undertake a broader review, perhaps encompassing every study that any person has self-published on the impact of TFA. But that would have included studies that have been through no sort of vetting at all – and included evaluations that were funded by TFA itself, which clearly introduces bias. Moreover, a review of every study would have been unnecessarily long.  (NEPC asked us to write a “Brief”.)

I told him that he reminded me of the meanest thing on God’s earth, an old coon dog barking up the wrong tree. Sketches of David Crockett p. 58. (New York).

But what Miller specifically proposes is that we should have just included two more studies of TFA – two studies that he indicates have findings in support of TFA. I’m not sure what sort of filter this would be, perhaps a shows-TFA-has-a-positive-impact filter? A call-up-TFA’s-VP-for-research-partnerships-and-ask-him-what-should-be-included filter? Yes, peer review isn’t perfect, but these alternatives seem far from objective – especially when the studies we reviewed in this brief did have at least some limited positive findings for TFA, which, as we state numerous times in the brief, is consistent with the rest of the literature on TFA.

I’m not the evil academic mastermind with some bias against TFA that Miller seems to think I am, but if I were, I would have cherry picked studies that were unequivocally negative in estimating the impact of TFA. The studies we reviewed are balanced – and they mirror the rest of the literature on TFA. Miller has not proposed a more “objective” procedure by which to select studies to review. But even if he did, the question here is simply whether our approach of focusing on peer-reviewed studies is reasonable and offers a useful contribution, to help us understand the impact of TFA. Does Miller deny this reasonableness and this contribution? If not, then I don’t see the point in his line of attack. Moreover, he and TFA or its funders could easily commission a review that includes these two studies. We are curious to know how including them would dramatically change the conclusions drawn from the years of research on TFA.

Instead of having treed their game, gentlemen will find themselves still barking up the wrong tree. Mr. Duncan of Ohio in the United States House of Representatives, July 7: Congressional Globe, p. 474, Appendix.

In voicing his strong concerns about peer-reviewed research, Miller talks about publication bias.  This sort of bias actually works against Miller’s argument.  The TFA studies that were peer reviewed have mixed findings – the types of studies that journal editors would not find “sexy” (to use his word).  These sorts of nuanced results are typical of serious research and serious researchers. But the two evaluations Miller keeps mentioning apparently have strong positive findings – those that would (if he’s correct) be more likely to be published in a peer reviewed journal.  So, the question is, why did not the researchers of these studies publish them in a peer reviewed journal?

I’ll admit to being confused about Miller’s second point. It’s a policy brief, and we include policy recommendations. He mentions a “dearth of ‘official’ knowledge” (peer reviewed research) that could inform policy making – what dearth is this?  On almost all major education policy issues, there is a solid research base that can and should inform policy making. As applied to TFA, this statement is particularly confusing. We note in our 2014 brief that there was little peer reviewed research on the impact of TFA since our 2010 brief was published; but there’s no dearth of peer reviewed research in general. In that 2010 brief we examined all the peer reviewed research, plus all the research TFA had posted on its website that touted its impact on student achievement. We would have used the same method this time, but TFA has now removed all claims of research-based impact from its website. Relatedly, in the TFA contract with Houston Independent School District, TFA “does not make and has not made any representation and warranty as to the fitness of any candidate presented by Teach For America” – see Page 11, Section IV.D of the contract with Houston. (See Sneak Preview: How Citizens Can Hold Teach For America Accountable)

The same reckless indifference which causes a puppy to bark up the wrong tree. Chemung (NY) Democrat, September 18. 1839

Miller concludes his post with a personal attack against my co-author. One of the things he says in this attack is that my co-author has a “demonstrable bias in the handling of evidence around Teach For America’s impact on student achievement.” There’s no question that Julian Vasquez Heilig has written a great deal on TFA, particularly since our 2010 brief. This writing largely arises out of his analyses of research evidence. That’s to be expected. I find Miller’s statement stunning because I cannot think of one credible researcher who would not say that the research on the impact of TFA isn’t mixed or would otherwise disagree with our summary of the research. They may not agree with the conclusions we draw or the recommendations we make, but there isn’t much room for interpretation in the actual summary of the evidence.

For all of Cloaking Inequity’s posts on TFA go here.