Once again, Matt Barnum points us to a study showing positive effects from a "no excuses" charter school. The paper, by Matthew Davis and Blake Heller, takes advantage of the fact that, for three years, the Noble charter school network on Chicago had more applicants than it had available seats, and therefore held a lottery. The research compared students who were and weren't offered a seat a Noble, and found those who were offered charter admission were somewhat more likely to attend and persist in college.

I've written quite a bit about the limitations of these charter "lottery" studies before: see here, here, and here. And one of the biggest issues I've consistently found is that the definition of the treatment -- here defined as acceptance into a "no excuses" charter school -- is quite fuzzy.

What exactly does "no excuses" actually mean? Davis and Heller cite five factors identified by Dobbie & Fryer in a 2013 paper: "frequent teacher feedback, data driven instruction, high-dosage tutoring, increased instructional time, and a relentless focus on academic achievement.”* But Dobbie & Fryer never describe how these factors vary when compared to public district schools; they only studied how they vary within a group of NYC charter schools. 

So we really don't know if these are the behaviors that lead to the effects found in the lottery studies when charters are compared to counterfactual public district schools. This is serious limitation, because it may well be that the five factors aren't the differences that really lead to these charter school outcome gains.



In other words: what if "no excuses" really means something other than what this research assumes it means?



Davis and Heller do cite another source to help define "no excuses" -- a source that Dobbie & Fryer, tellingly, also cite. A source that shows up time and again in the charter "lottery" studies to help explain what a "no excuses" charter school is: see here, here, here, here, here, and here, among others.

No Excuses, by Abigail & Stephan Thernstrom, is the touchstone for these researchers' descriptions of the charter schools they study. Unfortunately -- and I'm going to be charitable here -- the text is simply not worthy of citation in a serious journal article. 

I'd say the book's description of its qualitative methods was inadequate, but that would be incorrect: it doesn't describe its methods at all. For example: aside from trafficking in some highly questionable racial profiling, the book features two chapters where it describes "superb" "high-poverty public schools." (p. 43)

But we have no idea why the Thernstroms picked these schools or what methods they used to study them; the only clue is on page 43: "We chose these particular examples of fabulous education only because they came to our attention and we visited them."** We further have no idea how they came to the conclusion these schools differ in behaviors or practices from public district schools.

And yet this is a primary source many of the lottery studies use to define their treatment. When Davis and Heller assert the Noble schools got a 10 percentage point gain in college persistence because of the chain's "no excuses" pedagogy,*** they are echoing the Thernstroms' assertions about the value of increased instructional time and high expectations...

Except that's not all the Thernstroms have to say about what makes a "no excuses" school. In fact, No Excuses is quite blunt about how "fabulous" schools get the gains that they get. What follows is an extended excerpt from the book that cuts right to the heart of how the Thernstroms believe "no excuses" charter schools operate. Certainly, this passage isn't the entirety of the Thernstroms' argument; however, they are remarkably candid about the role of self-selection in these "fabulous" schools.

If quantitative researchers are willing to cite No Excuses, they should at the very least acknowledge this part of the Thernstroms' thesis.

[All emphases are mine.]

* * *

(From No Excuses by Abigail & Stephan Thernstrom. Simon & Schuster (2003), pp. 47-49.)

Schools That "Cream"?



Not all minority students in the inner city are alike, of course. Some are more academically promising than others. Some have parents who place a particularly high value on education. Do the schools we so admire cream? Are the schools getting unrepresentative academic results because their students are not representative of the neighborhoods from which they draw?



With one exception, none of the schools selects students on the basis of test scores, grades, or recommendations. It is true that, in theory, Rafe Asquith's kids are part of a "gifted" program, although the school's definition of "gifted" means he has, in fact, a mixture of academically talented and struggling youngsters. There are about ten such "gifted" classes at Hobart, and the others do not yield remotely comparable results. Some of these schools, like the Amistead Academy in New Haven and the KIPP academies, simply select their students by a lottery.



At almost all the schools we celebrate, though, there is an application process that tends to -- and is intended to -- discourage families unlikely to cooperate with the school. Indeed, one of the five pillars upon which the KIPP schools rest is "choice and commitment." Students, their parents, and the faculty of a KIPP academy "have made a choice to be at the school," the literature reads. "No one is assigned or forced to attend these schools." Choice entails commitment -- a commitment that not every parent and student is willing to make. And thus KIPP and other schools of choice are, in this minimal sense, self-selective. Families interested in KIPP must be willing to come in for initial interviews, and to sign on to a rigorous program that includes long days, a long year, and summer school, as well as a strict behavioral code.



Parents attracted to North Star come to an open house, where Norman Atkins, the cofounder, gives them a "very negative sell," as he describes it. The school is not for them, he says, if they want to take the family on vacation in July, if they don't like school uniforms, if two hours of homework a night seems excessive. If they still find North Star appealing, they watch a simulated school day on a Saturday and can pick up an application. Students are picked randomly from the applicant pool. Among the schools we refer to in the pages below, only the Fredrick Douglass Academy in Harlem looks at test scores and other indices of academic and personal strength in admitting students.



The commitment that KIPP demands is spelled out in contracts that set expectations and are signed by teachers, parents, and students. Students must agree, for instance, to "always work, think, and behave in the best way [they] know how and... do whatever it takes for [them] and [their] fellow students to learn." They must promise to attend school faithfully, do their homework, and "behave so as to protect the safety, interests, and rights of all individuals in the classroom." And they must understand that that are "responsible for [their] own behavior." The contract parents sign is similar, but includes provisions on helping their "child in the best way we know how" and making themselves available to their children and the school. They, too, must agree that the ultimate responsibility for the "behavior and actions" of their child lies with the family.



North Star parents are urged to sign a voluntary "Covenant" that includes such items as: "We will make certain that our child attends school every day, except in cases of illness or another legitimate reason," and "we will check our child's homework each night and provide quiet time and space for the work to be completed." At South Boston Harbor Academy (SBHA) and most of the other successful schools we visited, there were similar agreements, although many parents inevitably fail to keep their end of the bargain. In running SBHA, the biggest surprise, the principal told us, was the lack of parental support for the high standards; parents often took their children out of school for an appointment or family vacation or made excuses for inexcusable absences.



And yet, with almost no exceptions, the children who violate the terms, or whose parents do, are not expelled. Norman Atkins stated the problem well: "There is always," he said, "in good schools serving inner-city populations, a tension between rigor and rescue." North Star can stick to the letter of its law and toss a recalcitrant student out, but where will he or she go?



Nevertheless, the fact that these are schools of choice is not incidental to their success. "Most of what we do here, if it's not reinforced at home, becomes incredibly challenging," one North Star staff member said to a doctoral candidate writing a dissertation on the school. Others at North Star, KIPP, and elsewhere have echoed the sentiment. Families that choose a school have at least, on paper, committed themselves to supporting the academic program. And schools of choice gain leverage from the simple fact that students who come willingly can be told, you have another option: Return to a school where fights are common, kids talk in class and curse teachers, and no one does any homework. But, remember, since the students at these schools learn little, their future is dim.



"This school is all about choice," James Verrilli, cofounder of North Star, noted in his opening remarks to students on their first day a few years back. "See that back door? See any locks on it? Is this a prison? Am I forcing you to be here? ... If you cannot live by our rules, if you cannot adapt to this place, I can show you the back door." Norman Atkins, that same day, delivered much the same message: "We have 479 students, 479 students, who want to be sitting where you're sitting, who wish deep down that they had your seat. And you got it. So if you're not making the most of your experience here -- if you're not working to your absolute top potential, if you're not going home, running home to do your homework every night, if you're not respecting your teachers and each other with all your energy -- some of the other kids would like your spot."



That is precisely the message, of course, that no regular public school delivers. An urban principal whose school we visited told us that she was sending her own children to parochial schools. She wanted a school that could impose limits, with consequences for those who refused to abide by them. Why couldn't she do that, herself, in the school she runs? we asked. "I'm powerless," she answered. The students are stuck with the school and the school is stuck with them.

 

* * *

While I think this passage pretty much speaks for itself, let me add a few quick thoughts:



- One response to the "creaming" argument I often read from the economists who conduct these lottery studies is that the random assignment to treatment controls for unobserved variables. In other words: because the lotteries are random, we can assume that, on average, things like parental motivation are the same in both the group of students who are accepted into the charter schools, and the group of students who are not.



There are two problems with -- or perhaps more precisely, limitations on -- this argument. First, at least some of the students who "lost" the lottery do not enroll in the counterfactual public district schools. They might go to a parochial or private school, or go to a public school out of state, or be home schooled. It's reasonable to think at least some parents who are highly motivated and have greater means will choose these options for their children; to the extent that they do, there is a difference between the control and treatment groups. We can try to measure that difference with crude binary variables or error-prone test scores, but those are ultimately proxies for what we really care about.



[ADDING: To be clear, it appears this isn't an issue with Davis and Heller's study, because the students are tracked to college regardless of where they eventually went to high school. This is an issue in other lottery studies, however, especially if the outcomes measured are statewide accountability tests. Private school students don't take these tests, so they can't be included in the control group.]



The greater limitation, however, is one that is often lost in the discussion of these studies, and is directly related to the passage above: The estimated effects of charter schools in these studies can only be generalized to students who enter the lottery.



We really have no idea whether or not expanding charter schools will yield the same effects if the pool of applicants expands as well. The passage above suggests that the students and families who enter the lottery perceive themselves as quite different from the families who don't enter the lottery. The charter school officials and the Thernstroms seem to agree. Is the charter effect really little more than a peer effect? That would greatly limit the effects of charter school expansion.



- The Thernstroms make a big deal about the relatively low expulsion rate for "no excuses" charters. But we've got enough data now to know that's not really how the charter attrition game is played.

 

Charter schools don't have to expel students; they just have to hassle them with high suspension rates or constant disciplinary actions for minor infractions. Eventually, the kids will voluntarily transfer out.



- One rebuttal I often hear from charter supporters is that things are changing so quickly within the sector that the last piece of research which might call into question charter "success" is outdated. I will concede that KIPP, for example, seems to have had something of a change of heart in the last couple of years, and is backing away somewhat from strict disciplinary measures and high attrition rates.



But part of that is they've built a brand, and everyone now knows what it is. Parents send their kids to a KIPP school knowing what they'll get. It's not like a decade ago when most people weren't sure what a "no excuses" charter school was; parents in charter-heavy districts are much more likely to know what they're in for, which means the self-selection increasingly can take place before students enter the lottery.



In any case, there's at least some evidence that as KIPP has built up, it's effects have cooled off. Charter effects are not guaranteed to continue as populations expand.



- Notice how much Atkins relies on being able to claim he has a waitlist for students to get into North Star. It's worth remembering that the size of these lists is largely a myth. And as I've said before: if the waitlist is so long, why doesn't North Star backfill? And why spend money on advertising and marketing if you can't even enroll everyone who allegedly wants your product?



Waitlists are marketing tools. When you can claim a large waitlist, you can make families believe your school is special. And if schooling is, to at least some extent, a positional good, a waitlist sends the message that a school is offering something lots of people want, but only some people get. That's a big part of the appeal of charter schools -- don't let anyone tell you otherwise.

 

* * *

I've said it so many times I've run out of ways to say it: I'm not against charter schools per se. But it's well past time to start being honest about why certain charters get the gains that they do.



Noble students are somewhat more likely to go to college and persist.**** That's great -- but Noble has some other features that are important to consider before we declare it a model fit for replication. Its cohorts, for example, shed large numbers of students -- including those in Davis & Heller's study:

 

Noble has also engaged in some questionable disciplinary practices (to their credit, Davis and Heller note this), some questionable recruitment practices, and some questionable student retention practices. The leaders of Noble appear to believe they cannot maintain their schools with a unionized faculty. Why? What else is going on at Noble that has nothing to do with "frequent teacher feedback, data driven instruction, high-dosage tutoring, increased instructional time, and a relentless focus on academic achievement"?



Yet another charter lottery study isn't going to answer this critical question. We need to get past facile explanations of the "success" of a small number of charter schools and really start digging into why they get the (largely marginal) gains that they get. And if we are to believe the sources cited by research showing positive charter school effects, one of the factors leading to charter gains is student self-selection.

 

Easy to say; hard to define...

ADDING: I have to note that No Excuses has perhaps the most hacktastic chapter ever written on school funding. An excerpt:

Consider the recent historical record. Per pupil spending on America's elementary and secondary schools (in constant dollars) nearly doubled between 1970 and 2000. Our educational system is not performing twice as well as a result. In fact, on the basis of the evidence NAEP has collected about what American students know, it seems safe to say that the nation's schools, overall, are no better than they were three decades ago. Black students have come a considerable distance since 1970, but the rise of a large black middle class and the end of segregated schools in the South may account entirely for that change. (p. 153)

Hey, Bruce, got anything to say about this?





* We also don't know how charters are able to offer more instruction and individual tutoring. Are they collecting more resources through philanthropy? Are they free-riding on public school salaries? If so, it's unlikely they can continue to use those strategies if the sector expands.



** That quote is followed by an endnote citing Samuel Casey Carter's No Excuses, a 2000 book published by the right-wing Heritage Foundation. It's essentially a catalog of 21 schools chosen ostensibly because at least three-quarters of their students qualify for free lunch yet score above the 65th percentile on a national standardized test. But no sampling methodologies are documented, no qualitative validity methods are described, and the tests are not consistent across all schools.



Again: this is not a source that should be cited in a serious, peer-reviewed journal article.



*** First: I'm going to skip a discussion of whether this is a practically significant gain or not. I'll only suggest this may be a point open for debate.



Second: yes, Davis and Heller do, in the last paragraph of the the paper, acknowledge that the exact mechanics of how "no excuses" style teaching leads to gains is still unknown. But the idea that "no excuses" might be student self-selection, or peer effects, or increased resources, or any number of other factors isn't a part of their argument:

Viewed in concert with work by Dobbie and Fryer (2015) and Angrist et al. (2016), we see an increasingly clear picture of a rapidly growing style of school management that boosts the college enrollment of poor urban high schoolers. Unfortunately, the similarity of the educational approaches considered in each study means that we can say very little about what mechanisms might be driving our results (at most, one might infer that the neighborhood-level interventions of the Harlem Children's Zone are not essential for increasing college enrollment, a conclusion shared by Dobbie and Fryer (2015)). Accordingly, understanding which "No Excuses" strategies are most important for generating long-term gains is an important topic for future research. Nevertheless, we are confident that this educational model has proved to be – and will continue to be – an effective part of the fight to increase low-income students' human capital. 

They are "confident" that "no excuses" works, even if they don't know how it works. But what if "no excuses" isn't anything that Dobbie & Fryer lay out? What if it's something else, like self-selection & peer effects? Shouldn't we address this possibility before suggesting things like "a relentless focus on academic achievement" -- whatever that is -- makes all the difference?



**** It's worth noting that there are several large assumptions Davis and Heller make in their study. They use last names as a proxy for race, they've got a spotty dataset for tracking college attendance, and they make assumptions about where the control students attended school.



That's not to say this is a bad study -- it's not. But it is, like all social science research, limited. And before we jump to conclusions and use research like this to guide policy, we should stop and carefully consider its limitations.