UPDATE: Bruce was writing about this exchange as I was. Go read what he says, then come back. Much more to come.
 

Tomorrow morning, if you pick up a copy of the Star-Ledger (chances are, you won't), you will see a big, fat, wet kiss to the KIPP/TEAM charter schools in Newark, right on the front of the "Perspective" section. The piece went on-line a few hours ago; you can comment if you would like...

But listen to this story before you do:

Over a month ago, I got an email from Diane Ravitch, the country's best known advocate for public education and most prominent critic of corporate-style education "reform." Diane cc'd it to me and Bruce Baker, a professor of education policy at Rutgers and my advisor in the PhD program there.

It seems that Julie O'Connor, the author of the Star-Ledger's piece, was looking for comments about KIPP/TEAM. Given Diane's stature and her well-known skepticism about charter schools, it's clear that O'Connor was looking for a contrary point of view.

Diane added a few remarks, but she also referred O'Connor to Bruce and me, knowing we've done scads of work on Newark's charters: see here, here, here, here, here, here, here, here, here, here, here, here, here, here, and here (and that's just for starters).

Bruce and I then began an exchange with O'Connor, which I have reposted, in its entirety, below. Bruce did the reconstruction, and I added my original graphs.

Understand that Bruce is one of the busiest people I know, and I teach full-time while working on my doctorate. Nonetheless, we took a considerable amount of our time to explain, in great detail, why a simple "doing more with less" framework (yes, that is an exact quote from the piece) is far too simplistic and misleading when it comes to evaluating KIPP/TEAM.

You are welcome to come to your own conclusions based on this exchange. Here are mine:

- It's clear O'Connor was in the tank for KIPP/TEAM from the start. Several times, especially in response to Bruce, she either doesn't understand -- or chooses not to understand -- what we are saying. Several times, Bruce refers O'Connor to his blog posts; they are quite clear in their methodologies and sources, but it's as if O'Connor never even read them.

- KIPP/TEAM is feeding O'Connor talking points. She keeps returning to the same arguments in her exchange with us -- and these are the arguments that make it into the article. Over and over, she asks Bruce or me to rebut claims KIPP/TEAM is making, rather than acknowledging the separate points we are making. It's as if, in O'Connor's mind, the debate can only be waged on TEAM/KIPP's terms; any other points of view, no matter how valid, need not be discussed.

- O'Connor ignores the most controversial aspects of charter school expansion in Newark. Where is any mention in O'Connor's piece of Pink Hula Hoop, the byzantine real estate deal involving KIPP/TEAM first uncovered by Star-Ledger alumnus Bob Braun? What about the controversial discipline practices of the KIPP network? Or the reports of "weaknesses in KIPP Academy’s internal controls over financial operations"? Or the controversy over KIPP's attrition rates?

O'Connor glosses over the fact that KIPP/TEAM has substantial philanthropic support, one of the reasons the KIPP chain spends much more per student than comparable district schools. I'd like to think my comment below at least made her consider the possibility that this contributes greatly to the chain's "success." But in her piece, she breezily takes the charter's word that they don't spend more on operations, not bothering to confirm whether this is actually the case.

I'm going to have much more to say about this sorry exercise in "journalism." For now, however, let's let Bruce have the last word on how charter school propaganda is spread by willing saps like Julie O'Connor and the Star-Ledger:

I enter into this blog post knowing full well that this is a lose-lose deal.  Rating and comparing school quality, effectiveness or efficiency with existing publicly available data is, well, difficult if not impossible. But I’m going there in this post. 

Why? Well, one reason I’m going there is that I’m sick of getting e-mail and phone inquiry after inquiry about the same charter schools – and only charter schools – asking how/why are they creating miracle outcomes. I try to explain that there may be more to the story. The reporter then says that the charter school’s data person says I’m wrong – validating their miracle outcomes (despite their own data not being publicly available/replicable, etc. and often with reference to awesome outcomes reported in popularly cited studies of totally different charter schools). 

But we may be having our conversation about the wrong schools to begin with.  The whole conversation starts perhaps with a call from the school’s own PR lackey to the local paper, along with a self-congratulatory press release, or alternatively, from the local news outlet itself following up on preconceived notions of which schools are doing miracle work (for a slow news day).  It’s not just that it seems always to be about charter schools, but that it seems to be about the same charter schools every time. [emphasis mine]

That is exactly right. When "journalists" lazily regurgitate PR from charter schools, they do their readers a huge disservice.

More to come...

* * * * *

April 6, 2015

Reporter Inquiry

Prof. Ravitch,

I’m on the editorial board at The Star-Ledger in New Jersey, and I’m working on a cover story for our Perspective section about the KIPP schools in our state. The college attendance stats of KIPP seniors in Newark seem pretty impressive, and I was wondering if you have the same reaction, and what you think of KIPP’s forays into Camden.

Would really appreciate it if you could give me a call at xxx-xxx-xxxx. Would like to discuss KIPP in the context of your criticisms of the broader charter school movement, and whether or not you think it is an exception.

Many thanks,

Julie O'Connor
 

The Handoff

Julie, 

I suggest you talk to Mark Weber and Bruce Baker at Rutgers, who have studied charters in NJ. I lean on their research. The question is not whether one chain can produce successful graduates, but whether charters in general are helping the most vulnerable schools, whether they are reducing the funding and capacity of public schools, and whether their success-when it exists--is the result of selection and attrition.

Diane Ravitch
 

Reporter

Ok, thanks for your prompt reply. 

Prof. Baker emailed me his report on free/reduced lunch and the TEAM schools, but I have been unable to reach him on the phone to discuss KIPP or my follow up questions.

Basically, I am looking for a reaction to two claims from KIPP that seem impressive: The college attendance rates (last year, 95 percent of KIPP seniors went to college, 89% to a 4-year, 6 percent to a 2-year), and the fact that KIPP kids in elementary and high school equal or outperform the average for the state of NJ (some years they do in middle school, too, though this year they didn’t). 

KIPP kids are 87% free/reduced lunch and the state is in the 30s. I understand that Baker and others are skeptical about comparing KIPP kids to their peers in the Newark district. But what about comparing them to the state average? And what about their college attendance rates?

I would like to discuss the criticisms of the charter school movement and whether you view KIPP as an exception, or more of the same. Prof. Baker, can you please give me a call as soon as you get a chance? xxx-xxx-xxxx. We are hoping to run the story in the next week or so.

Many thanks,

Julie
 

Baker to Reporter

My point is, and shall continue to be that news stories on education should NOT be driven by some PR prompt from specific schools touting their "successes" through anecdotes. Thus, my only reaction is the reaction I posted previously about school performance, given analyses across all schools, using comparable, publicly available data:

https://schoolfinance101.wordpress.com/2015/01/30/ed-writers-try-looking-beyond-propaganda-press-releases-for-success-stories/

The bottom line is that KIPP schools performance on comparable measures of student growth, controlling for demography, resources, etc., are relatively average (marginally above average). Many district schools, including ones in Newark, far outperform them.
 

Reporter to Baker

Ok. Even if KIPP students aren’t representative of their district, isn’t it still impressive that they are beating the state average, given that their student population is significantly poorer? 

KIPP says 93 percent of their students stay with them (7 percent leave their schools each year for any reason).

If what this tells us is that KIPP students have high scores and go to college, how do they fit into criticisms of the larger charter school movement? And what do you think of KIPP’s expansion into Camden?

Prof. Baker, read your blog post and would like to discuss. I am not sure how you are measuring growth in these ranked schools. Are you skeptical about the accuracy of the college attendance rates and performance numbers reported by KIPP? If so, why? Please give me a call. xxx-xxx-xxxx.

Thanks.
 

Baker to Reporter

Not without running a model of demographics against the same outcome measures across all schools, to see how/whether they truly deviate, statistically, from expectations. Anecdotes of this type are unhelpful for understanding what’s “impressive” statistically or not.  

For measuring growth, I’m using the state’s own reported school Median Growth Percentile – for 2012, 2013 and 2014.

Skeptical or not, context is what’s needed for them to really mean anything. The context of all other schools, and their demographics, to evaluate statistically whether the KIPP schools actually deviate from what would otherwise be expected (given enough schools to estimate a model of expectations).
 

Reporter to Baker

Ok. Is the state average not considered a good measure of how schools are doing? 

Is your central point in creating your own measurement for whether schools deviate from expectations that KIPP schools have more resources and classroom time and better class sizes, and that’s why their students are doing so well? 

Are you trying to account for those factors in your outcome measure, since you might not find such conditions in traditional district schools? That seems to be your argument in this blog post:

https://schoolfinance101.wordpress.com/2013/03/01/the-non-reformy-lessons-of-kipp/

Trying to understand your general view of KIPP’s performance.
 

Baker to Reporter (w/head banging against desk)

No. State average is NOT a useful comparison.  Given the number of things that vary across schools, one needs to look at any given school in the context of all schools, with all available measures. Not just compare one school to the state average and say, for example, “it’s got higher poverty, and higher outcomes than the state average.” That comparison misses a lot of other factors that may vary across schools. One needs to see how those factors affect the outcome measure across schools and then compare against the overall pattern.

Second – I’m not “creating” my own measurement. I’m doing what I describe above. Taking the state’s measures, and making comparisons among “otherwise similar” schools along the trend of schools, given their various attributes. That is, how much higher, or lower than expected, does a school score (on growth) given all of those factors that vary.

Now, I also use the state’s growth measure,  because, for all its shortcomings, it is actually the best available New Jersey measure of what a school might be contributing to student outcomes (rather than what kids come in with, or who leaves and when). But that measure too is ONLY useful if you control for/account for the various factors. Quite simply, this is how credible analysis of this type is done, knowing full well that even this approach can’t capture some factors that affect outcomes that really aren’t about how good/bad a school is.

Their performance tends to be marginally above average, to about average, considering all schools including district schools. For that matter, several Newark district schools have higher performance. Discovery Charter school is the standout among charters. North Star seems to do well, but I believe that the model isn’t really capturing the effect of their substantially greater attrition, or different student population. But who knows.  But then again, Robert Treat has very different student population and tends to show very weak gains with adjustment for the included factors.
 

Reporter (who clearly never bothered to read the original post)

What factors that vary are you trying to account for? It is things like resources, classroom time and class sizes?
 

Baker to Reporter (direct response to ignorant question)

They are all listed in the blog post!

https://schoolfinance101.wordpress.com/2015/01/30/ed-writers-try-looking-beyond-propaganda-press-releases-for-success-stories/

Outcome is Growth

Corrected for:

1. prior average scale score level
2. % free lunch
3. % disability (because I cant' break out by severity, charters like TEAM actually get an advantage here)
4. % Ell
5. total staffing expense per pupil
6. school grade range served
7. school size
    

More Exasperated Baker to Reporter

Schools in Newark: https://schoolfinance101.files.wordpress.com/2015/01/slide18.jpg

Charter Schools Statewide: https://schoolfinance101.files.wordpress.com/2015/01/slide24.jpg

So again, I ask, why do you feel the necessity to write a story on KIPP schools? And why the apparent obsession on trying to find a miracle in KIPP? How do these supposed miracles (that generally aren’t) come across your desk?

An objective statistical run of all schools in the state, using the state’s own best available measure as the outcome, finds TEAM in Newark to be a decent – relatively above average - school, but no miracle. There are no miracles in this complex endeavor. That’s fine. They do a pretty good job, and seem to do a better job of serving a more representative student population than some others (see also:https://schoolfinance101.wordpress.com/2013/11/27/where-are-the-most-economically-segregated-charter-schools-why-does-it-matter/)

I’m not trying to rain on their parade. I’m just pointing out that if we take all of the data from schools around the state and try to figure out who’s actually “doing better than expected” given who they serve and the resources they have, we don’t identify KIPP as the standout.
 

Weber to Reporter

Julie, I am going to encourage you to read Bruce’s entire post, as it is far more sophisticated and comprehensive than what I am going to include here.

That said, let me put this in very simple — admittedly, TOO simple — terms:

[Note: I made this graph just for this exchange. That's how much I wanted O'Connor to understand my point.]

This is a very quick and very dirty scatterplot that shows the average scores on the NJASK Grade 8 English Language Arts (ELA) exam from last year for every school in the state. I’ve highlighted TEAM on this graph.

The NJASK score is on the vertical or y-axis. On the horizontal or x-axis is the percentage of students who qualify for free or reduced price lunch, a proxy measure for student economic disadvantage (a student’s family has to be at 185% or below the poverty line to qualify for FRPL).

The first and most obvious thing to notice is the relationship between how many FRPL kids a school has and its average test score. Clearly, when FRPL goes up, test scores go down. 70% of the variation in these scores can be statistically explained by the percentage of FRPL kids at the school. 

We all know this. Poverty matters.

The green line through the middle is called a regression line: it’s a kinda-sorta “average” that predicts how well a school will do given its FRPL percentage. If you’re above the line, you’re doing better than prediction; if you’re below the line, you’re doing worse.

TEAM is above the line - hooray for them. But how many other schools do you see across the state that are at least as far above the line as TEAM? How many are way, way further above that line compared to TEAM?

Again: what Bruce did in his post was far more sophisticated than this, because he’s using a statistical model to account for other things that will affect student outcomes, like percentages of special education kids and how much a school spends per pupil on staff (yes, money does matter). He’s also judging outcomes on SGPs, which is arguably a better measure of a school effectiveness.

I’m boiling this down, however, to reinforce his point: yes, TEAM is a better-than-average school. Again, good for them… but why all the outsized attention? Why are you writing a story about them and not the many, many other schools that “beat prediction” much better than TEAM? How many district schools could be considered “miracles” relative to TEAM that get ignored by the op-ed pages of your newspaper?

Julie, you and I both know I have been the Star-Ledger Editorial Page’s harshest critic on education. I’ve admitted before that sometimes I have gone too far… but can you understand my frustration? Can you understand how unfair it appears to those of us who have taken the time to study Bruce’s work that TEAM gets all the accolades while many schools that — by TEAM’s own standards — are doing a BETTER job than they are, yet continue to be ignored?

I am asking you to listen to Bruce carefully and take the time to understand what he is saying. This stuff matters. You control arguably the most important space for punditry in the state. You owe it to your readers to get this stuff right.

If I can help further, let me know.

Mark Weber
 

Reporter (still not bothering to read, and returning to anecdotes provided by school)

What about the 95 percent of KIPP seniors that went to college last year? That seems impressive to me. 

Also, when you say comparing KIPP to the state average doesn’t mean anything without "running a model of demographics against the same outcome measures across all schools, to see how/whether they truly deviate, statistically, from expectations” — isn’t that what the Mathematica study does? Control for any differences in student population? 
 

Baker (even more exasperated) to Reporter

Why don't you write it that way then - that it seems impressive to you.  I'm not going there, with your representation of data, passed along to you most likely by the school, without opportunity run appropriate models on the data. And I don't have time to be doing that right now, or quibbling with you over your strange incessant desire to write a story on how awesome you think these schools are, without ever bothering to look at the schools in the context of all schools, where many others may, in fact be even more impressive.

And are you speaking of some Mathematica study of TEAM Academy specifically, and their graduation and college matriculation rates? Or Mathematica studies of KIPP schools generally/nationally ? [I believe only the latter exists -http://www.mathematica-mpr.com/~/media/publications/PDFs/education/kipp_middle.pdf]  

Yes, the network's results are solid. Not miraculous. But solid. Driven in part, perhaps by selection issues (see methods critiques below), and in part by resources. KIPP schools in many contexts substantially outspend their "competition" offering higher salaries, much smaller classes, longer days/years, etc. Certainly won't deny that those types of resources matter.

Comments on related methods here: https://schoolfinance101.wordpress.com/2012/12/20/thoughts-on-randomized-vs-randomized-charter-school-studies/

and: https://schoolfinance101.wordpress.com/2013/07/12/thinking-writing-about-educational-research-policy-implications/

There are indeed limitations these methods.

Some information here on where TEAM fits on resource/demographics, etc in Newark: https://njedpolicy.wordpress.com/2015/01/13/research-note-resource-equity-student-sorting-across-newark-district-charter-schools/
 

Weber to Reporter

Related to the issue of resources:

Find attached the 2012 tax forms for TEAM, Friends of TEAM, and KIPP. You can access these easily at guidestar.org.

You will notice on page 42 of the KIPP 990 that TEAM received $1,053,147 in direct support from KIPP. This likely does not include all sorts of administrative, logistical, marketing, lobbying, etc. activity KIPP undertakes on behalf of TEAM.

On page 21 of the Friends of TEAM 990, you’ll find a $1,005,332 grant to TEAM. On page 9, you’ll see the group took a rental income loss of $1,813,501, likely to the school’s benefit (were I you, I’d certainly ask them about this).

In 2011-12, TEAM enrolled 1,504.5 students. If you take the grants from KIPP and FOT together, that comes to $1,368 additional expenditures per child, not including the rental loss that FOT took. So far as I know, this extra funding is not reported in the NJ Taxpayers Guide to Education Spending.

Let me be clear: it is, in my opinion (an opinion backed up by a substantial and growing body of research) that spending this extra money on behalf of these students will help their academic growth. This is a good thing.

But it is exactly the sort of issue that is not addressed by the Mathematica report, nor by any number of other “studies” that purport to show the superiority of KIPP’s methods by holding all things constant.

So how does TEAM spend all this extra money? Well, here’s one way:

At all stages of a teachers career, TEAM pays a higher salary, even when adjusted for experience, than NPS (and way more than Newark’s “local” charters). When you pay more and offer better working conditions, you can attract people who are willing to work longer hours (to a point).

But they manage to keep salary costs low by also doing this:

Notice the high number of teachers with only one year of experience at TEAM? Notice how they barely have any teachers with more than 15 years of experience? That’s when the NPS salary guide gives veteran teachers a big boost.

Is this a smart strategy? Absolutely. Is it sustainable? I say almost certainly not. Does TEAM really think they can keep recycling their staff AND expand the number of students enrolled? Are there really that many young people out there willing to make teaching at TEAM a temporary career? And is that really good for the city and its students?

As Bruce says: TEAM does a good job. They are, by the numbers, a good school. But I would argue KIPP's methods are not replicable at a large scale. In fact, THEY’D probably agree with me, because they have said over and over again that they are not interested in taking over an entire district.

Julie, if you are willing to dig into this and go behind the talking points the KIPP publicity machine feeds the press, I think you will find TEAM’s “success” raises more questions than it answers:

- If more money is good for charter schools, why isn’t it good for pubic schools?

- Is it good for the teaching profession to encourage the growth of schools that appear to run on a policy of churning much of their staff?

- When we get past the issues of different student populations, attrition, extra resources, hiring practices, test prep, etc., what, exactly, is so special about KIPP/TEAM?

Mark