The new and improved New York State tests have arrived and the higher standards have taken much of the air out of many charter chain’s balloons.  Schools, in particular, from KIPP and Democracy Prep have seen their status change from ‘above average’ to ‘below average’ overnight, at least based on the ‘reform’ definition of quality.

But there is still one charter chain, standing tall, and that is the Success Academy network.  ‘Reformers’ must be conflicted.  On the one hand they have a charter network that is doing well.  On the other, many of the sacred cows of reform, including KIPP, are not looking so good.  If Success really has demonstrated that it has the recipe that the other chains do not have, then in the name of ‘accountability,’ wouldn’t it be best to stop investing in these other charter networks and focus solely on the one that seems to be working?

Still, I’d say that this is a ‘victory’ for the reform crowd right now since the Success schools seem to finally prove that ‘Poverty is not destiny’ — as long as you define destiny as below average scores on a multiple choice math and reading test.

I’ve examined the various statistics about Success and will summarize and analyze them here with the hope of shedding light on what things this network might be doing that ‘works’ and also illuminating some of the problems with this network.

An example of the press gushing about this network, see this recent New York Post article.  Here is an excerpt:

Of the 1,500 kids in her Harlem and South Bronx schools who took the Common Core exams, 82 percent got a passing score in math, and 58 percent passed English.

Across the city, the pass rates were 26 percent in English and 30 percent in math. Many other charter schools also had dismal showings, and not all passing scores are equal. Moskowitz’s students scored a disproportionate number of 4’s, meaning the top range.

Her gloriously lopsided results — her network topped Scarsdale schools by 14 points in math! — would be suspicious, except they happen routinely. Year after year, even as the number of students in her Success Academies grows, she cracks the code on getting disadvantaged children to excel.

…

The numbers tell the tale. Success Academy Bronx 2 was the top-performing nonselective school in the city and ranked third out of more than 3,500 schools across the state. Some 97 percent of its students passed math and 77 percent passed English, despite a poverty rate of 85 percent. The school did not have a single white or Asian student on exam day.

The first thing I did was take a look at the Success Academy Bronx 2 school.  This is a K-3 school so the only students who took the state tests in the school were the 97 third graders.  And, yes, 97% of those passed.  Now since they have only one grade that was tested, it is also true that the pass rate for the entire school is 97%, but I think it is a bit misleading to leave out this detail while other schools are K-5 or K-8.  Still, 97% is a very impressive score.

As far as serving the ‘same kids,’ it is interesting to compare the demographics of a Success Academy that shares a building with a public school.  Harlem Success Academy 1 and P.S. 149 is a representative example.  In 2012 Harlem Success 1 had 14.2% students with disabilities, 5% of students with limited English proficiency, 65% free lunch, and 13% reduced price lunch while their public school roommate had 28.2% students with disabilities, 8% LEP, 74% free lunch, and 5% reduced price lunch.

The Success Academies Network has 14 schools, but most of those schools don’t yet have a third grade class.  One school is K-7, three are K-5, and three are K-3.  In looking at their results, I focused on their fourth grade results because there are four schools to analyze there, and for each school I can do a comparison of last year’s results to this year’s results.

On average, they scored 54% on reading compared to 27% across the city and 80% on math compared to 35% across the city.  I made a scatter plot of all the 4th grade schools comparing their 2012 scores to their 2013 scores, seeing if there were any ‘outliers’ who were outperforming schools that had comparable 2012 scores.  I made the public schools blue, the charters red, and the Success charters yellow.

So the next thing I looked at was their student attrition.  If they ‘lost’ many students, these scores are tainted.  Now there is only one Success school that has been around since 2007.  That school started with 83 kindergarteners and 73 first graders.  Those cohorts just tested in 6th and 7th grade, respectively.  The school has ‘lost’ a big chunk of those original 156 kids.  Of those 73 first graders in 2007, only 35 took the seventh grade test.  Of the 83 kindergarteners, only 47 took the sixth grade test last spring.  Overall, they have ‘lost’ 47% of the original two cohorts.  If this is one of the costs of having such high test scores, I’m not sure if it is worth it.

For the four cohorts that just took the fourth grade tests, those 316 students were, back in 2009, 443 kindergarteners, so they have ‘lost’ 29% of those cohorts.  Now their high test scores aren’t completely explained by this nearly 30% attrition rate, but it is still something worth noting as we consider if this program is ‘scalable’ or not.

When a school is ‘healthy,’ teacher are happy there and want to stay there.  The Success schools are known to have huge attrition of teachers, in the neighborhood of 50% per year.  I received this comment recently from a teacher who taught at one, which indicates that there are some good things about the school, but that there is a lot of test prep as well:

I worked at Success Academy for two years. I hated it and I was miserable, as were most of my co-teachers, but the reason Success continues and will continue to do well on tests is that they have a progressive, research based curriculum in K-2. It’s not that the teaching is awe inspiring, it’s just that it’s consistently good. Though the staff skewed young, some of the teachers even had their own kids at the school. The school is heavy TFA — but the corps members have to work only as assistants their first year. The curriculum is semi-scripted. For example, they give us the basic teaching points for each unit, but it’s up to the teacher to decide how they want to deliver the lessons, and if their students need more of this or less of that. Success is also really on top of their RTI so its a bit harder for kids who get behind in pre-testing grades to go unnoticed. Also, they’ve been using common core for way longer than it was required.

Combine that with hardcore test prep and you get good results. My co-workers and I hated working for Success, but everyone, including myself have taken much of what we learned at Success and taken it into our new settings. I did see a little bit of counseling out, but the creaming people talk about is way overblown.

My larger concern is how bogus these test are. In my opinion, Success Academies deliver a high quality and relatively well-rounded education for low-income minority children. My concern is that even with a good education, kids need a ridiculous amount of test prep to pass the test. The test prep at Success runs like a machine. The kids get small group instruction all day during test prep–even the principals often stop their administrative duties to teach small groups. They even pull lower grade teachers to teach groups of test preps while assistants lead their classes. I hated that place, but I can’t deny their curriculum is excellent and they know how to do test prep right.

I guess the question is what, besides attrition and test prep, is helping this school get such good test scores, and are there any lessons that other schools can learn from them?  For one thing, I know that Success Academy schools generally have two teachers per class.  This is something that would be very costly to scale.  Also, though they do hire new TFA teachers, those teachers are never lead teachers, but function more as teaching assistants.  This, I think, is a pretty good use for TFAers and I’ve suggested before that all first year TFAers should serve this kind of role.  Also, at least according to the commenter who was not thrilled with her experience there, they do seem to have a good curriculum.  Another thing I should note is that I am less skeptical of a school where they start in kindergarten than I am with a KIPP middle school that starts with kids in 5th grade.  Early childhood education is something that I support, and the success of Success is an example of this helping.

Still, I’m not convinced that what they have gained in their high test scores outweighs what they have lost, particularly 50% of their original kids at the first school, and 30%, so far, of the kids who are now fourth graders.

In general, these good test scores, I think, should make the ‘reformers’ more nervous than elated.  From my perspective, I don’t think that the scores are devastating to my cause.  I don’t think they really prove that there are super teachers out there who can get the ‘same kids’ to excel, even if it is just on standardized tests, since I’m not convinced they are truly the ‘same kids.’  But the ‘reformers’ should be very careful about this.  They already had Success as a big success story, as well as a bunch of others like KIPP and Democracy Prep.  Now they still have Success, but they have lost some of their schools they used to take credit for.  I’m not sure how they can reconcile their idea that test scores are an accurate measure of school quality with the fact that many of the schools they have been touting have lost their luster by that measure.

And what ‘excuse’ is there for these other schools.  Surely behind closed doors they are accusing Success of some kind of manipulation, either by extensive test prep or by booting even more kids than they do.  I wonder if this could start some kind of charter civil war.

When I expose a charter school that has low test scores, remember that I’m not doing this because I think that the school is ‘bad’ because of those low scores.  For me, it is more of a “give them a taste of their own medicine.”  I don’t think a few days of testing in Math and reading can capture all that is learned in a school year.  The ‘reformers’ can keep pretending that they do and then one day there will be an expensive enough test that will be ‘test-prep proof’ and then suddenly they may find that they have no more schools to hold up as examples.