Three years ago, the Achievement School District (ASD) in Tennessee began their mission, summarized in the statement on their website under the heading ‘Building the Possible.’

“The Achievement School District was created to catapult the bottom 5% of schools in Tennessee straight to the top 25% in the state.”

The timetable for this goal is just five years from the time the school enters the ASD.  As I wrote to current ASD superintendent, Chris Barbic, who I’ve known for over 20 years from back in the days when we were both TFA teachers in Houston at the same time, this is not a feasible goal.  It isn’t that I don’t think schools or teachers are capable of improvement, I just think that there is a limit to what can be accomplished by only focusing on replacing teachers and giving schools over to charters.

I don’t know if this was an intentional thing or not, but look at the ASD logo.

See that triangle thing?  It is called a Penrose Triangle or sometimes a Penrose Tribar.  It is an optical illusion since something like this cannot be built.  According to Wikipedia, in the 1950’s Penrose, himself, described the object as “impossibility in its purest form.”  In a very candid moment, Barbic admitted during a panel discussion that even if the goal was unattainable, it served a purpose because it created an energy around a lofty goal which attracted funders and talent to the district.

I have been following the progress of the ASD since it started and have been doing a yearly summary, in addition to other posts throughout the year.  The 2013 edition can be found here and the 2014 edition can be found here.  I plan to continue this yearly update until the end of the fifth year in 2017.

At the end of July 2015, Tennessee released the state test score data for all the schools and all the districts.  Every year when the data is released, Tennessee and the ASD are ready with their own spin on the results.

The website Chalkbeat Tennessee titled their article about the data release ‘At critical moment, state run Achievement School District posts big gains at its original schools.’  In the article they had this interesting bar graph supposedly supporting this claim.

So the first thing I noticed is that while math is ‘up’ by about 11% in the past three years, Reading is ‘down’ by about 4%.  And reading had gone up a bit last year, but now its down again.  The Science, I don’t know much about that test so I’m not going to focus on it.  Not that I don’t think science matters, but I’ve never heard a ‘reformer’ talking about anything but math and reading so I’m not going to let science into mix here.

Now reading in the whole state of Tennessee is down 1.5% over the past 3 years, it has actually dropped a little each year since it peaked in 2013.  It is now at the level it was before all the Race To The Top reform happened there, amazingly.  So the ‘big gains’ are the 6 points in math.  But isn’t reading really really important?  What good is a program that increases math scores at the expense of reading scores?  It’s like a drug that gets you to lose weight but also causes Cancer.  The other thing to notice is that these percentages, even the 27% after the ‘big gain’ is way below the state average of 55.6% this year.  And the reading at 13.8% is way way below the state average of 48.4%.

I decided to do a little fact checking on this.  What I found surprised even me.

As the ASD has been around for three years, there are three ‘cohorts’ of schools.  There are the six original schools which have been in the district for all three years.  Then they added eleven more schools the second year, and six more the third year.

For my analysis, I am looking only at the six original ASD schools since I have three years of data for them so they are the most relevant.  The six schools in the original cohort are Brick Church College Prep, Cornerstone Prep — Lester Campus, Corning Achievement Elementary School, Frayser Achievement Elementary School, Humes Preparatory Academy — Upper School, and Westside Achievement Middle School.

Last summer I analyzed the scores from these six schools and determined that two had improved scores, two had about the same scores, and two had worse scores than before the takeover.  It definitely seemed like mixed results to me.  On their ‘growth’ metric the entire district actually got the lowest possible score, a 1 out of 5.

Of the six schools, the one that was doing the best, as of last year, Brick Church, was somewhat different than the other four because they were ‘phase in’ schools meaning that instead of the charter operator taking over the entire school, current students and all, they just took the newest students entering the school.  So these schools were grades 5-8 and the first year they just took the new incoming 5th graders and they would ‘grow’ the school one grade at a time until they eventually took over the whole school.  As these two schools were outperforming the others, it seemed to show that the more disruption the better.  Brick Church had such big gains during the 2013-2014 school year that Barbic declared that it was on target to make it into the top 25% one year ahead of schedule, after just four years!

One year ago Barbic said in an interview that three of the first six schools were on track to make it into the top 25% in five years.

Here’s a quote from that article:

The special statewide district is taking over the lowest-performing schools in the state with a goal of moving them into the top 25 percent in just five years. Now in year three, superintendent Chris Barbic says he’s encouraged.

“You know, when we first talked about this, this was a goal that folks thought was completely crazy. And I think we’re learning is that not only is it not crazy, but we’ve got three of our first six schools that are on track to do it.”

One of those three schools on the right trajectory is Brick Church Pike College Prep in Nashville, which is slowly being converted into a charter school run by LEAD Academy. Barbic says if Brick Church matches this year’s student growth in math and reading, it would leap into the top quartile a year early.

He made this prediction based on the assumption that Brick Church’s 17 point gain in math and 24 point gain in reading was not a statistical outlier but something that could be repeated for the next two years.  Of course Brick Church was the first school I checked their 2014-2015 scores for and they had huge drops (negative gains they call them!) of around 14 points for math and 17 points for reading, taking them way off the trajectory Barbic had predicted.

Looking over the ASD scores for reading and math I noticed that all six of the original schools had gone down in reading while half of them had gone down in math.  The one bright spot in the original cohort was Frayser’s math scores going from 14.6% to 37.6%, a gain of 23 percentage points.  But looking at Frayser’s reading scores they had gone down 1.3% to a minuscule 7.6% proficient, one of the lowest scores in the state.  I don’t know how they got their math scores up so much while having nearly no students pass reading.  Regardless, I’m not sure that Frayser should be throwing any sort of victory party.

The question that nobody seems to be asking is the most obvious one:  At what percentile are each of these six schools now that they have been part of the ASD for three years?  Originally they were in the bottom 5% in order to be eligible for the ASD.  The mission of the ASD is to get them up to the top 25% in 5 years (though Barbic recently said that it could actually be 6 or 7).  We hear about how Frayser’s math scores went up by 23 points this year and that ASD itself ‘outgrew’ the rest of the state in math by increasing it by 6 points.  We also hear that schools that have been in the ASD for two or three years got the highest possible ‘growth’ score this year, which is a 5.  But they never answer the simple question:  At what percentile are the original six schools at now?

Of course there are different ways to assign a single numerical grade to a school in order to rank them and see what percentile each school is at.  I devised the most simple metric possible and which I think the most data-driven ‘reformers’ would approve of.  Just add together the percent proficient in math to the percent proficient in reading to get a score that has a minimum of 0 and a maximum of 200. If a school had 80% passing math and 70% passing reading, they have a total score of 150, for example.  Yes, there’s a lot more to schools than just these two numbers, but I wanted to keep things simple and inline with the sorts of calculations that ‘reformers,’ themselves, like to use.

By this metric the top performing ASD school from the first cohort was Coming with a score of 48.6 followed by Brick Church (47.9), Frayser (45.2), Westside (42.1), Cornerstone (37.6), and Hume (33.1).  To check where these scores ranked compared to all the Tennessee schools, I calculated this metric for all 1358 schools that had 3-8 math and reading and sorted them from high to low.

Here are the results:

As you can see, four of the original six schools are still in the bottom 5% while the other two have now ‘catapulted’ to the bottom 6%.  Perhaps this is one reason that Chris Barbic recently announced he is resigning at the end of the year.

Throughout the country, there are states that are considering creating their own ASD based on the supposed success of this one and the Recovery School District in Louisiana, on which this one is based.  Senate Democrats actually tried, and failed, to get an amendment into the reauthorization of the ESEA that would mandate that the bottom 5% of schools in each state become an ASD, essentially.  I hope that my very simple calculations are compelling evidence that the ASD does not live up to the hype.  Getting two out of six schools from the bottom 5% to the bottom 6% has not earned them the right to replicate around the country.

If you want to crunch the numbers yourself to verify my results or to find some of your own, here is my excel file I adapted from the publicly released state data.  And here is the raw data from the state.