Joel Klein responds to Aaron Pallas’ initial column (Pallas' Response Follows)

Aaron Pallas, an ed school professor at Teachers College, appears to be unwilling to acknowledge that our public schools are failing to effectively educate huge numbers of our kids, or that there’s much we can do about it. He struggles to debunk existing examples of demonstrable success perhaps fearing that we might otherwise ask why do we keep doing so poorly when we have proof that we can do so much better.

To that end, last week Pallas penned a piece in this column challenging my assertion in a Washington Post op ed that our “schools can get much better results with th[e] same kids than they’re now generally getting.” Employing a locution that I never used, and that cannot fairly be inferred from what I said, he tries to portray my view as placing “the emphasis on what schools can extract from kids.” (His italics.)

No, Professor Pallas, I don’t think knowledge resides in kids and, like iron ore, all we need to do is carefully extract it. What I do think is that our schools, and especially our teachers, need to do a much better job of educating our kids – that is, teaching them the skills and knowledge they will need to be successful in the 21st century. As I put it in my piece, “teachers matter, big time.”

To illustrate my point, I used a recent, powerful example, involving the well-known KIPP charter schools. KIPP followed its 8th grade graduates, who were overwhelmingly from poor (85%) and minority (95%) families, and found that, ten years later, 1/3 of them had graduated from college, a rate that was about four times their expected graduation rate and the same as that of white students.

Pallas tries to dismiss this compelling demonstration of what a great education can accomplish on the ground that the study didn’t explain “how children are selected into and out of the [KIPP] program and its schools.” In fact, just a couple of weeks ago in this very column, Ryan Hill, executive director of the KIPP schools in Newark, rebutted this argument at length, relying on his own knowledge and experience as well as on a carefully designed research study by the well-respected Mathematica group.

Based on this analysis, Hill shows that the success of KIPP students is “not the result of having especially highly motivated parents, high entering test scores, or any other unusual advantages over their peers in district schools.” Their success is instead, Hill explains, “a credit to their hard work, and to their caring teachers’ ceaseless drive to find better and better ways to serve all our kids.”

But let’s assume Pallas’ insinuation had some basis — that KIPP did, in fact, have an advantage through its selection or discharge processes. That still couldn’t possibly explain a college-success rate of four times the expected rate for the demographic group at issue. Let’s give it a discount, and say it’s only three times (or even twice) the expected rate. That’s still an enormous tribute to KIPP and its teachers. And it immediately raises the question: why aren’t other schools getting the same results?

As I also said in my op ed, there are many other examples proving that essentially the same kids can achieve at very different levels depending on the school they attend. I’m sure Pallas is familiar with this research, but let me provide a few recent examples: a study by MDRC showing that the new small schools in NYC significantly outperformed other high schools in the city across all demographic groups; studies by Margaret Raymond and Caroline Hoxby at Stanford University showing that charter schools in NYC significantly outperformed nearby district schools; and a study by Tom Kane at Harvard showing the same for Boston’s charter schools.

All of these studies were done by, in Pallas’s words, “reputable social scientist[s],” applying rigorous research protocols, including comparing only those students who actually applied to the schools in question, some of whom were admitted by lottery, and others of whom didn’t get in despite trying.

Unsatisfied with challenging the basic premise that our schools can achieve much better results, Pallas went on to take a gratuitous shot at the performance of NYC’s schools under my tenure as chancellor. He relies on NAEP scores, which I will come to in a moment. But here, too, Pallas chooses to ignore significant research, this time by James Kemple, who heads the Research Alliance for New York City Schools at New York University.

Applying a “rigorous analytic method,” Kemple found “compelling evidence of strong positive effects on student outcomes from the constellation of [NYC] reforms” adopted during my chancellorship. These positive effects were observed in every area studied – 4th and 8th grade English language arts and math, as well as high-school graduation rates. In addition, the improvements “grew over time,” and “accrued to both general education students and to students with disabilities, with especially large effects for the latter.”

The increase in graduation rates alone is remarkable and, to my knowledge, unmatched anywhere else in the country. After a decade of being flat, they went up more than 20 points under Mayor Michael Bloomberg’s administration. Indeed, the State of New York has kept track of graduation rates for the past five years, during which time the city went up 14 points, while the other four large cities in the state (which started at the same level as the City) went up an average of 3 points. That’s dramatic evidence of NYC’s success.

Pallas ignores all this, and focuses on the City’s NAEP results, the national tests that are administered throughout the country to 4th and 8th graders in math and reading, on a sampled basis. Without describing the actual results, he concludes that NYC achieved “modest growth.” Here are the facts: in the fourth grade, NYC went up 11 points in math from 2003-2009 and 11 points in reading from 2002-2009 (4th grade reading is the only NAEP test NYC took in 2002).

According to the National Council on Education Statistics, which administers NAEP, 10 points is roughly equivalent to a year’s worth of learning, meaning that kids in the 4th grade in NYC are now a year ahead of where they were 6 or 7 years ago. Indeed, NYC’s fourth graders are performing at the same level as the nation as a whole, which is quite an accomplishment given that the City educates a much more challenging population of students than does the nation.

The results in eighth grade are mixed: from 2003-2009, the city went up 7 points in math – almost 3/4s of a year — and was flat in reading (though it did go up 3 points from 2007-2009, suggesting the benefits of the good results in the fourth grade were starting to show up in the eighth grade as well).

Rather than use the actual numbers demonstrating significant gains on three of the four NAEP assessments, Pallas compares NYC’s performance to other large cities, and finds that the City didn’t outperform them. But many of the comparison cities implemented reforms that were similar, if not identical, to those adopted in NYC. In any event, the fact that others experienced significant improvement on NAEP in no way detracts from the big increases in NYC. No one has suggested that gains on NAEP don’t reflect real progress.

To similar effect, Pallas claims NYC’s NAEP scores didn’t close the racial and ethnic achievement gaps in the City. That’s only true because the City’s white students showed the same large gains as did its minority students. Surely, we don’t want to close gaps by slowing the progress of white students. More to the point, the City’s African-American and Latino students did, in fact, close several gaps on NAEP when compared to white students throughout the nation, which is what ultimately matters.

As I have said elsewhere, NYC is not remotely where it needs to be. There are still far too many students who aren’t graduating and, according to newly articulated state standards, far too many who aren’t college ready. But the progress that has been made under Mayor Bloomberg’s leadership is clearly substantial. That progress didn’t come from what Pallas calls “unproven reforms.”

On the contrary, those reforms, based largely on sound principles of accountability and parental choice, led to the progress that was achieved. And the reason there wasn’t more progress is because the defenders of the status quo – supported by the nihilist rhetoric of Pallas and others – blocked or limited more of precisely these kinds of effective reforms. 

Pallas responds --

It’s an appealing narrative: A new generation of school leaders takes on entrenched interests. Reforms are new and bold. Outcomes soar.

This is the story Joel Klein sought to tell in his op-ed in The Washington Post two weeks ago. He named a number of alumni from his administration who had taken on leadership positions in school districts and states across the country, and—coupling them with other recent appointments offsetting his and Michelle Rhee’s resignations—proclaimed, “What a difference a few months can make!” Pointing to the success of graduates of two KIPP charter schools, Klein claimed that the KIPP students “essentially eliminated the achievement gaps with respect to race, ethnicity and poverty.” He then generalized from these two cases to argue that all schools can do much better, relying on reforms such as holding teachers accountable for student performance and providing greater choice for families.

The narrative falls apart, however, if one looks beyond the two KIPP schools to a broader array of evidence about the impact of reforms Klein has championed. In my blog post on Klein’s op-ed, I suggested setting the KIPP story aside—because Klein and the other reformers he mentioned had nothing to do with the apparent success of those schools. Instead, I drew attention to student performance during the era that Klein and his like-minded colleagues were at the helm of the New York City schools. After eight years, did achievement rise precipitously? Did New York City make significant progress in closing the achievement gaps that separate students of different racial and ethnic backgrounds? Relying mainly on analysis of the performance of New York City students on the National Assessment of Educational Progress, often referred to as the “Nation’s Report Card,” I concluded that the modest gains over time observed in New York City were comparable to those in other parts of the country, and that the achievement gap was nearly as large at the end of Klein’s tenure in 2010 as when he took over in 2002.

Not surprisingly, Klein took umbrage at my characterization of what was achieved during his tenure. (See his response in the column to the right.) He likes KIPP a lot, and took some time to rebut my concern about the claims that might be drawn from the data on those two KIPP schools, but that’s a distraction from his administration’s record in New York City—as are his references to studies of charter-school effects in Boston and New York City. (No mention of Margaret Raymond’s finding that only 17 percent of charter schools show gains that are better than their traditional public-school counterparts—but why mess up a good story?) Instead, Klein relies heavily on a study of gains on state tests in reading and math and in graduation rates conducted by my colleague Jim Kemple.

Beyond the name-calling, there is a profound difference in how Klein and I make sense of the available data. I have a great deal of ambivalence about state tests and high-school graduation rates because they have proven vulnerable to manipulation, which undermines their validity as measures of student achievement. Tests and graduation rates can tell us something about what students have learned, but there is a great deal of uncertainty about what these measures mean. This uncertainty argues for caution in drawing claims about the success of school-reform efforts based on these data.

Joel Klein, on the other hand, created an accountability system in New York City that treated test scores and graduation rates as virtually the only outcomes of schooling worthy of attention—ascribing in the process a precision to these numbers that is unwarranted. It makes one wonder what specifically Klein thinks schooling is about and what an educated student ought to know upon leaving high school. I consider myself a pretty careful observer of Klein and his record, but I cannot recall ever having seen a detailed account in his own words of what students should know and be able to do.

Why am I ambivalent about the New York State tests in reading and math? Over the past several years, there has been a sharp divergence between trends on federal tests and those reported by New York State. The New York State Department of Education acknowledged last year that state tests in reading and math administered in grades 3-8 became easier and more predictable over time, calling into question their ability to measure what students have learned. When the state recalibrated the scores, most of the “gains” of the previous eight years vanished.

And while Jim Kemple’s analysis found that proficiency rates on state tests rose faster between 2002 and 2010 in New York City than in other parts of New York State, those gains were rooted in a set of tests and proficiency cutoffs that the state of New York has asserted were too easy and too predictable. I’m reluctant, then, to treat this as strong evidence of meaningful academic growth during the Klein era.

As for high-school graduation rates, it’s true that these rates have been rising rapidly in New York City. Whereas 47 percent of the entering ninth-grade class of 2001 graduated from high school in four years, 65 percent of those entering ninth grade in 2006 did so. And the gains observed for black and Hispanic students were especially large, rising from 40 to 61 percent, and 37 to 58 percent, respectively. Taken at face value, these patterns suggested that school achievement had risen steadily and the racial/ethnic achievement gap had shrunk substantially.

But consider that three out of every four New York City high-school graduates who enter the city’s community colleges require remediation in basic skills—and that fully 50 percent of those who enter the CUNY system, including its senior colleges, need remediation. One wonders, then, whether completing high school in New York City truly represents a high level of academic accomplishment.

Not according to the state of New York, which is in the process of phasing out a tiered system of diplomas that distinguished among Local Diplomas, Regents Diplomas, and Regents Diplomas with Advanced Designation. Whereas all students have been required to accumulate 44 credits in various subject areas in high school, the type of diploma awarded has rested on performance on the Regents exams. A Local Diploma could be earned by achieving a score of 55 or higher on each of five Regents exams in English, mathematics, science, U.S. history and government, and global history and geography. A Regents Diploma had a higher threshold of performance; students could earn a Regents Diploma by scoring 65 or higher on each of these five exams. An Advanced Regents Diploma required a 65 or higher on additional examinations in mathematics, science and a language other than English. The gains in the graduation rate observed in New York City have been more pronounced for the less rigorous Local Diploma, which is the one being phased out.

The accountability system that Joel Klein imposed on high schools created incentives for awarding credits to students who hadn’t demonstrated mastery of course content. Regents exams—scored by a school’s own teachers—have shown a peculiar bulge in performance at the threshold score of 65, suggesting there are pressures to pass students just below the cutoff score. This casts serious doubt on whether a New York City high-school diploma is a reliable indicator that a young person is prepared for life after high school.

But scoring a 65 is not a high level of performance on the Regents exams, because many students scoring at this level are subject to remediation in college.  This year, the New York State Board of Regents released a new measure of college readiness characterized as an aspirational performance measure.  It tabulates the number of students from each entering ninth-grade high school cohort who scored high enough on the Regents exams to predict placing out of remedial courses or earning at least a C grade in college-level courses at two- and four-year institutions in New York City and around the state of New York.

How do the black and Latino youth in New York City’s high schools, who make up 70 percent of the system’s enrollment, fare on the state’s measures of college readiness? You won’t find this number in any of the PowerPoint decks prepared by the New York City Department of Education, but it’s easy enough to calculate from publicly available data. For black and Latino students who entered high school in 2006—and who thus had the benefit of four years of elementary and middle school, and all four years of high school, under the policies and practices of Joel Klein’s administration—the proportion emerging ready for college in 2010 was 12 percent.

Twelve percent of black and Latino students graduating from high school ready for college—after eight years on Chancellor Klein’s watch.

Proven reforms? You be the jud

This is the story Joel Klein sought to tell in his op-ed in The Washington Post two weeks ago. He named a number of alumni from his administration who had taken on leadership positions in school districts and states across the country, and—coupling them with other recent appointments offsetting his and Michelle Rhee’s resignations—proclaimed, “What a difference a few months can make!” Pointing to the success of graduates of two KIPP charter schools, Klein claimed that the KIPP students “essentially eliminated the achievement gaps with respect to race, ethnicity and poverty.” He then generalized from these two cases to argue that all schools can do much better, relying on reforms such as holding teachers accountable for student performance and providing greater choice for families.

The narrative falls apart, however, if one looks beyond the two KIPP schools to a broader array of evidence about the impact of reforms Klein has championed. In my blog post on Klein’s op-ed, I suggested setting the KIPP story aside—because Klein and the other reformers he mentioned had nothing to do with the apparent success of those schools. Instead, I drew attention to student performance during the era that Klein and his like-minded colleagues were at the helm of the New York City schools. After eight years, did achievement rise precipitously? Did New York City make significant progress in closing the achievement gaps that separate students of different racial and ethnic backgrounds? Relying mainly on analysis of the performance of New York City students on the National Assessment of Educational Progress, often referred to as the “Nation’s Report Card,” I concluded that the modest gains over time observed in New York City were comparable to those in other parts of the country, and that the achievement gap was nearly as large at the end of Klein’s tenure in 2010 as when he took over in 2002.

Not surprisingly, Klein took umbrage at my characterization of what was achieved during his tenure. (See his response in the column to the right.) He likes KIPP a lot, and took some time to rebut my concern about the claims that might be drawn from the data on those two KIPP schools, but that’s a distraction from his administration’s record in New York City—as are his references to studies of charter-school effects in Boston and New York City. (No mention of Margaret Raymond’s finding that only 17 percent of charter schools show gains that are better than their traditional public-school counterparts—but why mess up a good story?) Instead, Klein relies heavily on a study of gains on state tests in reading and math and in graduation rates conducted by my colleague Jim Kemple.

Beyond the name-calling, there is a profound difference in how Klein and I make sense of the available data. I have a great deal of ambivalence about state tests and high-school graduation rates because they have proven vulnerable to manipulation, which undermines their validity as measures of student achievement. Tests and graduation rates can tell us something about what students have learned, but there is a great deal of uncertainty about what these measures mean. This uncertainty argues for caution in drawing claims about the success of school-reform efforts based on these data.

Joel Klein, on the other hand, created an accountability system in New York City that treated test scores and graduation rates as virtually the only outcomes of schooling worthy of attention—ascribing in the process a precision to these numbers that is unwarranted. It makes one wonder what specifically Klein thinks schooling is about and what an educated student ought to know upon leaving high school. I consider myself a pretty careful observer of Klein and his record, but I cannot recall ever having seen a detailed account in his own words of what students should know and be able to do.

Why am I ambivalent about the New York State tests in reading and math? Over the past several years, there has been a sharp divergence between trends on federal tests and those reported by New York State. The New York State Department of Education acknowledged last year that state tests in reading and math administered in grades 3-8 became easier and more predictable over time, calling into question their ability to measure what students have learned. When the state recalibrated the scores, most of the “gains” of the previous eight years vanished.

And while Jim Kemple’s analysis found that proficiency rates on state tests rose faster between 2002 and 2010 in New York City than in other parts of New York State, those gains were rooted in a set of tests and proficiency cutoffs that the state of New York has asserted were too easy and too predictable. I’m reluctant, then, to treat this as strong evidence of meaningful academic growth during the Klein era.

As for high-school graduation rates, it’s true that these rates have been rising rapidly in New York City. Whereas 47 percent of the entering ninth-grade class of 2001 graduated from high school in four years, 65 percent of those entering ninth grade in 2006 did so. And the gains observed for black and Hispanic students were especially large, rising from 40 to 61 percent, and 37 to 58 percent, respectively. Taken at face value, these patterns suggested that school achievement had risen steadily and the racial/ethnic achievement gap had shrunk substantially.

But consider that three out of every four New York City high-school graduates who enter the city’s community colleges require remediation in basic skills—and that fully 50 percent of those who enter the CUNY system, including its senior colleges, need remediation. One wonders, then, whether completing high school in New York City truly represents a high level of academic accomplishment.

Not according to the state of New York, which is in the process of phasing out a tiered system of diplomas that distinguished among Local Diplomas, Regents Diplomas, and Regents Diplomas with Advanced Designation. Whereas all students have been required to accumulate 44 credits in various subject areas in high school, the type of diploma awarded has rested on performance on the Regents exams. A Local Diploma could be earned by achieving a score of 55 or higher on each of five Regents exams in English, mathematics, science, U.S. history and government, and global history and geography. A Regents Diploma had a higher threshold of performance; students could earn a Regents Diploma by scoring 65 or higher on each of these five exams. An Advanced Regents Diploma required a 65 or higher on additional examinations in mathematics, science and a language other than English. The gains in the graduation rate observed in New York City have been more pronounced for the less rigorous Local Diploma, which is the one being phased out.

The accountability system that Joel Klein imposed on high schools created incentives for awarding credits to students who hadn’t demonstrated mastery of course content. Regents exams—scored by a school’s own teachers—have shown a peculiar bulge in performance at the threshold score of 65, suggesting there are pressures to pass students just below the cutoff score. This casts serious doubt on whether a New York City high-school diploma is a reliable indicator that a young person is prepared for life after high school.

But scoring a 65 is not a high level of performance on the Regents exams, because many students scoring at this level are subject to remediation in college.  This year, the New York State Board of Regents released a new measure of college readiness characterized as an aspirational performance measure.  It tabulates the number of students from each entering ninth-grade high school cohort who scored high enough on the Regents exams to predict placing out of remedial courses or earning at least a C grade in college-level courses at two- and four-year institutions in New York City and around the state of New York.

How do the black and Latino youth in New York City’s high schools, who make up 70 percent of the system’s enrollment, fare on the state’s measures of college readiness? You won’t find this number in any of the PowerPoint decks prepared by the New York City Department of Education, but it’s easy enough to calculate from publicly available data. For black and Latino students who entered high school in 2006—and who thus had the benefit of four years of elementary and middle school, and all four years of high school, under the policies and practices of Joel Klein’s administration—the proportion emerging ready for college in 2010 was 12 percent.

Twelve percent of black and Latino students graduating from high school ready for college—after eight years on Chancellor Klein’s watch.

Proven reforms? You be the judge.