Last month, the National Education Policy Center released a new report: Big Claims, Little Evidence, Lots of Money: The Reality Behind the Summit Learning Program and the Push to Adopt Digital Personalized Learning Programs. It looks at one of the most prominent digital learning platforms, and how money and power are able to push such programs despite any real evidence that they work.

Summit Schools started out in 2003 with a low-tech focus on personalized education; in 2014, Mark Zuckerberg discovered the school and decided to gift it not just with money, but with technology. Zuckerberg was fresh off a high-profile edu-failure in Newark, and he had gleaned one particular lesson from that:

The most important lesson we've learned is to focus on problems we have some unique ability to help solve.
When the Chan-Zuckerberg Initiative took form, its educational focus was on digital personalized learning. And Summit seemed like the perfect vehicle for that push. Summit Schools created the Summit Learning Platform, an algorithm-driven software system that delivers lessons to students via computer. Human “mentors” are on duty nearby to help out, but the program is the teacher.

Not everyone has loved it. Parents have occasionally revolted. The program has been accused of racism. But the program, offered free of charge, has spread to about 400 schools, making it one of the most successful digital platforms out there. Then, in 2018, Summit spun the digital program offinto a non-profit entity whose initial four-person board included Diane Tavenner, Summit founder; Priscilla Chan; and Peggy Alford, the CFO for CZI. 

Given that Summit is now widely used, seen as a model for personalized digital learning, and operating under the wing of a top US tech billionaire, it seems worthwhile to look under the hood. The NEPC, a non-profit education policy research center located in the School of Education at the University of Colorado at Boulder, has done just that.

Here are some of the major findings from the report.

The first is how “reluctant, and in many instances, unwilling to provide basic information about its educational program and platform” Summit turned out to be. The 2018 non-profit (T.L.P. Education) now operates as a kind of cloak of secrecy over many aspects of the operation. The experience of the researchers echoes the experience of parents, who often find Summit unresponsive. NEPC found the staff “unfailingly polite, but nonresponsive.” Requests were ignored, side-stepped, or greeted with some version of “we don’t have any information about that.”

Summit has constructed an image as a successful program. It repeatedly claims to be “evidence-based” and “grounded in science,” but it has never allowed an independent evaluation of any aspect of its product. Summit’s own self-promotion depends largely on anecdotes and testimonials. And some of its claims stretch credulity; it has said that 100% of its students are “eligible for a four-year college,” but no Summit charter school has ever graduated 100% of its senior class. Summit also claims that its students graduate college at twice the national average, but told NEPC that it has no records related to these claims. 

Summit rejects the notion that standardized tests can measure the cognitive skills that they claim to prize, and they are absolutely correct to do so. But as NEPC notes, Summit’s own Cognitive Skills Rubric seems not to have been checked for either validity or reliability.

Many of Summit’s claims seem more like the puffery of marketing than the rigor of science, with the public record providing no support and Summit either unwilling or unable to provide evidence. But marketing, NEPC finds, is a big part of Summit’s success. It has attracted money and support from many major players, including the Gates Foundation, Silicon Schools Fund, and XQ Institute (the ed reform project of Laurene Powell Jobs). In 2015, Summit made a deal with Facebook to enhance software and develop a nationwide marketing strategy. This has included a 2017 publication, The Science of Summit, ”which purports to show that SPS’s pedagogical approach is research-based.” NEPC finds the report offers no actual research evidence.

Summit’s marketing also leans heavily on the non-digital aspect of the program. “Your child’s education will be delivered via computer screen,” is not a winning sales pitch, and so Summit emphasizes other aspects. The “free” part is a big hit, particularly when linked to the success of the original Summit charter schools. This is a marketing approach unique to tech-based charters—”You can’t send your child to Super Tech Charter High, but now we can offer you practically almost kind of the same sort of experience in a software package.” This ignores the importance of local school culture, a factor that as yet cannot be loaded into software.

NEPC finds one other major concern with Summit’s digital program. Anything managed digitally can be collected digitally. Summit promises to collect and analyze a great deal of data in order to “personalize” the student’s experience, but that means that the program collects a great deal of personal data, and CZI has access to that data in perpetuity. As NEPC observes, “It is important to note that the Chan Zuckerberg Initiative (CZI) is not a charity or a philanthropic organization; it is a business.” It’s probably also worth noting that old internet wisdom, “If you aren’t paying for it, you’re the product being sold.”

Computer software often carries the illusion of objectivity, but as NEPC correctly points out, software is written by humans, and any algorithms carry the stamp, the biases, the ideology of the people writing them. Exposing a student to a program like Summit’s is like sending them to a school where they never meet the teachers and families are never allowed to know who designed the education program or the principles that guided them. 

NEPC’s conclusion is direct:

Our analysis suggests that, rhetoric notwithstanding, the Summit Learning Program does not deliver on its promise to provide a higher quality education, with superior student outcomes, in the schools that adopt it. Moreover, aside from any valid education purpose, the Summit Learning Platform approach to assessment, coupled with enabling contract language, opens the door to the transfer of large amounts of student data to third parties without oversight or accountability.

A well-marketed nothingburger. All hat, no cowboy. All sizzle, no steak. Choose your favorite metaphor; this NEPC report suggests you should not support Summit’s digital education program.

Shortly after the report came out, Summit (which had previously been just somehow unable to really respond to NEPC) put up a blog post as rebuttal to the report. NEPC replied to that reply, and without getting into the nuts and bolts (you can find it all here if you wish), they rightly pointed out that Summit's reply was simply more of the same-- not transparent, and leaning on assertions rather than actual evidence.