As the new year begins, I’ve been pondering what I might recommend as guiding principles for the path forward for education policy in New York City under its new Mayor, Bill de Blasio, who is often referred to on Twitter as BDB. So here are my thoughts for the way forward, from one BDB (Bruce D. Baker) to another.

Note that I had drafted much of this content last spring when convening with a group of scholars to discuss the path forward for NYC education policies. Not being as well versed in the specifics of NYC education policies, but having at least written academically about some, I kept my ideas broad, and applicable to many educational settings across the U.S.

My recommendations fall into two broad categories:

Develop a robust, balanced, least intrusive system of indicators for evaluating New York City Schools and then use that information appropriately

NYC BOE policies of the past ten years have been rife with data abuse (though at times, merely in an effort to comply with state required data abuse). School closures have been based on ill-conceived measures of “school failure” which do little more than target the city’s neediest student populations, imposing on them repeated disruptions.

New York City’s teacher performance reports, albeit better than many, apply the worst form of statistical reductionism to quantify teacher “quality,” taking noisy statistical estimates of the association between teachers-of-record and assigned students test score gains (applying only the most convenient statistical corrections) in limited curricular areas and grades, and assuming levels of precision and accuracy that are completely unwarranted.

Such data abuse – on both counts [school closures and teacher ratings] – is reprehensible.

Right-sized (NOT BIG) data can indeed be useful for guiding decision-making in large, complex urban education systems. But data should never be the exclusive determinant of policies or other high stakes decisions.

Human judgment matters, including human interpretation of the meaning and usefulness of data as it informs decisions which ultimately affect other human beings.

New York City should give serious consideration to how data are collected, maintained and ultimately used for informing policy and decision making. Four guiding principles are:

1. Emphasis should be on understanding what the data can and cannot tell us about schools, their climate, students and their achievement and the role of teachers, leaders, programs and services. Policies should emphasize how various constituents can make sense of data, coupled with their knowledge and experience, to inform the path forward.
2. Data should NEVER dictate decisions. Rather, data may inform them. Along these same lines, despite ill-conceived requirements of state policies, imprecise information (which includes nearly all social science measures) should NEVER be treated as determinative, attaching specific consequences to specific scores or estimates (splitting hairs that cannot or shouldn’t be split).
3. Data systems must better capture the scope of public service that is public schooling in a modern era. This means collecting more than just that which is most easily quantifiable, and more than just achievement test scores on mandated core curriculum.
4. Data should be collected in the least intrusive manner necessary to draw valid inference, or provide valid descriptive profiles.

School principals and leadership teams should have available to them sufficient and appropriate data to guide – Not Dictate – building level management decisions. Information might include typical measures of student achievement as well as measures of gain, but also include measures on longer term outcomes of students who attended any given school (graduation, college attendance/persistence) – linked longitudinally to both outcome data while they were in attendance at the school as well as data on programs and services in which they participated. Data on students should similarly be traceable backwards. Data might also include indicators of parent and student perceptions of school environment, etc.

Data should attempt to capture not only limited, easily measurable “outcomes” but also more accurately measure inputs and resources as well as characterize ongoing educational processes.  After all, a central objective of data collection and maintenance is to be able to make connections between inputs, process and outcomes. And the central objective of city leaders should be to ensure equitable and adequate inputs and processes, to support the achievement of more equitable, more adequate outcomes. Not the other way around.

An important consideration is that data should be collected more strategically so as to be far less intrusive than current practices on the actual educational processes being monitored with the data. That is, the goal of actors within the system should not be to improve the measured data elements, but rather to more substantively improve their practices in ways that lead to shifts in the measured data elements, assuming we’re measuring the right things (often a bold assumption).

Lengthy performance assessments, achievement tests or survey instruments need not be given every year to every child. Appropriate sampling can achieve robust data with far less intrusion (or expense). Such is the design of assessment systems like the National Assessment of Educational Progress. Providing samples of items to samples of students across schools can reduce cost, reduce intrusion, reduce the likelihood of teaching to the test, item familiarity and other threats to validity, and thus provide more useful information. This approach also reduces the digital record maintained on any one student.

As a basic rule of thumb, high stakes decisions should never be made with low quality information.  One should never conclude with certainty based on uncertain information.

The city should avoid the urge to apply categories to otherwise continuous and noisy data – such as applying specific cut points and imposing quality/value judgments based on those cut-points. Few if any measures collected in social science, including test scores from multiple choice assessments given to nine and ten year old children, are sufficiently precise for making high stakes determinations by splitting hairs between getting 20 versus 21 (or even 20 vs 25) correct responses. Most of the types of data collected in such an environment are simply not sufficiently precise for such determinations.

Complex decisions like school closures and reorganization require multiple perspectives and varied forms of information/data. A school should not be closed simply on the basis of one or a handful of bad performance indicators – typical school report card elements. The role of schools in communities should not be completely ignored. Nor should the extent to which the same children’s lives/educations are repeatedly disrupted. More mundane considerations including transportation efficiency & facilities quality/efficiency/fit are likely more relevant than student outcomes when considering school closings. In fact, rarely if ever are “low tested outcomes” a legitimate reason for school closure. Rather, they are usually an indicator of other underlying processes – some non-school and perhaps some school processes – requiring far more thoughtful intervention than the current slash-and-burn approach to “failing schools.”

Similarly, data may inform but should never dictate human resource decisions. Such is the core problem with recently adopted statewide teacher and principal evaluation models that prescribe percentages of evaluations that must be dictated by X, Y or Z, and that require specific personnel actions be taken when the numbers fall into preset categories.

As my colleagues and I explain in a recent article,

Arguably, a more reasonable and efficient use of these quantifiable metrics in human resource management might be to use them as a knowingly noisy pre-screening tool to identify where problems might exist across hundreds of classrooms in a large district. Value-added estimates might serve as a first step toward planning which classrooms to observe more frequently. Under such a model, when observations are completed, one might decide that the initial signal provided by the value-added estimate was simply wrong. One might also find that it produced useful insights regarding a teacher’s (or group of teachers’) effectiveness at helping students develop certain tested skills.

School leaders or leadership teams should clearly have the authority to make the case that a teacher is ineffective and that the teacher even if tenured should be dismissed on that basis. It may also be the case that the evidence would actually include data on student outcomes – growth, etc. The key, in our view, is that the leaders making the decision – indicated by their presentation of the evidence – would show that they have reasonably used information to make an informed management decision. Their reasonable interpretation of relevant information would constitute due process, as would their attempts to guide the teacher’s improvement on measures over which the teacher actually had control.

Put simply, mindless reliance on prescribed metrics is not effective human resource management whether in the private sector or in public schools.

Additional Readings:

• Friday Thoughts on Data, Assessment & Informed Decision Making in Schools. http://schoolfinance101.wordpress.com/2012/12/07/friday-thoughts-on-data-assessment-informed-decision-making-in-schools/
• Closing schools: Good Reasons and Bad Reasons http://schoolfinance101.wordpress.com/2012/02/08/closing-schools-good-reasons-and-bad-reasons/
• “Corporate Reform” or Failed, Desperate Corporate Management? http://schoolfinance101.wordpress.com/2013/07/26/corporate-reform-or-failed-desperate-corporate-management/
• Baker, B.D., Green, P.C., Oluwole, J. (2013) The legal consequences of mandating high stakes decisions based on low quality information: Teacher Evaluation in the Race-to-the-Top Era. Education Policy Analysis Archives http://epaa.asu.edu/ojs/article/view/1298/1043
• Follow up on Ed Waivers, Junk Rating Systems & and Misplaced Blame – New York City’s “Failing” Schools http://schoolfinance101.wordpress.com/2012/09/07/follow-up-on-ed-waivers-junk-rating-systems-and-misplaced-blame-new-york-citys-failing-schools/
   

Balance choice with support for equity & access

The theme of individual (parent/child) liberty via parental choice that has dominated the past decade of education policy in New York City (and elsewhere) must be counterbalanced with a greater emphasis on equity and equality of opportunity and access. This requires considering carefully the geographic and socioeconomic distribution of educational opportunities at all grade levels across all children.

Choice in and of itself does not ensure equity. This false premise promoted by many “education reformers” runs counter to centuries of political theory, which explains that liberty (a core tenet of choice) and equality are at constant tension with one another (only at some extreme point might “meet and be confounded together”Tocqueville, Alexis de.  Democracy in America, volume 2, part II, chapter 1).

Implicit in policy preferences for choice program expansion is the notion that more children should have the choice to attend “higher quality” schooling options and that such options will emerge, as a function of the competitive marketplace for quality schooling with little attention to the level of resources provided or other prerequisite conditions for sustaining an equitable distribution of quality schooling.

The notion that one would provide via public subsidy, “higher quality” alternatives means also consciously providing lower quality ones. That is, consciously endorsing a policy of such inequity that the parents of children presently attending “low(er) quality” schools will endure the transaction costs (family/child disruption, geographic inconvenience) to move their child from their neighborhood schools. This is simply wrongheaded.

Even more wrongheaded are policies yielding outright deprivation by labeling neighborhood schools as failing and shuttering them on false pretenses (low test scores as a method of placing blame), leaving parents to scramble to find an acceptable alternative (one that is merely better than nothing).  Such policies create a false sense of demand for those alternatives (typically charters), further advancing current policy preferences. That is, the argument that charter waiting lists provide validation for further charter expansion, even when those waiting lists have been induced by school closures.

Recent policy preferences are built on the assumption that liberty achieved by choice programs serves as substitute for the provision of broad based, equitable and adequate financing. Studies purporting significant advantages achieved by students attending charter schools have invariably neglected to evaluate their access to financial resources (see also) (while selectively evaluating outcomes of children attending those schools with access to resources), frequently downplaying the importance of money or relevance of equity traditionally conceived.

Advocates suggest that if some children are made better off by the presence of higher quality options, all are better off and certainly no one is worse off.  This too is false. In forthcoming work, I explain that:

Baker and Green (2008) as well as Koski and Reich (2006) explain that to a large extent education operates as a positional good, whereby the advantages obtained by some necessarily translate to disadvantages for others. For example, Baker and Green (2008) explain that “In a system where children are guaranteed only minimally adequate K–12 education, but where many receive far superior opportunities, those with only minimally adequate education will have limited opportunities in higher education or the workplace.” (p. 210) This concern is particularly pronounced in a city like New York where children and families are constantly jockeying for position to gain access to selective admissions public middle and secondary schools, and where the majority of charter schools serve elementary and middle grades. The competitive position of children in otherwise similar district or charter schools with fewer resources is compromised by the presence of better resourced district or charter schools. Though surely, all would be less well off if all were substantially though equally deprived.

Variation in resources across private providers, as well as across charter schools tends to be even greater than variation across traditional public schools (Baker, 2009, Baker, Libby & Wiley, 2012). Further, higher and lower quality private and charter schools are not equitably distributed geographically and broadly available to all. In the most extreme case, in New Orleans following Hurricane Katrina where traditional district schools were largely wiped out, and where choice based solutions were imposed during the recovery, entire sections of the city were left without secondary level options and provided a sparse few elementary and middle level options (Buras, 2011).

Baker, Libby and Wiley show that in New York City, charter expansion has yielded vastly inequitable choices. Most New York City charter school networks serve far fewer children qualifying for free lunch (<130% poverty level), far fewer English language learners and far fewer children with disabilities than same grade level schools in the same borough of the city. These patterns of student sorting induce inequities across schools. But, these schools also have widely varied access to financial resources despite being equitably funded by the city. Some charter networks are able to outspend demographically similar district schools by over $5,000 per pupil, and to provide class sizes that are 4 to 6 (or more) students smaller.

Put simply, one cannot assume that providing a “system of great schools” will necessarily yield an equitable system of high quality, operationally efficient schools. It hasn’t and it won’t, in New Orleans, New York, in Sweden or anywhere. City leaders must actively manage the provision of an equitable, high quality and operationally efficient school system rather than simply assuming that a system of great schools will necessarily accomplish that goal.

Moving forward in the short term:

1. The city should develop more transparent, comparable reporting of district and charter school site-based revenues and expenditures, inclusive of a) private contributions by source and b) in-kind expenditures from parent organizations, including salaries and benefits of centrally employed staff.  More detailed reporting of soft money and in-kind contributions may provide insights regarding policy efforts to improve resource equity between charter and district schools, and among charter schools.
2. All schools operating within the city should be brought under the same policy umbrella to ensure more equitable distributions of students and the resources to serve them. This means financing charter schools in accordance with the student populations they serve, via weighted student funding. This also means considering policy alternatives for balancing resource access across schools, given their widely varied access to private resources.
3. City leaders should push state leaders for the billions in resources stilled owed city school children in the years since the ruling in Campaign for Fiscal Equity.
4. Finally, given the increased organizational complexities of privately governed and managed charter schools, the city should take steps to ensure that children’s and employees’ rights remain equally protected (when compared with their peers in “government operated” district schools).  The choice between private management and public provision of schoolingis not benign and should not be taken lightly.  Increasingly federal and state case law is revealing that children’s and employees rights are substantively lessoned in schools managed and operated by private entities. In addition, taxpayer rights to gain access to records and finances of private providers have also been interpreted by courts as more limited (than access to similar information from government entities).

Additional Readings

• Baker, B.D., Libby, K., & Wiley, K. (2012) Spending by the Major Charter Management Organizations: Comparing charter school and local public district financial resources in New York, Ohio, and Texas.Boulder, CO: National Education Policy Center. Retrieved [date] from http://nepc.colorado.edu/sites/default/files/rb-charterspending_0.pdf
• Baker, B.D., Libby, K., Wiley, K. Charter School Expansion & Within District Equity: Confluence or Conflict? Education Finance and Policy
• Baker, B.D. (2012). Review of “New York State Special Education Enrollment Analysis.” Boulder, CO: National Education Policy Center. Retrieved [date] from http://nepc.colorado.edu/thinktank/review-ny-special-ed.
• Buras, K. L. (2011). Race, charter schools, and conscious capitalism: On the spatial politics of whiteness as property (and the unconscionable assault on black New Orleans). Harvard Educational Review, 81(2), 296-331.