Understanding the Fictions of School Reform: The Metrics Machine & The False Proxy
Randy L. Hoover
One of the quickest ways to be viewed as a whacko is to suggest that there is a major conspiracy behind something. In those very rare instances when a conspiracy can be proven, getting people to examine the evidence and the motives for themselves is often extremely difficult, especially when they are educators. If I have learned anything in my many years as an educator, it is that most educators, while wonderful people they are very naïve about the world of the political. The notable exception is when our unions sound warning alarms of political danger for the profession, teachers will respond in force with great dedication because they see their union as credible in being vigilant against threats to the profession. Sadly, the alarms have been silent at times when they should have been blaring.
The short of it is that understanding the two central issues in this paper is fundamental to understanding the entire big-picture view of what has happened to teachers, students, and administrators as a direct result of the highly political school reform laws and policies enacted since No Child Left Behind (NCLB) was enacted in 2001. The metrics machine and the false proxy discussed herein have a means-ends relationship carefully set up to make public schools and their educators appear to be the political enemy in the fight to improve public education in the United States. Public schools and their educators are not the enemy nor are they the problem. This paper is intended to delineate that what has happened through school reform legislation is a highly calculated plan to demonize teachers, to destroy teacher unions, and to privatize the functions public schools via charter schools and various forms of vouchers. The reform legislation is also carefully calculated to divert millions of state tax dollars to private corporations such as Pearson via the standardized tests and related curriculum materials.
The time is long overdue to expose the insidious nature of educational reform policy and to restore the dignity and centrality that public school educators, especially classroom teachers have earned and well deserve. At the heart of the reform system is a very sophisticated mechanism of metrics that produces a proxy for actual reform. The problem is that the proxy is a false proxy—it does not reveal anything about school or teacher effectiveness that is accurate or credible. The fuel for the metrics machine is, as we all know too well, is the standardized test scores. Together, the metrics machine and its false proxy have sold our citizenry, indeed even many educators a bogus sense of educational accountability. The first step in righting this wrong is for educators to understand how the game has been rigged.
The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.
—Donald T. Campbell, Assessing the Impact of Planned Social Change. (1976)
The comment above speaks precisely to the problem of how a metrics machine and the numbers it produces function in education reform. The numeric indicators of school and educator effectiveness have completely distorted the reality of public education and have intentionally corrupted the process of constructive change. We have reached the point in Ohio at which anything about schools and their educators that cannot be expressed as a number, as an explicit simplistic quantity, is neither important nor of any value.
For educators or anyone else to begin to understand the reality of school reform as it currently exists requires first and foremost understanding the insidious nature of the data and manipulation procedures used as the means to represent performance at every level—the metrics machine and the proxies it produces. A proxy, in this sense, is a number used to represent the value of something in place of something else. For example, in Ohio, the state uses the percent of students enrolled in the federal lunch subsidy program as a proxy for the level of wealth-poverty for a particular school district. Another proxy example might be using the unemployment rate as an indicator of the economic well being of a nation. While neither of the proxies used in these two examples is fully precise, they do serve to provide somewhat credible ideas of reality.
In both examples, data are collected and run through a system of metrics to produce the numeric proxy used to represent information to the public. In other words, the metrics system is a numbers machine that produces a quantitative social indicator used as a public rationale for decision making. The metrics machine produces the proxy for allegedly real-world conditions. The central issue, often completely ignored or hidden, is the credibility of the proxy in accurately representing that which it claims to represent.
Just as is true for virtually all quantitative social proxies, the reality is that most social phenomena are qualitative concepts not easily amenable to quantification. In the case of school accountability, the complexity of schooling, from the purpose of schooling to the intricacies of teacher performance, is impossible to quantify with any accuracy. Aaron Regunberg (2012) says it well when writing about the false proxy:
Unfortunately, the world of education is not a simple one. Schools are incredibly complex places, and school districts are more complicated still. Teaching is an intricate art, and learning is not such a straightforward task either, so it's tough to come up with straightforward but comprehensive measurements in any or all of these areas. What's more, the very purpose of education remains an incessantly dense and knotty question that we seem reluctant to ask, let alone answer.
All educators understand this at least at the intuitive level. Yet amid the sound and fury of the bully politics of school reform, the reduction to absurdity of schooling, teaching, and learning complexities into a standardized test score is indefensible on many levels, especially those of common sense and established mathematics.
But the decision to reduce elements that are irreducible is a human decision, not a mathematical decision. The decision to create a false proxy tells us that either those in power making the decision are ignorant and oblivious to the real complexities of teaching and learning, or those making the decision have ulterior motives for doing so. As educators, we need to begin asking the question of who is advantaged by the false proxy system and who is disadvantaged? A hint to finding the answer is to follow the money trail.
A standardized achievement test score is nothing more than just a score on a test that has little, if any, validity, reliability, or credibility in reference to any real-world school phenomena. These test scores and the metrics machine they are fed into create an arbitrary goal for all schooling and teaching: Raising test scores. In doing so, any value as legitimately accurate indicators is lost, and the entire process of schooling is completely distorted in highly undesirable ways. The power of the false proxy supplants every other aspect of teaching and learning for empowerment in a democratic society.
To understand how the hijacking of public schools by corporate special interest groups and political ideologues who wish to divert millions of taxpayer dollars in Ohio alone to private bank accounts and create a nationalized system of schooling respectively, we need to highlight the means used to convince the public that what they are doing is necessary and beneficial for all. The understanding also requires at least a basic willingness to accept the insidious nature of school reform starting with the Reagan administration and continuing right through into the Obama administration. For the reader who is not a student of political reality, suffice it to say that school reform, for the most part, is driven by corporate greed, but with a good bit of right-wing anti-teacher (Tea Party) extremism added to the mix. As every educator in Ohio knows only too well, the political animosity against Ohio's public school educators has dominated the actions of our governors and legislatures since George Voinovich was in office. John Kasich has only added his bloviating zeal to the anti-educator and anti-public school mentality.
The baseline reality for the issues of the metrics machine and the false proxy is that the metrics machine is intentionally designed to produce numbers used to serve the ends of making public schools and their educators look like failures in need of heavy-handed government punishments. The metrics machine is the means by which a false proxy is created, enabling the public to be sold totally indefensible information about how bad schools and their educators are in order to garner support for the horrendously ill-informed and mean-spirited policies in the name of reform. These policies entrench corporate profitmaking by legitimizing the need for charter schools, vouchers, school choice, testing and testing's associated materials (e.g., Pearson's products). The worse public schools and their educators are perceived to be, the easier it is subsequently to enact legislation and create policy that empowers corporate special interests to milk millions in taxpayer money from the state, most of it directly from the funds of local school districts.
The metrics system is the heart of the system of pseudo accountability, because the proxy numbers it creates are accepted as indicators of actual school performance and effectiveness by the public no matter how demonstrably contrived and invalid they are. What was once a system of high stakes consequence primarily for students has been morphed into high stakes sanctions aimed more at our public schools and the educators who staff them. The mandated use of standardized achievement test scores as the basis for reform has taken on a life of its own, as seen clearly in the pseudo accountability demands of Ohio's value added evaluation requirements. The test scores and the metric-produced claims flowing from them that are used to rate and rank schools and educators have become a false proxy supplanting any and all constructive reform.
The notion of the false proxy originally appeared in a blog by Seth Godin (2012), a marketing guru and entrepreneur:
Once you find the simple proxy and decide to make it go up, there are lots of available tactics that have nothing at all to do with improving the very thing you set out to achieve in the first place. When we fall in love with a proxy, we spend our time improving the proxy instead of focusing on our original (more important) goal instead.
For our fight to recapture the vitality of public schools and to reestablish the centrality of a remarkably high quality teaching workforce, we have to recognize that the false proxy in any of its forms represents an overly simplistic portrayal of schools and educators, having nothing to do with actual performance at all. The more unrelated the proxy is to the phenomenon of actually improving teaching and schooling, the more it becomes a false proxy. But because it is so easily understood by the public, the false proxy numbers become a complete surrogate for rational reform dominating the entire public focus, even though it is demonstrably invalid conceptually and mathematically.
Though they are few, even some educators have bought into the false proxy, something The Teacher Advocate project is intended to help correct. Arguably, our unions have failed totally to sufficiently educate memberships about the realities of the metrics machine's false proxy just as they have failed to deal with exposing the elephant in the middle of all school reform—the failure of the tests to be valid assessments.
From the standpoint of the most patently absurd and most mathematically irresponsible false proxy, Ohio's value added model (VAM) is by far the most egregious element of the accountability system. Coldly calculated to promote the demonization of knowledgeable and dedicated classroom teachers, VAM has become the false proxy de jour of the reformists in Ohio as well as in the Obama administration at the federal level. Ohio's VAM is fraught with violations of several established statistical principles. Among them, Ohio's VAM violates the strict requirement that data used for VAM be from demonstrably valid tests, which is a fatal flaw to start with. It also fails to use student randomization properly, fails to use sufficient years of performance data (even though invalid) by the teacher, and fails to expose its entire calculus publicly because it is proprietary knowledge and therefore not open to public or professional scrutiny.
Such is the case that as a false proxy, VAM is central to Ohio's teacher evaluation system (OTES). When we add the state requirements for student learning outcomes (SLO) and add teachers having to create their own tests where no vendor tests are available in order to produce value added scores, we have created a false proxy for educator evaluation that is monumentally invalid, unreliable, and not worthy of an iota of credibility from any honest citizen. Yet entire professional careers are at stake in the VAM results.
In terms of our educators, the state mandates and requirements flowing from the false proxies have removed virtually all professional decision latitude from teachers and rendered them powerless to do what they know is right for the students they teach. School administrators are forced into the role of the evil middlemen as they force and enforce policies, programs, and procedures (often purchased from profiteering vendors) supposedly designed to increase test scores. These layers of mindless paperwork and arbitrary procedures imposed on teachers by the state and by local districts desperate to raise test scores all stem from the contrived metrics system that churns out the numbers of the false proxy for reform. The standardized test scores feed the entire system. And in this sits the evidence that ultimately one false proxy, the test score, drives the entire system. In doing so, it compounds the error of its nature.
Given what we now know about the false proxies, the end result leaves us with a better understanding of an inescapable conclusion of malevolence being wrought on all those associated with public schooling in Ohio. Ultimately, the false proxy with its rigged metrics threatens school closings, fuels voucher and charter school expansion at the expense of public schools, demoralizes intelligent, dedicated educators, and completely misleads the citizens. But what the gaming of the system has not done is to secure an empowering curriculum affording Ohio's public school students the opportunity to learn: how to think with and apply the subject area knowledge and concepts to real world problems and issues.
Likewise, the system also denies our students any and all opportunity to experience and learn ideals of social justice, principles of equity, or critical reflectivity, the most important human qualities for empowering citizens in a democratic society. Indeed, because the reform movement proxy is rooted so deeply in test scores, students are denied the opportunity to learn the value of education in any sense of the word. The goal of schooling is now solely reduced to doing well on the tests—no more and no less.
A Graphic Example of How the False Proxy Misrepresents Performance
A good example of how the false proxy misrepresents school performance and misleads the public about the nature of school performance can be seen graphically in the comparison of Figure 1 and Figure 2 below. Figure 1 represents the performance of Ohio school districts on the tests in 2007 (percent passing) using the average lived experience level of the district students. The graph shows the high correlation between the lived experience conditions and test performance (r = 0.78). Each data point represents an Ohio school district.
The scatter plot in Figure 1 graphically shows district performance being a function of the socioeconomics of the districts—the red regression line is angled upward to the right. It is clear that schools in the lower range of the Lived Experience Index (LEI) perform worse than those in the upper range. The critical question that emerges when examining Ohio's false proxy is whether the schools in the lower range perform as they do because the educators in those districts are not good professionally or because poverty has a significant impact on student achievement. Likewise, we can ask whether students of wealth perform as they do because they have better educators or because the advantages of wealth are better preparation for passing the tests. Ohio's false proxy insists that the answer to both questions is educator performance, not poverty or wealth, that determines test performance. The state's metrics machine does not in any way allow for the effects of wealth-poverty in considering test performance. The inference of the false proxy produced by the metrics machine stands in direct contradiction to volumes of educational research. Even at the commonsense level, Ohio's false proxy claim is not worthy of belief.
To accept Ohio's proxy claim means that we must accept that it is a coincidence that the apparent performance of Ohio's educators ranges consistently from the most effective being employed at the wealthiest populated districts through the moderately effective at the middle class districts on down to the least effective in the poorest populated districts. To support a claim such as this requires our believing that somehow teachers are miraculously predestined for employment according to their teaching ability.
In Ohio, as in many other states, the false proxies are carefully crafted so that there are no schools with poorly performing students, only schools with poorly performing educators. As we have shown in the se ction on Commonsense Revelations, for this statement to be true, we must accept the assumption that all students at any given grade level are identical in ability, background, intellect, developmental level, and attitude toward school. This assumption is demonstrably false to everyone, yet it is a fully operational premise in Ohio's system.
All of my professional experience and all of my research lead me to make the following statement with absolute certainty: If we were to switch the educators of any high performing school district with the educators of any low performing district using Ohio's reporting metrics, the standardized test results would not change for either district even over many years. Indeed, I would add that it is likely the test scores in the historically lower performing district would actually drop slightly, because the former educators of the high performing district would be inexperienced dealing with the wide and vast needs of the children of poverty.
Figure 2 is another graphic representation of the distribution of district scores, but in this graph, the primary factors of socio-economics in the form of the Lived Experience Index are controlled for mathematically. The result is a very different looking scatter plot from the graph in Figure 1. That the shape of the scatter plot in Figure 2 is conspicuously different from the one shown in Figure 1 is significant because it represents a more random distribution—an approximately equal number of average-, higher-, and lower-performance districts across the range of socio-economic levels.
Though perhaps not immediately obvious, the more random distribution shown by this scatter plot yields a look at district performance that accommodates both commonsense and statistical analysis. Commonsense tells us that in any statewide system of school districts, some districts should perform better than others with some performing as expected and some performing worse than expected regardless of their socio-economic level. This is exactly what the statistical analysis shows, thus confirming what commonsense logic tells us.
The importance of the second graph for understanding the powerful design of the false proxy comes when we realize that many districts that are rated and ranked extremely low by Ohio's metrics, such as shown in Figure 1, are shown to actually be extremely high performing in spite oftheir low socio-economic conditions. The same phenomenon is also true of wealthy school districts in that many of them rated very high by Ohio's metrics are actually performing significantly below average when we control for lived experience. The point here is that performance varies widely depending on the particular proxy (false or otherwise) being used. It is also important to realize that the second graph, the one controlling for lived experience of the students, is also a false proxy for performance. Although the results shown for district performance are vastly more credible than the Ohio ratings (Figure 1), the results are still based solely upon the standardized tests.
The insidious nature of a false proxy's power to misrepresent school performance and to mislead the public can be seen in this example from the first empirical study I conducted to examine the performance of 593 Ohio school districts across 16 different proficiency tests required by the State of Ohio (Hoover, 2000). Using Ohio's performance system, Youngstown City Schools ranked dead last in performance among all 593 school districts. However, when the data were examined controlling for the socio-economic level of the districts, Youngstown City Schools performed within the top 9% of Ohio's school districts. Indeed, Youngstown schools could have been rewarded for exemplary performance using this particular metric. However, given that Youngstown had the highest poverty rate in Ohio doomed the outstandingly effective work of its educators and students to be judged as unacceptable by the State. This tremendous disparity in rank speaks to how effective Ohio's false proxy is in grossly misrepresenting actual performance. (NOTE: As time permits, I will be adding several more examples of similar misrepresentations of district performance across middle class, upper class, suburban, and rural Ohio school districts.)
As long as states like Ohio continue to ignore the effects that student living conditions have on achievement test scores, there will be gross misrepresentation of student, school, and educator performance across all socio-economic levels, lower class, middle class, and upper class equally. The tremendous incongruence between real and actual performance created through this particular false proxy is extreme in its misrepresentation to any and all stakeholders. But the consequences are highest for Ohio's dedicated public school educators—their careers are at stake.
Campbell, D.T. (1976). Assessing the impact of planned social change. Hanover, NH: Public Affairs Center, Dartmouth College.
Godin, S. (2012). Seth's Blog. Retrieved from http://sethgodin.typepad.com/seths_blog/2012/11/avoiding-the-false-proxy-trap.html
Hoover, R. (2014). The neoliberal metrics of the false proxy and pseudo accountability. The phenomenon of Obama and the agenda for education: Can hope audaciously trump neoliberalism? Carr, P. & Porfilio, B., Eds. 2nd edition (in press). Charlotte: Information Age Publishing.
Hoover, R. (2014). The Pseudoaccountability of school reform: Injustice by (false) proxy. In, P. Carr et al (Eds.), Social context reform: Equity and opportunity—not accountability—in education reform. Routledge Press.
Regunberg, A. (2012). Education's false proxy trap. Retrieved from http://www.golocalprov.com/news/aaron-regunberg-educations-false-proxy-trap