Forces and Factors Affecting
Ohio Proficiency Test Performance:
A Study of 593 Ohio School Districts
Randy L. Hoover, Ph.D.
Department of Teacher Education
Beeghly College of Education
Youngstown State University
Section One: Overview
The following pages contain information, data, analysis, and summary findings regarding a major study of Ohio school district performance on the 1997 Ohio Proficiency Tests (OPT). The data are for 593 of the 611 Ohio School districts. Data for 18 districts were excluded due to either missing test scores or because of their extremely small size such as North Bass Island. A complete list of the districts used in the study and the basic data for those districts may be found in the appendix to this study.
This study examines the 593 Ohio districts on all sections of the 1997 fourth-grade, sixth-grade, ninth-grade, and twelfth-grade tests. Thus, as the outcome measure of district performance, the study uses 16 sets of scores for each Ohio School district. All data used in this study are taken directly from the online Ohio Department of Education's Educational Management Information System (EMIS) of the State of Ohio and have not been derived from any secondary source. The variables examined against the 1997 district test data are also from the 1997 EMIS collection. The data from 1997 were selected for analysis because they are the most recent online data available from the Ohio Department of Education and the State of Ohio. and they are the most complete data available that is easily accessed by the public.
The data were analyzed using linear regression and Pearson's correlation (Pearson's r) procedures. A simplified explanation of the analysis is contained in the next section. However, it is important to point out that the statistical analyses used are very simple an very straightforward in terms of the range of potentially very complex statistical procedures. The statistical operations used in the study are quite typical of those used across many fields and disciplines including medicine, marketing, political science, and economics.
While certain results may call for additional and more sophisticated analysis, the results contained herein speak for themselves and for the power of basic statistical analysis. Further, given the power of the primary results of the procedures and the statistical significance of those results, no additional more complex procedures were deemed necessary to achieve the basic ends of the study.
As with any research of education and social phenomena, there is always room for interpretation and reflective judgment. While this certainly applies to this particular study, the basic finding regarding district-level Ohio Proficiency test performance is remarkably clear: Performance on the Ohio Proficiency Test is most significantly related to the social-economic living conditions and experiences of the pupils to the extent that the tests are found to have no academic nor accountability validity whatsoever.
It is extremely important to know that findings do not single out students and districts in which levels of disadvantagement are high as being the only sector where the test is invalid. The findings clearly indicate that the range of performance across all social economic levels lacks validity in terms of assessing academic performance. Rejection of the findings regarding OPT validity (accepting the State of Ohio's interpretation of OPT results) means that we accept the position that wealth defines academic intelligence, that the wealthier the students the more intelligent than less wealthy students. This position is absurd even at a common sense level; money does not define academic intelligence or learning capabilities.
Part of the problem in understanding OPT for what it is (or is not) rests in understanding that there are many different variables that affect how, what and whether a child learns in school. Explicit in the OPT program and State of Ohio policies on school district accountability is the assumption that these high stakes tests accurately assess student academic achievement and that all students are the same in terms of how, what, and whether they learn. The findings of this study contradict this assumption.
Implicit in the claims and slogans of the those who are using the OPT and Ohio School Report Cards (OSRC) to assess public education in Ohio is the idea that district OPT performance is determined by one variable-- the teacher. Interestingly, the OPT proponents are often using the test more of an indicator of school district and teacher performance than of student performance as witnessed by the force of the Ohio School Report Cards. The results of this study show that neither student academic learning, school district effectiveness, nor teacher effectiveness are validly measured by these tests. Indeed, the findings indicate that OPT results and OSRC ratings are, in most cases, extremely misleading at best.
Contained within the subsequent sections of this study are the primary and secondary findings of the study. Each section covers a particular variable or related set of variables and uses graphs and narrative to attempt to explain the meaning and the significance of the findings being discussed. Though the primary research interest motivating this study is OPT district-level performance, this study would be incomplete without some analysis and discussion of the Ohio School Report Cards since OSRC is driven primarily by OPT district-level performance. Therefore, there is a section dealing with the validity problems of OSRC as related to the primary findings of the study of OPT district performance.
Section 2: Frequently Asked Questions (FAQ)
What did the study involve?
Briefly stated, this research study involved the examination of 593 Ohio school districts across 40 variables using 16 sets of OPT scores for each school district. All data were collected from EMIS online data banks and the data were analyzed using statistical methods such as regression analysis and correlation analysis. Both school and non-school variables were used.
What is the purpose of this study?
The purpose of the study was to attempt to identify both school and non-school variables most significantly associated with district test performance in order to illuminate the degree to which OPT is a valid and reasonable mechanism for assessing school performance in terms of academic achievement and educator accountability. Similarly, an attempt was made to isolate and examine any variables found to be likely significant in contributing to actual district performance.
What is the difference between a school variable and a non-school variable?
School variables are those forces and factors that schools can control and adjust such as class size, per pupil expenditure, and teacher salary among many others. Non-school variables are forces and factors over which schools have no control such as mean family income, property values, and poverty levels among many others.
What are the primary findings?
The study found that OPT district test performance is most strongly connected to the living conditions and the lived experiences of the students in terms of economic, social, and environmental factors. District test performance was found to correlate extremely high with advantagement-disadvantagement: The greater the wealth of the students of the school district, the better the district OPT performance. In this study, the term "Presage Factor" is used to indicate the social, economic, and environmental variables of advantagement-disadvantagement.
What do the primary findings tell us about the OSRC?
The findings also show the Ohio School Report Card to be equally as invalid as OPT performance. This finding is not too surprising when we consider that OPT performance is the primary element that drives OSRC ratings. In other words, if OPT does not carry significant validity, then OSRC will not either because it is primarily a function of district OPT performance.
What exactly is the Presage Factor?
The Presage Factor is a combination of the Ohio Department of Education's online EMIS variables that represent measures of advantagement-disadvantagement. It combines the following EMIS measures: percent ADC, percent enrolled in the subsidized school lunch program, percent economically disadvantaged, and mean family income. These variables are combined in a very straightforward manner using a simple calculus to derive a scaled measure of advantagement-disadvantagement. Section Three gives the precise formula for calculating the Presage Factor.
What is meant by advantagement-disadvantagement?
Advantagement-disadvantagement is intended to represent the continuum of social-economic forces and factors that are indicated by the Presage Factor. They are the forces and factors that shape the lived experience of all children. The knowledge, culture, values, attitudes, and meanings that children bring to school are a largely shaped by their lived experiences. This particular term is not used the same way as the terms "educationally disadvantaged" or "educationally advantaged." These terms refer to how schooling itself, through its practices and processes, is structured to reward or punish students for the knowledge, values, and cultural meanings they bring to school.
What are linear regression and statistical correlation?
Linear regression is used to examine the relationship between two variables such as the Presage Factor and the percent passing the OPT. Basically it allows us to perceive how the change in one set of variables relates to corresponding change in the other set of variables. Statistical correlation then allows us to determine the strength of the relationship between the two sets of variables. The correlation used in this study is called "Pearson's correlation" or "Pearson's r."
It is this correlation that tells how significant the association is between the sets of variables. Correlation analysis yields what is called the "correlation coefficient" or "r." The range of "r" is from -1.0 to 1.0. The closer that "r" is to -1.0 or 1.0, the stronger the relationship between the two sets of variables being analyzed. For example, where r=1.0, the correlation is perfect... where r=0.0, there is no relationship whatsoever. In cases where "r" is negative, the correlation is said to be inverse, meaning that as the value of one variable increases, the value of the other decreases. (See the graph of percent passing and percent ADC for an example of an inverse correlation.) In cases where "r" is positive, as the value of one variable increases so does the value of the other variable.
In social science research, a perfect correlation is rarely, if ever, found. Indeed, correlations approaching either r=-0.50 or r=0.50 are usually considered relatively significant. It is suggested that you consult a good statistics text for better understanding of the details and assumptions involved with regression analysis and correlation. It needs to be noted that the primary finding of this study regarding the relationship between advantagement-disadvantagement and OPT district performance is r=0.80, a significantly high correlation by any statistical standards.
What are residuals?
A residual is the difference between what the linear regression predicts a given value will be and what the value actually is based upon the line generated by the mathematics of linear regression. It is essentially the mathematical distance of a data point above or below the regression line. In the case of this study, district residuals from the Presage Score/Percent Passing regression are used to postulate actual performance. Doing this gives us some idea of performance controlling for the Presage Factor.
What exactly is a z-Score and why use it?
A z-score (often called a "standard score") is a transformation of a raw score into standard deviation units. Using z-scores allows us to immediately know how far above or below the mean is any given score, thus allowing us to visualize how extreme the score is relative to all other scores. The mean of any z-score distribution is always zero. Using z-scores does not alter the distribution of scores in any way and does not affect the analysis or the findings. Converting to z-scores is a linear transformation and does not change the results of the data analysis in any way other than to make the data more understandable.
The advantage of the z-score is found in allowing us to understand one score relative to other scores. For example, the Presage score as a raw score for Youngstown City School District is -173.08, which does not tell us how extreme the disadvantagement is. The Presage z-score for Youngstown is -3.82, which tells us that it is 3.82 standard deviations below the State average, thus allowing us to see that Youngstown's students are very deeply in social-economic disadvantagement.
What exactly is standard deviation?
Most simply put, standard deviation describes how a set of scores is distributed around the mean of the set. For use in this study, basic knowledge of standard deviation is helpful in reading and understanding the z-scores. Z-scores tell us how many standard deviations above or below the mean a score is. Z-scores greater than 1.0 or lower than -1.0 suggest more significant scores beyond those within 1.0 and -1.0. In the case of reasonably normal distributions such as with the data in this study, approximately 68% of the scores will fall within the 1.0 and -1.0 range of the first standard deviation and 95% of the scores will fall within the limits of the second standard deviation. Scores in the third standard deviation may be thought of as being extreme. Thus, the example of Youngstown given above as having a Presage z-score of -3.82 tells us that it is a case of children living in extremely disadvantaged environments.
How significant or powerful are the findings?
The correlation between the measure of advantagement-disadvantagement (Presage Factor) and OPT performance are extremely high (r=0.80). Indeed, these findings about this relationship are about as high as are ever found in social science research... the findings are very significant both statistically, conceptually, and practically.
Can OPT scores be raised through school interventions?
The question as to whether OPT scores can be raised can certainly be answered in the affirmative, though it is not considered within the study. However, any educational imperative to raise scores must not be based on an invalid test nor must it be directed toward any form of high stakes testing. Instead, it must be driven by the vision of empowerment, the idea that what students are taught in schools must be personally experienced by the students. Knowledge must be taught in such a manner that it is felt as relevant and usable in the mind of the learner. To empower learners requires constructing learning activities that become personally felt lived experiences for the students in the classrooms, not abstract rote exercises over facts and ideas that the students perceive as meaningless and irrelevant. The usability of academic knowledge must be taught by the teachers and must be experienced by the students if we are to empower learners and raise scores significantly.
What do the findings tell us about the validity of the OPT as an assessment of academic achievement?
The findings tell us that OPT performance is in no manner a valid measure of academic achievement: The OPT measures almost exclusively only the quality of life in which the students of the district live.
What do these findings suggest about the validity of the Ohio School Report Card?
The findings tell us that the Ohio School Report Card, because it is almost entirely based upon OPT performance, is a totally invalid assessment of actual school district performance and should not be used. OSRC is extremely misleading, and the general public should be outraged about its use. Likewise, the State Legislature and Governor should be held accountable for misleading the citizens of Ohio and using state monies for such an invalid assessment of school district performance.
What do the findings tell us about accountability on the part of districts, administrators, teachers, and Ohio's public school pupils?
Accountability is the least understood term in the American political lexicon. For true accountability to be invoked, we must understand that valid accountability is a function of the decision latitude and amount of performance control vested in those being held accountable. In other words, it is wrong to hold districts, administrators, teachers, or students accountable for a test that measures variables over which they have absolutely no control. This study finds beyond the shadow of any doubt that the OPT is not a measure of virtually anything related to in-school variables; it is a measure of non-school variables, forces, and factors. Therefore, to hold those associated with schools accountable for OPT performance is absurd and wrong. It is tantamount to holding the TV weather person accountable for today's weather.
From this study, is it possible to assess with some degree of validity the actual levels of Ohio school district performance?
The answer here is both yes and no. It is "yes" in terms of knowing that the Presage Factor is so very powerful that if we control for its effects, we begin to get a much clearer and certainly much more valid picture of how each district is actually performing. It is "no" in the sense that this performance even controlling for the Presage variables still is primarily based upon the OPT itself. To assess school district performance using the OPT would be foolish and wrong in that it is the public school student who suffers most from the test. In other words, why hurt and mislead the children and parents of Ohio to assess district performance using an invalid instrument.
When will copies of the findings be made public?
The study itself was officially released to the public and media as of 12:01 AM, February 27, 2000. On April 27, 2000 the findings will be presented at Ohio's Teaching Learning Conference 2000 in Columbus. The presentation is scheduled for 8:45 am, tentatively in Room C214 in the Columbus Convention Center. Copies of the final study will be available at this presentation and will be available online March 1, 2000 at http://cc.ysu.edu/~rlhoover/OPT.
The Primary Findings:
Advantage-Disadvantage as Predictor of District Performance
The fundamental purpose of this study was to examine what forces and factors may be affecting district-level performance on the Ohio Proficiency Tests and to attempt to determine to what degree these variables shape district-level performance. To this end, two categories of variables were used: school variables and non-school variables. School variables are those forces and factors that schools can control and adjust such as class size, per pupil expenditure, and teacher salary among many others. Non-School variables are forces and factors over which schools have no control such as mean family income, property values, and poverty levels among many others.
As briefly discussed previously, the primary finding is that OPT performance is affected most significantly by non-school variables representing the lived experiences of the children attending the school district. The lived experiences of children come from and happen within the advantagement- disadvantagement of their environments. These experiences of real-life are non-school variables that clearly shape how, what, and whether a child learns.
The term "Presage Factor" was chosen to indicate the data used collectively as a measure of the non-school variables that serve as the indicator of the degree of advantagement- disadvantagement experienced in the lives of the district's school children. The term was chosen because the word "presage" means to predict, foresee, or foreshadow, which is what knowledge of basic living conditions within the district allows us to do with OPT performance when we can mathematically quantify elements of those basic living conditions.
The graph below shows the power of the Presage Factor as a measure of advantagement- disadvantagement in predicting district OPT performance. The "Y" axis represents the mean percent of a district's students passing across the four sections of the 4th, 6th, 9th, and 12th grade 1997 Ohio Proficiency Tests: %Passing = [(%4Math + %4Reading + %4Writing + %4Citizenship + %6Math + %6Reading + %6Writing + %6Citizenship + %9Math + %9Reading + %9Writing + %9Citizenship + %12Math + %12Reading + %12Writing + %12Citizenship)/16].
The "X" axis represents the Presage Factor expressed in raw scores. The presage score is a measure of the degree of social-economic disadvantagement-advantagement derived from EMIS data that combines the percent of the student population of the school district for Aid to Dependent Children, percent enrolled in the Free or Reduced Lunch Program, percent listed by the State of Ohio in Economic Disadvantagement, and Mean Family Income. The formula or algorithm for the Presage Factor is: Presage Score = (Mean Family Income/1000) - (%FreeReducedLunch + %ADC + %EcoDis).
From the data analysis represented in the graph below, we find that performance across the 593 Ohio districts included in this study is associated with non-school environmental conditions of advantagement-disadvantagement to the extent of r = 0.80. This is an extremely high correlation and clearly brings the validity of OPT into serious question.
Interpretation of the correlation coefficient of r=0.80 tells us that, conservatively, the non-school related effects of advantagement-disadvantagement defined by the Presage Factor determine 64% of OPT performance. It is important to note that this 64% determination is restricted to the effects of the Presage Factor and, by definition, does not include other possible advantagement-disadvantagement effects outside the realm of those included in the Presage Factor.
Indeed, the idea that advantagement-disadvantagement limited to the scope of the Presage Factor determines 64% of OPT performance is a conservative interpretation of the overall power of social-economic living conditions because it may well be excluding other significant non-school forces and factors. There is a real possibility that there are still social-economic effects beyond the range of those comprising the Presage Factor, though extremely powerful in its own predictive power. For more possible insights to additional non-school variable effects beyond those within the scope of the Presage Factor, see the sections on "Actual District Performance: Controlling for the Presage Factor" and "Percent African-American and Percent White as Variables Across Presage Score, Percent Passing, and Actual Performance."
Because of the discovery that OPT performance is overwhelmingly determined by the social-economic living conditions that the students of the district experience growing up, the inescapable conclusion is that OPT is not a valid measure of either school or teacher effectiveness and should not be used for accountability assessment. The OPT is invalid because the results of this study show that it does not measure what it claims to measure: Student performance on the OPT is, at best, academically meaningless. It is highly biased against economically disadvantaged students and highly biased in favor of economically advantaged students.
Using z-Scores for Graphs:
A z-score (often called a "standard score") is a transformation of a raw score into standard deviation units. Using z-scores allows us to immediately know how far above or below the mean is any given score, thus allowing us to visualize how extreme the score is relative to all other scores. The mean of any z-score distribution is always zero. Using z-scores does not alter the distribution of scores in any way and does not affect the analysis or the findings. Converting to z-scores is a linear transformation and does not change the results of the data analysis in any way other than to make the data more understandable.
The advantage of the z-score is found in allowing us to understand one score relative to other scores. For example the Presage score as a raw score for Youngstown City School District is -173.08, which does not tell us how extreme the disadvantagement is. The Presage z-score for Youngstown is -3.82, which tells us that it is 3.82 standard deviations below the State average, thus allowing us to see that Youngstown's students are very deeply in social-economic disadvantagement. Likewise, the presage score for Indian Hill Exempted School District is 164.76, a figure that alone tells us little about the meaning of the score. However, the z-score for Indian Hill is 4.37, which tells us that it is a very advantaged district.
Most simply put, standard deviation describes how a set of scores is distributed around the mean (average) of the set. For use in this study, basic knowledge of standard deviation is helpful in reading and understanding the z-scores. Z-scores tell us how many standard deviations above or below the mean a score is. Z-scores above the mean are positive numbers and those below are negative numbers.
Z-scores greater than 1.0 or lower than -1.0 tell us that these scores are significantly more extreme than those within 1.0 and -1.0. In the case of reasonably normal distributions such as with the data in this study, approximately 68% of the scores will fall within the 1.0 and -1.0 range of the first standard deviation, and 95% of the scores will fall within the limits of the second standard deviation. Scores in the second standard deviation are more extreme than those in the first standard deviation, and those in the third standard deviation may be thought of as being very extreme. Thus, the example of Youngstown given above as having a Presage z-score of -3.82 tells us that it is a case of children living in extremely disadvantaged environments relative to what is typical within the State of Ohio.
The following graph is a z-score version of the previous graph showing the relationship between percent passing and the presage score. Both percent passing and the presage scores have been transformed into z-scores. You will note that the graph is virtually identical to the previous one and has exactly the same correlation coefficient (r=0.80). However, because we now have z-scores to view, we can easily see the categories near, above, or below the mean for each district.
In addition, categories of advantagement-disadvantagement have been added to the graph using the z-score divisions of standard deviation. The center column "Middle Class" is divided down the middle by the mean (average) for the state. Using the z-score divisions for standard deviations above and below the mean, we can then classify levels of advantagement-disadvantagement based upon those mathematical divisions, thus making it more clear as to just how the different districts can be seen to compare with each other. Youngstown City and Indian hill districts that were used previously as examples of z-scores are both circled on the graph, showing the z-score significance visually.
Though categorical descriptors have not been added to the x-axis, we can still see how far above or below the state mean the various districts fall. If we were to create a grid by marking off the z-score standard deviations for the percent passing 1997, we would see that districts cluster in very similar ways where passing and presage scores have similarly high or low z-scores. This grouping is simply another way of seeing how districts with higher levels of advantagement cluster with higher levels of percent passing as low advantaged districts cluster with low percent passing. Once again, note how Youngstown City and Indian Hill are respectively low-low and high-high within the clusterings that are shaped by the data as arrayed by z-score graphing.
Data Supporting the Presage Factor Significance:
What is somewhat unusual is that the variables combined through the calculus of the presage formula yield a more powerful predictive correlation than do any one of the individual variables used in the formulation. However fortuitous, it is important and illuminating to understand the significant degree to which district test performance is predicted by the individual variables of Free/Reduced Lunch enrollment, ADC, Economic Disadvantagement, and Mean Family Income. The following four graphs visually represent these component variables used in the presage formula. I believe they help us understand the gravity of using tests such as OPT where the bias is so clearly shown.
The graph of percent enrolled in the free/subsidized lunch program shows an inverse correlation of r=0.73, which should be considered an extremely significant correlation. It is an inverse correlation simply because as the percent enrolled in the program increases, district performance drops. The primary evidence the finding provides is to validate the association of test performance with a specific measure of advantagement-disadvantagement.
The State of Ohio's own measure of economic disadvantagement also shows significant correlation with district OPT performance. As with the previous graph, the correlation is inverse, telling us that as the percent of economic disadvantagement goes up, district test performance goes down.
The graph of mean income provides us with both a significant correlation and a telling view of the mean income data itself. The correlation between the mean income of a district and OPT performance is r=0.58. Though lower than the correlations seen in the previous findings, r=0.58 is still a highly significant correlation coefficient. In terms of the coefficient of determination (r2), we find mean family income conservatively determining about 33% of district OPT performance.
However, the distribution is somewhat curvilinear. A curvilinear distribution is one in which the distribution points have a visible curvature of some sort. The curvilinearity is visible in the mean income graph as the array of points can be seen to bend to the right toward the quadrant formed by the above-average mean income and above-average district performance area of the graph.
Two findings can be drawn from the curvilinear spread. The first finding is the statistical reality that because the data array is clearly curvilinear, the correlation coefficient is underestimating the degree of association between the two variables. This means that the correlation coefficient of r=0.58 is most likely considerably lower than the actual degree of correlation. In other words, though r=0.58 is a relatively high correlation, it belies the reality of there being actually a higher correlation than seen due to the curvilinearity.
The second finding is serendipitous to the study but both relevant and interesting taken within the context of OPT and the effects of non-school variables on district performance. In examining the curved nature of the data, we can see implicit evidence of how mean income changes dramatically as we move from the upper middle class to the upper classes.
Because income distribution is the primary determiner of relative advantagement-disadvantagement disparity, the decision was made to examine how the continuum of advantagement-disadvantagement has been shaped by mean income changes over the past ten years and how it may have exacerbated the extremes of poverty and wealth affecting the lived experiences of Ohio's children.
In other words, because we can think of the presage score as representing a point on the continuum of advantagement-disadvantagement and because the range (length) of that continuum represents the scope of disparity in living conditions, we can examine how that scope may have changed over the past several years. The relevance of this side-bar analysis to this study is to provide a context for better understanding who is intrinsically advantaged and who is intrinsically disadvantaged by OPT and how those may have changed as a function of the distribution of wealth over the past few years. The following graph shows how district mean family income has changed over the years 1987 through 1998.
This graph shows how district mean family income changed from 1987 to 1998 in terms of the presage scores. The most striking finding is that income increased far greater for the wealthiest districts than for the less wealthy ones. Indeed, when the graph is examined closely, we see that increases in family income are relatively slight from the extremely disadvantaged upward through the middle class until we reach the upper end of the middle class and into the advantaged class, where it changed dramatically.
The most contrasting districts have been identified on the graph to better understand the extremes of the advantagement- disadvantagement continuum. As would be expected given the power of the Presage Factor, the mean percent passing for the 5 districts with the greatest increase in mean family income is 91.4%; the mean percent passing for the six districts with the least change in mean family income is 52.9%.
What the comparison in the above paragraph tells us is that OPT is very tightly tied to an explicit association with wealth. The degree to which the association with wealth is a function of living conditions and the lived experiences of the district's children is told in the elements that comprise the Presage Factor and their individual contributions shown in this section above. However, the question also arises as to the degree of local financial contribution to the local districts funds given the wealth available to commit funds. Findings regarding funding variables will be examined briefly in Section Five, after examination of district performance controlling for the effects of the Presage Factor in Section Four.
Section Six: Teacher Data
The following data analyze several variables directly related and indirectly related to the teachers in Ohio's districts relative to the variables of Presage Factor, percent passing, and actual performance. It should never go unnoticed that classroom teachers bear the brunt of the accountability effects of using OPT as an assessment mechanism for teacher effectiveness via the district ratings of OPT. Likewise, stakeholders, the media. and educational administrators who accept OPT and OSRC at face value make classroom teachers the target of their focus when angry or frustrated about low district scores.
In my many and frequent discussions with classroom teachers, including those from districts where passing levels are above average, speak volumes to the problems OPT and OSRC have created for classroom teachers. Almost without exception, they articulate how OPT-driven management has taken from them the last vestiges of reflective, professional decision making about what is best for the children in their classrooms.
This graph shows the correlation between teacher salaries and the presage scores for the districts. In terms of the correlation coefficient (r=0.35), there is a moderately high degree of association between advantagement-disadvantagement and teacher salaries. Simply put, we clearly see that teacher salaries increase as a function of the wealth of the districts. In terms of school variables, as opposed to non-school variables, this finding is significant in terms of understanding additional apparent inequities across Ohio's school districts. The finding here also underscores the problem of the spectrum of advantagement-disadvantagement when we consider the strong tendency for the more disadvantaged districts to have the most underpaid teaching staffs.
The analysis of percent passing and teacher salary yields a moderately high correlation. This finding supports the notion from the graph of presage scores and teacher salary that advantaged districts tend to pay their teachers more than less advantaged districts. Likewise, the finding here supports the notion of OPT advantagement-disadvantagement bias because of the association of higher salary with higher percent passing. However, because percent passing is a function of OPT bias as established in the primary findings of this study, the claim that performance is a function of salary may be misleading.
In examining actual performance we see a slight correlation between teacher salary and performance. This finding suggests that to some degree, teacher salary is a positive school variable. We must remember that actual performance is derived from controlling only for the effects of Presage Factor. The finding is not absolute because we cannot declare actual performance to be a robust measure of real academic performance beyond its presage control. However, relative to the other variables run against actual performance, teacher salary has the highest correlation with the exception of extra academic performance which will be presented later in this section.
The following sets of graphs examine district teacher degree status, the percent having no degree, bachelor's degree, and master's degree or higher.
The analysis of the association of presage scores with the percent of teachers without a degree shows us that the correlation is not significant. However, the association of non-degree teachers tends to increase with greater disadvantagement.
The graph of the analysis of percent passing with non-degree teachers yields what we would expect given the finding of the Presage Factor. Since percent passing is so closely defined by presage effects, this graph supports the findings in the first graph of this set.
Controlling for presage effects, this analysis of percent non-degree teachers across actual performance yields a non-significant correlation because r=0.02 is extremely low. What minuscule correlation there is relates positively to increased actual performance. However, no claim to any statistical significance can be made.
The following graphs and analyses are best understood when examined in conjunction with the master's degree graphs and findings because percent of teachers with master's degrees and percent of teachers with bachelor's degrees are fundamentally complimentary numbers, excluding the small percent of non-degree teachers. In other words, for any given district, the number of non-degree, bachelor's degrees, and master's degrees held by the teachers equals 100%.
The graph of presage score and teachers with bachelor degrees shows a slight inverse correlation. This finding tells us that the percent of bachelor degrees decreases somewhat as advantagement increases. This finding in and of itself may be seen as somewhat puzzling until we examine the finding regarding the percent of teachers with master's degrees or higher. (See the third set of graphs in this section.) Taken with the findings of the analysis of master's degrees and presage scores, the conclusion is that as advantagement increases so does the percent of teachers with master's degrees or higher.
The findings of the analysis of percent passing and teachers with bachelor's degrees clearly show the artifacts of OPT bias along the advantagement-disadvantagement continuum. The correlation (r-0.23) is moderately significant and does show the tendency of wealthier districts to have greater numbers of teachers with degrees beyond the bachelor's level when we consider this finding along with the finding regarding master's degrees as discussed above.
Again, taking the finding of actual performance and teachers with bachelor's degrees along with its complement of teachers with master's degrees or beyond seen below, we find that actual performance does correlate inversely, though only slightly. (Refer to the discussion following the presentation of the graph showing actual performance and teachers with master's degrees or higher for more interpretation of this analysis.)
The following graphs and analyses are best understood when examined in conjunction with the bachelor's degree graphs and findings because percent of teachers with master's degrees and percent of teachers with bachelor's degrees are fundamentally complimentary numbers excluding the small percent of non-degree teachers. In other words, for any given district, the number of non-degree, bachelor's degrees, and master's degrees held by the teachers equals 100%.
As we would expect, the percent of teachers with master's degrees and beyond increases as a function of increase in advantagement. This finding is understandable in light of the extra expenditure required for paying salaries of teacher's with graduate degrees.
The finding of a moderate correlation between percent passing and percent of teachers with master's degrees is not unexpected given the very high correlation (r=0.80) between percent passing and presage scores. In other words, districts with greater advantagement are more likely to have more teachers with advanced college degrees than those with less advantagement.
The comparison of actual performance to the percent of teachers with advanced degrees shows a slight positive correlation. The analysis here tells us that within the previously discussed limits of actual performance in controlling for the presage effects, teacher's having advanced degrees does contribute somewhat to actual performance.
The next section deals with the analysis of teacher experience across the variables of presage score, percent passing, and actual performance.
The correlation between presage score and years of teacher experience is non-significant, though it shows a very slight inverse correlation that says there is a very slight tendency for more advantaged districts to have teachers with slightly less experience than less advantaged. However, the association is too low to support any claim other than there is no significant difference in the average years experience across Ohio's school districts in terms of advantagement-disadvantagement.
The finding from the analysis of percent passing and teacher experience shows an almost perfectly random correlation. In other words, there is no difference whatsoever in terms of teaching experience and percent passing OPT.
The analysis of actual performance and teaching experience shows a slight positive correlation. Though the correlation is slight, it nonetheless appears to be a possible contributor to academic performance when we control for the effects of advantagement-disadvantagement.
Two variables related to teachers and teaching have been included in this section for possible illumination of the study's findings. They are class size and extra-academic opportunities.
The EMIS provides two similar sources of information regarding teacher/student ratios, class size and teachers per 1000 students. Both variables yield almost exactly the same findings. Since class size is a more familiar concept than teachers per 1000 students, I chose to use it.
Class size is found to be inversely correlated to presage scores and is only slightly significant. This finding simply tells us that class size tends to be slightly lower the more advantaged the district is in terms of presage scores.
Though slightly lower in terms of its statistical correlation, percent passing and class size reiterate the relations between presage effects and percent passing.
Analyzing class size and actual performance yields a non-significant correlation that approaches randomness in the relations between the two variables.
Extra Academic Opportunities
The state defines extra academic opportunities as extracurricular activities that are academic in nature, such as debate team, French club, math club and similar activities open to student involvement outside the regular academic classes. Recreational and sports activities are not considered as extra academic opportunities.
Analysis of the variables of extra academic opportunity and presage scores yields a moderate correlation. This means that extra academic opportunities increase as the advantagement as measured by the presage score increases.
The correlation found with extra academic opportunities and percent passing is moderately high and tells us that the districts with greater numbers of such opportunities tend to perform better on OPT. However, it is important to remember the bias of OPT toward more advantaged districts. Because of this, conclusions regarding the actual effects are somewhat unclear, but the suggestion that extra academic opportunities contribute to improving district test performance is evident.
The examination of extra academic opportunity and actual district performance shows a moderate correlation. This finding lends strength to the power of such opportunities in affecting actual test performance within the parameters of the Presage Factor. As well, this finding suggests that the idea discussed in the previous graph that extra academic performance positively affects percent passing may have greater credibility.
Teacher Data Comments:
Examination of the analyses and findings regarding the variables within this section on teachers as they may interrelate, indicates that most of the results are verifications of what we might expect given the power of the non-school forces and factors associated with district levels of advantagement-disadvantagement as described by the Presage Factor. However, if we can accept that actual performance is indeed a usable measure of what may be happening academically in Ohio's schools, several findings in this section suggest themselves as variables contributing to that performance.
Teacher salary, having a master's degree or higher, years of teaching experience, and extra academic opportunities stand out as variables contributing to some degree to actual district performance. To examine further the efficacy of these variables, the four were converted to z-scores, added, and averaged to create a single measure. For lack of a better term, the combination into a single variable is called the "Teacher-Curriculum" variable (TC) to represent the domains of schooling from which the variables arise. The following three graphs examine the TC variable to better understand the potential of the four elements operating together.
The correlation with presage scores is moderate and suggests that the variable is associated with the advantagement defined by the presage scores.
The teaching-curriculum variable attains an moderately high correlation when associated with percent passing. The degree of association is higher than with the presage scores as seen in the preceding graph, thus suggesting it is more closely associated with percent passing than with presage scores themselves.
Whereas three of the four variables that comprise the TC variable have less than r=0.15 correlation and the fourth variable of extra academic opportunity has a correlation coefficient of r=0.24, the combination of all four exceeds the average of the four coefficients. Again, it is important to remember that actual performance is a measure of OPT performance controlling for presage effects that represent the overwhelming bias of OPT. Another way to view this is to think of actual performance scores as possibly valid representations of academic performance. If this assumption is true, then the formulation of the TC variable begins to reach into the myriad of complex possibilities that shape authentic academic performance.
The point here is not to posit a new way to assess district performance, but to demonstrate how complex the processes of teaching, schooling, and learning are in the real world of education. More specifically, the findings of this study to this point do not only tell us that OPT is a highly invalid assessment mechanism, but the findings also expose the tremendous complexity and difficulty of authentically and validly assessing academic performance on any level.
Even if the calculus used to formulate actual performance results in a valid assessment of district performance to a greater degree than does OPT, the problem still exists that it is based upon a high stakes test. The pressure to pass the test, the time spent practicing to take the test, and the denial of the credential of a high school diploma for those innocent children who often narrowly fail to make the cutoff score is all born by the children and parents of Ohio's public school population. Thus, there is no suggestion whatsoever that the derived actual scores should in any way be used to hold anyone accountable because of the damage such testing does to the children and to the curriculum they should have the opportunity to experience.
Section Seven: Percent African-American and Percent White as Variables Across Presage Score, Percent Passing, and Actual Performance
The following are three sets of graphed data addressing how OPT may be seen to play out across African-American and White school district populations. The first set of graphs below compares the association between the Presage variable and the percent African-American and the percent White of the district student population. The second set of graphs represents the percent passing per district as a function of percent African-American and percent White student populations. The third set gives a comparison of performance controlling for the Presage variable. It is vital that these three sets of findings be viewed together for optimal understanding of the general comparative effects of OPT on these two groups.
The first graph in the set tells us that there is a moderate negative correlation between advantagement and the percent African-American student population. The second one shows that there is a moderate positive correlation between advantagement and the percent White student population of the district. The graphs are essentially inverses of each other as would be expected because as percent White goes up, the percent of African-American must go down and vice versa.
Most simply stated, these graphs tell us that the greater the White population of the school district, the greater the level of advantagement; the greater the African-American district population, the greater the level of disadvantagement. Taken together, the findings support the argument that the effects of social-economic advantagement-disadvantagement are seen to a moderate degree in the racial composition of Ohio's school districts. The next set of graphs represents the findings of how the two populations are associated with OPT performance.
This set of graphs shows us that there is a moderately significant differential between African-American and White performance on the OPT. The first graph in the set tells us that the greater the percent African-Americans in the district, the more likely fewer students will be achieving passing OPT scores. The second one shows the opposite for White students. Again, the graphs are basically mirror images of each other, as noted previously.
Two points for objective interpretation are very important here. The first point is that the findings definitely reveal OPT bias against students in predominantly African-American school districts. It is also logically true that the findings may be interpreted as definitely revealing OPT bias in favor of predominantly White school districts. However, the second important point is that while the effects are real in terms of bias, it cannot and must not be concluded from this data array that the OPT bias is caused directly by racial differences between the two groups.
While the findings do show the racial bias to be real, attributing the bias to a specific source requires more critical examination of the data. This is so because the study's primary and most powerful finding is that social-economic advantagement-disadvantagement is the most significant predictor of performance. In other words, the research question arises of whether the demonstrated OPT bias shown here is a function of social-economics or race, or both.
Indeed, it is somewhat interesting that the correlation coefficients for the first two sets of graphs are quite similar (r=-0.34, r=0.030 and r=-0.35, r=0.31). Considering the fact that we know from the primary findings of this study that the Presage factor is unusually powerful as a variable (r=0.80) of advantagement-disadvantagement for predicting OPT performance, to determine first-order racial/cultural OPT effects, we need to examine actual OPT performance controlling for the Presage factor. In other words, the racial differential shown in this second set of graphs must be examined further before suggesting that it is caused by racial/cultural differences and not by the social-economic factors of the Presage variable.
The graphs below show actual performance (performance controlling for the Presage variable) for White and for African-American populations and yield statistically non-significant effects in and of themselves. These non-significant effects are, however, very significant in understanding and knowing that when the factors of advantagement-disadvantagement as defined by the Presage Factor are removed, we find that race is not the primary factor affecting academic achievement in terms of district level OPT performance. However, given the correlations (r=-0.15 for African-Americans and r=0.11 for Whites), we do see effects that could very well represent racial/cultural bias inherent in the tests.
Though the correlations are low, the question does arise as to the source of why there would be any difference between African-Americans and Whites across OPT performance when controlling for the Presage Factor. Though nothing definitive about the primary source of the differential is immediately apparent, I would suggest two possibilities be given consideration.
The first possibility is that there are other social-economic effects showing up that are not within the scope of the Presage Factor that are experienced differentially by the two groups such as is seen in the group correlations in the first set of graphs. The second possibility and certainly the more serious of the two is that OPT contains significant racial/cultural bias.
My best professional judgment tells me that it is quite likely that the findings shown in the last set of graphs are related to racial/cultural OPT bias. I base this speculation on my experience and my intuition combined with the language/reading dependency of the test. Minimally, these data call for a complete and thorough examination of OPT for racial/cultural bias if the State of Ohio wishes to make any claims of test validity.
Together, all three sets of comparisons re-confirm that the social-economic-environmental factors that shape the conditions of advantagement-disadvantagement are the clear bias of OPT regardless of race. In other words, disadvantaged children are likely to perform more poorly on the test than advantaged children regardless of whether they are African-American or White.
The first two sets of graphs do tell us that African-American children are more likely to suffer from the conditions of disadvantagement than are White children and that because of this powerful effect, far more African-American children are victims of OPT bias than are White children.
The last set of graphs suggests that there is most likely some racial/cultural bias in the test. Though the effect of this bias on district level performance is significantly less than the effect of the Presage factor, it does make a difference especially in districts with high African-American student populations because passing rates load so powerfully on the Ohio School Report Card ratings.
If indeed, the third set of graphed data is showing the artifacts of inherent racial/cultural bias, then the effects are particularly significant for individual African-American students in Ohio's schools. My own professional judgment tells me that this scenario is probable given the legacy of racial/cultural bias in standardized testing. At the very least, these data indicate a moral responsibility and a legal obligation on the part of the State of Ohio to suspend testing until further study of the racial/cultural bias is openly and honestly conducted.
It is vital to recognize that the data represented in all three sets of graphs removes from discourse and discussion the question of African-American students ability to learn as well as other racial groups. There is simply no evidence whatsoever to support any arguments regarding inferior academic performance. Therefore, for anyone to make the claim that African-American children do not have the same native ability as Whites in terms of academic achievement is as absurd as it is ignorant and racist.
Given the previous point, it would be remiss to fail to point out that OPT district performance as reported by the state (See, "Percent Passing and Percent African-American Students" in the second set of graphs in this section) makes it appear that African-American students are academically inferior to White students. The findings of this study do not support this implicit claim made by the State of Ohio through the OPT and OSRC. Indeed, the data indicate the claim is totally false and dangerously misleading in its racial significance.
Again, the findings reported in this section of the study indicate a moral responsibility and a legal obligation on the part of the State of Ohio to suspend proficiency testing until the possible racial/cultural bias is thoroughly examined and the misleading racial overtones found in the results are corrected.
Section Eight: Advantagement-Disadvantagement as a Predictor of Ohio School Report Card Ratings
The Ohio School Report Card, 2000, correlates with the Presage Factor (r=0.78) almost as significantly as the 1997 OPT district performance does (r=0.80). Practically speaking, they are virtually the same. What this means is that OSRC carries with it the same advantagement-disadvantagement bias.
What these findings tell us is that to a very significant degree (conservatively, 60.1% based upon r=0.78), the OSRC reports social-economic living conditions of the district and not the academic growth of the pupils nor the effectiveness of educators in the district. OSRC is open to the old computer adage of "garbage in, garbage out." The fundamental unit of assessment that drives the OSRC ratings is the percent of the district's pupils passing the OPT; if the unit of assessment is flawed, so are the cumulative results reported in the OSRC. Again, as with OPT itself, validity in the statistical/mathematical sense of tests and measurements is the flaw of the OSRC.
By the time OSRC ratings reach the public, they are impersonal representations of Ohio's school children framed invisibly by their very real lives on the spectrum of advantagement-disadvantagement. Yet, OSRC is used to reward or punish the very people who have to deal with the day-to-day reality of those children's lives, educators being held accountable for that over which they have virtually no control or decision latitude whatsoever.
Given that the OSRC is aimed directly at assessing the district's educators in general and its teachers in particular, the findings of this study point to the following advisories:
Teachers and educators in districts rated low on the OSRC may, indeed, be performing extremely well such as is known from this study to be the case with Youngstown City Schools as noted in the previous section on actual district performance. In other words, beware that there may be no validity whatsoever to the rating given by the OSRC that places our most disadvantaged districts in the "academic emergency" category.
Teachers and educators in districts rated high on OSRC may, indeed, be performing nowhere near their potential. From the results of this study, this is shown to be the case with many OSRC top-ranked districts. In other words, just as noted above regarding underestimating low-ranking districts, the same caveat applies to the highest ranked districts.
Teachers and educators in districts given OSRC ratings anywhere in-between the extremes of the "academic watch" and "effective" categories likewise cannot be said to be valid with any certainty at all. These OSRC mid-range school districts represent the vast majority of Ohio's schools. Some of these districts are performing as claimed by OSRC ratings. However, an equal number are either performing far below what might be expected and others far above
There is an imminent reality implicit in this study that cuts directly to the assumptions and interpretation of OSRC as a measure of district educator effectiveness. That reality is a judgment that I make with reflective confidence based upon these findings and upon my professional experience as a classroom teacher and a university teacher educator. My judgment is that if we were to ever switch the staff of a district rated high by the OSRC with that of a district rated low, in five-year's time there would likely be no change whatsoever in the OPT ratings of either school district. Indeed, if any change were to be observed, it would most likely be that the scores of the low OSRC-rated district would drop slightly due to the highly rated educators having to deal with problems and issues in the lives of the children of the district that they have never experienced anywhere before in their professional experience.
Section 9: A Brief Closing Statement
The primary purpose of this study was to examine forces and factors that affect Ohio Proficiency Test performance. The primary finding is that OPT is extremely biased across the elements defined within the parameters of the Presage Factor. The significance of the primary finding is that OPT is not a valid measure of either academic performance or school accountability at any level including the Ohio School Report Card ratings.
However, nothing within the study's findings or inferences should be viewed as blaming or making excuses for students not learning, educators not educating, or districts not performing. On the contrary, the findings and inferences lead us away from excuse making into the realm of validly assessing accountability of actual academic and school performance. There is vast difference between an excuse and an explanation of OPT performance.
This study of OPT performance explains why scores are invalid regardless of social economic level. It is no more an excuse for poor performance than it is for high performance. Rather the findings show that regardless of social economic status, the results are not valid; OPT performance of advantaged districts is just as invalid as the performance of less advantaged districts. Indeed, when we control for social economic factors, the findings show that actual academic performance is evenly distributed across all levels of advantagement-disadvantagement. Children from disadvantaged environment are shown to be equally successful as those from advantaged environments.
Also, it was not the intent of the study to beg the question of educational accountability or academic standards. On the contrary, accountability and standards are both requisite to establishing a quality system of public schooling. However, it is incumbent upon stakeholders in general and state education policy makers in particular to establish assessments and standards that meet the well established standards for test validity and appropriateness. The simple irony implicit in the findings and inferences of the study is that the citizens of Ohio have a right to hold public schools accountable just as they have the right to hold accountable those who shape public school policy.
"The problem with truth is its verification,
the problem with fiction is its veracity."
|Miller New Clev.||Putnam||11||1.85||0.73||14.67|
|Benton Carroll Salem||Ottawa||17||1.78||0.16||-8.7|
|Cleveland Hts-Univ Hts||Cuyahoga||34||1.54||-1.78||-88.96|
|James A Garfield||Portage||150||0.67||-0.05||-17.58|
|St Henry Consolidated||Mercer||151||0.67||0.96||24.37|
|Liberty Un Thurston||Fairfield||172||0.55||0.04||-13.77|