The graph at the bottom of this post, compiled from the two most recent years of New England Common Assessment Program (NECAP) proficiency scores, contains the information you are probably used to seeing for describing the state of education in the cities and towns of Rhode Island. And if you are familiar with the data displayed there, then you are probably also familiar with the oft-expressed objection to its use for school-system accountability purposes, that the graph is mostly an illustration of the social-economic status of the various communities represented, more than of how well a school-system is doing or not doing with the students it has.

However, if instead of confining the analysis of test results to scores taken from one point in time, consideration is given to changes in scores in a group of students across time, then it becomes possible to begin to isolate the factors that impact test scores which are directly related to the school-system, for example, through an examination of the evolution of performance of similar student cohorts who start from comparable achievement levels. This idea is the basis of what is called the “value-added” method of analysis of education. Since much of the focus has been on high schools lately, let’s see what can be done with Rhode Island’s basic 11th-grade NECAP results, to take a step in this direction.

Rhode Island 11th-graders have been tested in the subjects of reading and math with the NECAP for the past two years. 3 years prior to those tests, many of those same 11th-graders took the NECAP as 8th-graders, and changes in performance at the school and district level, between grades 8 and 11, involving many of the same students, can be evaluated.

But exactly what changes should be measured? NECAP results characterize student achievement in terms of four categories, the numbers of students “proficient with distinction”, “proficient”, “partially proficient”, and “not proficient”. The summary that is most frequently reported to the public is the percentage of students in a district (or school) who are proficient or better. However, a comparison of proficiency percentages from year to year may not fully capture what is going on inside of a school. Between 8th and 11th grade, almost every district loses students. In some districts, according to the NECAP totals, that figure can be as high as 30% and the reasons for enrollment drops (e.g. dropouts, students leaving for private schools or charters, etc.) will skew percentage measures in ways that don’t necessarily reflect the quality of schooling.

So instead of going directly to results expressed as percentages, we will begin by taking a step back and looking at how the number of students proficient or better in 11th grade in each district changed from the number of students who were proficient in 8th grade in that same district. We can combine the two approximate cohorts of students for which 11th grade test data is available, and calculate the how the number of 11th grade students who scored proficient or better on the 2008 and 2009 NECAPs in each district changed from the number of 8th grade students who scored proficient or better on the 2005 and 2006 NECAPs

Several results become immediately obvious:

1. Even in many of the districts thought of as poorly performing, the number of students proficient or better in reading improved between the 8th and the 11th grade testing (even when total enrollments dropped).
2. In every school district in Rhode Island — including the ones thought of as well-performing — the number of students who are proficient or better at mathematics in the 11th grade is less than the number of students who were proficient or better in the 8th grade.

(N.B.: Results for North Kingstown and Portsmouth have been corrected from the original version of this post, to properly account for the fact that high-schools in these districts serve students from Jamestown and Little Compton, respectively.)

Next, we need to find an appropriate context with which to evaluate the absolute count data. That will be the subject of tomorrow’s post on the subject…