re: The State of Education - Aye, the Co-hort 'tis the thing

Andrew has inspired me to hop on his coattails concerning the way we look at NECAPs (so read his post first). Basically, I've been putting off posting how we can look at the same NECAP data in two ways. As Andrew explains, the "value-added" method would be to follow the cohorts (ie; the same group of kids from year to year). Andrews task of digging deeper into how we can tease out data comparing 8th grade and 11th grade scores for essentially the same group of kids is more difficult than what I am going to look at: comparing cohorts from year-to-year in elementary schools.

Here is a made-up example of how we're usually asked to "read" the NECAP scores. (Let's assume these are writing scores for "Quahog Elementary School"). Generally, we are given data that is "shaped" so that we look at the change from year to year for each grade like this:

GRADE	2005	2006	GROWTH	2007	GROWTH	2008	GROWTH	2009	GROWTH
3	63%	76%	13%	76%	0%	76%	0%	78%	2%
4	73%	65%	8%	77%	12%	75%	2%	73%	2%
5	63%	89%	22%	75%	14%	78%	3%	85%	10%
6	72%	68%	4%	78%	10%	75%	3%	85%	10%
AVERAGE	69%	75%	6%	77%	2%	76%	1%	80%	4%

This is the snapshot approach and it's flaw is that it compares one cohort of kids to its predecessor. Guess what? Cohorts are comprised of different mixes of kids--economic, family structure, academic expectations, etc. There is such a thing as a wicked smart class! So, instead, why not also track the grade-to-grade progress being made by the same set of kids (the same cohort)? To that end--to try to actually get some real value--we have to look diagonally at the above table. By doing so, we see something like this:

Year in Grade 3	2005	2006	GROWTH	2007	GROWTH	2008	GROWTH	2009	GROWTH
2005	63%	65%	2%	75%	10%	75%	0%	-	-
2006	-	76%	-	77%	1%	78%	1%	85%	6%
2007	-	-	-	76%	-	76%	0%	85%	10%

From "shaping the data" this way, we see that the 2005 and 2007 third graders both saw a 10% increase in proficiency between their 4th and 5th grade years. The 2006 third graders saw only a 1% increase between 4th and 5th grade, but a 6% increase between their 5th and 6th grade years. The NECAP is given in October, so the tests given in 5th grade are supposed to be on what they learned in 4th grade. As such, this data seems to indicate that the 4th grade teachers are pretty good at improving the proficiency rates of the students they've been given. Meanwhile, the 5th grade teachers are batting around .500 (from the data I've re-shaped) in improving vs. maintaining the status quo.

I believe that looking at the data by comparing different cohorts at the same grade level provides some value in assessing progress, however, I believe that a better method--one that is probably more fair to teachers and the students within a given cohort-- is the alternative method outlined above.