We continue with part three (part I & Part II) of our series with a few snippets from the NY Times article, “Big Study Links Good Teachers to Lasting Gain” by Annie Lowery, January 6, 2012 (warning today’s information is a bit technical):

The paper [The Long-term Impacts of Teachers: Teacher Value-added and Student Outcomes in Adulthood], by Raj Chetty and John N. Friedman of Harvard and Jonah E. Rockoff of Columbi… is the largest look yet at the controversial “value-added ratings,” which measure the impact individual teachers have on student test scores. 

But looking at an individual’s value-added score for three or four classes, the researchers found that some [teachers] consistently outperformed their peers.

That the variations or differences between really good and really bad teachers have lifelong impacts on children.”

Replacing a poor teacher with an average one would raise a single classroom’s lifetime earnings by about $266,000, the economists estimate. Multiply that by a career’s worth of classrooms.

“If you leave a low value-added teacher in your school for 10 years, rather than replacing him with an average teacher, you are hypothetically talking about $2.5 million in lost income,” said Professor Friedman, one of the coauthors.

To do the study, the researchers first tackled the question that has swirled controversy in so many school districts, including New York City’s: whether value-added scores are in fact a good measure of teacher quality. Mr. Jones might regularly help raise test scores more than Ms. Smith, but maybe that is because his students are from wealthier families, or because he has a harder-working class — factors that can be difficult for researchers to discern.

But controlling for numerous factors, including students’ backgrounds, the researchers found that the value-added scores consistently identified some teachers as better than others, even if individual teachers’ value-added scores varied from year to year.

After identifying excellent, average and poor teachers, the economists then set out to look at their students over the long term, analyzing information on earnings, college matriculation rates, the age they had children, and where they ended up living.  The results were striking. Looking only at test scores, previous studies had shown, the effect of a good teacher mostly fades after three or four years. But the broader view showed that the students still benefit for years to come.  Students with top teachers are less likely to become pregnant as teenagers, more likely to enroll in college, and more likely to earn more money as adults, the study found.

The authors argue that school districts should use value-added measures in evaluations, and to remove the lowest performers, despite the disruption and uncertainty involved.

Below are a few snippets directly from the study, The Long-term Impacts of Teachers:  Teacher Value-added and Student Outcomes in Adulthood:

Are teachers’ impacts on students’ test scores (“value-added”) a good measure of their quality? This question has sparked debate largely because of disagreement about (1) whether value-added (VA) provides unbiased estimates of teachers’ impacts on student achievement and (2) whether high-VA teachers improve students’ long-term outcomes. We address these two issues by analyzing school district data from grades 3-8 for 2.5 million children linked to tax records on parent characteristics and adult outcomes. We find no evidence of bias in VA estimates using previously unobserved parent characteristics and a quasi-experimental research design based on changes in teaching staff. Students assigned to high-VA teachers are more likely to attend college, attend higher- ranked colleges, earn higher salaries, live in higher SES neighborhoods, and save more for retirement. They are also less likely to have children as teenagers. Teachers have large impacts in all grades from 4 to 8. On average, a one standard deviation improvement in teacher VA in a single grade raises earnings by about 1% at age 28. Replacing a teacher whose VA is in the bottom 5% with an average teacher would increase the present value of students’ lifetime income by more than $250,000 for the average class- room in our sample. We conclude that good teachers create substantial economic value and that test score impacts are helpful in identifying such teachers.

First, using VA [value added] measures in high-stakes evaluations could induce responses such as teaching to the test or cheating, eroding the signal in VA measures.  This question can be addressed by testing whether VA measures from a high stakes testing environment provide as good of a proxy for long-term impacts as they do in our data.  If not, one may need to develop metrics that are more robust to such responses, as in Barlevy and Neal (2012).  Districts may also be able to use data on the persistence of test score gains to identify test manipulation, as in Jacob and Levitt (2003), and thereby develop a more robust estimate of VA.   Second, one must weigh the cost of errors in personnel decisions against the mean benefits from improving teacher value-added.   We quantified mean earnings gains from selecting teachers on VA but did not quantify the costs imposed on teachers or schools from the turnover generated by such policies.

Whether or not VA should be used as a policy tool, our results suggest that parents would place great value on having their child in the classroom of a high value-added teacher.   Consider a teacher whose true VA is 1 SD above the median who is contemplating leaving a school.   Each child would gain approximately $25,000 in total (undiscounted) lifetime earnings from having this teacher instead of the median teacher.  With an annual discount rate of 5%, the parents of a classroom of average size should be willing to pool resources and pay this teacher approximately$130,000 ($4,600 per parent) to stay and teach their children during the next school year.   Our analysis is of teacher entry and exit directly confirms that retaining such a high-VA teacher would improve students’ outcomes.

While these calculations show that good teachers have great value, they do not by themselves have implications for optimal teacher salaries or merit pay policies.  The most important lesson of this study is that finding policies to raise the quality of teaching whether via the use of value-added measures, changes in salary structure, or teacher training is likely to have substantial economic and social benefits in the long run.

We continue our discussion with a few paragraphs from “The Market for Teacher Quality” by Eric A. Hanushek, John F. Kain, Daniel M. O’Brien, and Steven G. Rivkin, NBER Working Paper No. 11154, February 2005

In this paper we use matched panel data on teachers and students for a large district in Texas to estimate variations in teacher quality through a semi-parametric approach based on value added to student achievement.  These estimates confirm the existence of substantial variation in teacher effectiveness, much of it within as opposed to between schools.  This within school heterogeneity has direct implications for the design of accountability and teacher incentive programs.  Our findings on teacher effectiveness are also consistent with prior evidence showing that certification and experience explain little of the quality variation, with the exception of sizeable improvement following the initial year of teaching.  The pattern of results supports the view that good teachers tend to be superior for students across the achievement distribution, but there is strong evidence that students benefit from being matched with teachers of the same race.

The analysis of the determinants of quality reveals some very important insights.  First, there is significant learning about the craft of teaching in the first few years of teaching.  The largest impact is the first year of experience, and experience effects disappear quickly after the first year.  Second, teachers who do well with students in one part of the achievement distribution also do well with students in other parts of the distribution, suggesting that teacher-student matching by initial preparation is not a particularly important policy issue.  Third, students benefit from having a same race teacher, quality held constant, and that benefit is sizeable.

A commonly voiced policy concern is that large urban districts lose their better teachers to other occupations or to suburban schools.  We find little if any support for the notion that the better teachers are the most likely to exit the public schools entirely.  To the contrary, teachers exiting Texas public schools are significantly less effective on average in the year prior to leaving than those who remain, and those changing districts are quite similar in terms of effectiveness.

The identification of large variation in the quality of instruction within schools has at least two additional implications related to school accountability.  Current state accountability systems typically aggregate the level of student scores for an entire school, in part on arguments about averaging out any measurement error.  First, even if the average level of test scores has a larger signal to noise ratio, the appropriately constructed achievement gain is a much more appropriate measure of value added and school effectiveness.  It is more closely related to current teacher performance and controls for important family and community differences that tend to confound estimates of teacher value added based on achievement levels.  And second, any formal or informal school evaluation program that aggregates performance to the school level or across years misses the majority of the variation in the quality of instruction.  This weakens the incentives for good teachers to enter and remain in teaching, for ineffective teachers to leave, and for all teachers to put forth greater effort.  These costs should be compared with the superiority of measuring performance at the school level in terms of fostering greater teacher cooperation within the school, a concept frequently advanced in policy discussions.

We continue tomorrow with Giving Kids the Chaff…

Pin It on Pinterest