Megan McArdle On What Is Wrong With Pilot Programs

See Beware the Stunning Pilot Program.

"If we could replicate the results of Chicago's Child-Parent Center (or Perry Pre-School, or Abecedarian), then yes, it would be a no-brainer. But that's a pretty big hurdle. As I've written before, it's a huge mistake to assume that a pilot program can be rolled out on a large scale. Pilot programs are run by top-notch experts who are committed to, and familiar with, the organization's goalst. Attentive to producing good data, they follow procedures much more closely than ordinary employees. They have all the enthusiasm of someone embarked on a noble project--one of temporary duration. They have none of the frustration and demotivation of people stuck in challenging but often tedious jobs.

With pilot programs, you always have to be on the lookout for the Hawthorne effect: people being studied often change their behavior in response to the fact of being studied, not to any particular intervention. The effect gets its name from a factory where researchers were studying the effect of lighting on worker productivity. What they found was that both raising and lowering the light level caused productivity to increase--the workers were responding to the researchers, not the lights. It's not hard to imagine that a parent who is informed that their child is part of a Very Important Childcare Study might change their parenting in response.

When programs are rolled out on large scale, they suffer from a number of problems. They cannot all be staffed with top-notch, motivated staff--Perry Pre-School, for example, required certified public school teachers with at least a bachelor's, while Head Start requires an associates degree. The permanent staff will not hew as rigorously to procedures and methods as people doing a temporary experiment. And of course, there is no Hawthorne effect.

What you want to look at is not pilot programs, but the programs with large-scale rollout. And so far, research seems to show that the modest-but-still-exciting results we see from early childhood pilot programs are not replicated at the state or national level."