Interesting thoughts on why some experiments don't transition well to larger scales

In The Value of Health Care Experiments Megan McArdle explains why some programs may work well in a small controled situation, but not in a larger environment, even when the processes are the same.  Here is a key point:

----------

Why don't we have more revolutions in human affairs?  For starters, because these revolutionary studies are usually working with a pretty small number of patients.  This means that there's going to be a lot of variance--some will, by chance, show good results; some will, by chance, seem like disasters.  The programs with "good results" will survive and get written up by social science journals and people like Atul Gawande; the programs that end up costing money will collapse and disappear into a welter of administrative embarassment.  Note that I don't say that this is what has happened in the case of these particular programs.  The problem is, with small programs like this, it always has to be at the back of your mind.  That's one of the major reasons why promising pilot programs are so rarely replicated successfully. 

But not the only reason. Even the programs that genuinely work have a lot of things going for them that a broader program won't.  They have a crack team of highly educated experts who are extremely excited about the program, and understand the ideas behind it backwards and forwards.  They work in a controlled environment, and usually have a decent amount of administrative support for their efforts.  They are time limited, which matters--people are willing to endure lots of things for a limited, known duration that they wouldn't do permanently. They are often offering bonuses for participation.

Then they get implemented in the real world, with ordinary people who don't particularly want to change the way they've always done things, don't really care about the noble ideas behind your program, and don't see any end to it.  And the effects disappear.  

----------

 

Public schools are a great example of this.  There are many, many school programs which have been implemented on a large scale because they once worked on a small scale.  If you want to be really depressed about government schools, read Left Back by Diane Ravitch.  It is a history of education reform in the United States over the last century.  The "Experts" would keep trying the same programs again and again, hoping that this time it would work, because they got it to work on a small scale.

 

(Hat tip: Common Sense)