This poignant post from Michael Goldstein ends with a few policy thoughts that largely support my previous post.

Goldstein’s second point is worth highlighting:

Anyway, in a small school, large-scale research isn’t the key determinant anyway. The team’s implementation is.

On the same day that Shanker Blog is assuring us that rigorous social science is worth it, Goldstein delivers researchers a healthy dose of humility. Rigorous research is all about doing the best we can to remove all the confounding explanatory factors that have an impact on our observed outcomes to isolate an intervention. But even in the most rigorous studies social scientists are often measuring the Average Treatment Effect.

How rarely do we truly encounter a completely average situation? The real impact in any particular school or organization can be dramatically different in magnitude and even direction because of all the pesky observed and unobserved confounding factors that researchers work so hard to be able to ignore.

So my advice? If you are down on the ground keep close to the research but keep closer to your intuition, provided you are ready, willing, and able to monitor, evaluate, and adjust.