From Winston Lin:

Comment: On p. 307, you write that robust standard errors “can be smaller than conventional standard errors for two reasons: the small sample bias we have discussed and their higher sampling variance.” A third reason is that heteroskedasticity can make the conventional s.e. upward-biased. In your Monte Carlo study, heteroskedasticity makes the conventional s.e. downward-biased, because the smaller group (the treatment group) has the larger variance. If you instead choose sigma > 1 (so the control group has the larger variance), the conventional s.e. will be upward-biased (because when we pool the residuals, we overestimate the variance of the treatment group mean more than we underestimate the variance of the control group mean).   I’m sure you’re aware of this, but it might be worth noting explicitly, to avoid giving the impression that robust s.e.’s “should” be larger than old-fashioned s.e.’s.

Thanks for writing such a helpful, insightful, and fun book!

Thanks for this insight, Winston.

Indeed, in writing section 8.1 on robust standard errors we have not really appreciated the fact that conventional standard errors may be either too small or too big when  there is heteroskedasticity.  Winston is right that it can go both ways.  The attached note describes the mechanics, and gives conditions  for the direction of the bias.  Basically, conventional standard errors are too big whenever covariate values far from the mean of the covariate distribution are associated with lower variance residuals (so small residuals for small and big values of x, and large residuals in the middle of the x range).  We think this is empirically not the common case but it might happen.  The leading case is probably that residual variance goes up with the value of x (true for example in the returns to schooling example: earnings are more variable for those with more schooling).  In this case, conventional standard errors will tend to be “about right” or too small as the discussion in 8.1 suggests.

JSP