I agree with this very sobering analysis. But most often adequate power is not possible. Proper power analysis for sample size for a “big” effect (r = .5; Cohen & Cohen, 1988) with power set to .8 = 29. Yes, that’s 29 for a big effect size. Not many of us (neuroimagers) can afford that. Perhaps a better approach is to remind the reader of the confidence intervals for our data.

I hadn’t seen your blog before tonight, but I’ve really enjoyed reading over the posts. Very thoughtful and insightful comments. With respect to sample size and power in fMRI (and other) studies, I’m surprised that the role of small effects in paradigms shifts in science hasn’t been mentioned more prominently. Most theories usually get the large effects right. It’s the small effects that are harder to measure and detect that often mitigate the direction that science takes (in physics, there are numerous examples such as quantum vs Newtonian mechanics). Surely, these should not be dismissed as trivial. Anyway … my 2 cents.

Yossi, yes, there are some other downsides to small samples, especially in fMRI studies. Aside from the data snooping issue you raise, there are at least the two concerns I raised above: that potentially very small effects can be overstated, and that researchers are very fond of reporting ‘selective’ activations when their data don’t actually imply selectivity. The latter, in particular, is a huge issue. But I do agree that studies shouldn’t be criticized just because they’re small, so long as the authors are careful to avoid some of the surrounding pitfalls.

Gael, that’s a fair point, though personally I’m more concerned with the substantive recommendations surrounding sample size than the tone. I’ve written a fair number of posts here that are similarly snarky, so I can sympathize with Friston’s approach; I just think in this case some of the specific issues were misconstrued.

Rich, on (1), yes, that’s a good point. I do think on balance it’s still fair to say that an effect is more likely to be big if it came from a small sample, it’s just that the level of certainty will typically not allow you to actually make anything much of that in the Discussion. On (2), I actually don’t agree that the presence of some significance result implies power was adequate. This gets back to the issue of selection bias: if you have a whole bunch of small effects, and you’re doing many tests at once, you’re going to end up detecting a small fraction of them, but power is still very low! And the results are then even more misleading, because they look highly selective and very strong, when in fact they’re not so at all. I talked more about this in my 2009 commentary, and this is basically the substance of Ioannidis’ 2008 critique.

1) the true effect size is not equally distributed across the confidence interval. It follows a normal curve centred on the CI. This means that given a CI around 0.75, the true effect is more likely to be 0.75 than at either ends of the interval (e.g., 0.06 or 0.95). It’s a small point, but it does somewhat counter the implication you made that the effect is as likely to be 0.06 (trivial) as 0.75 (large).

2) I wonder if you are slightly misrepresenting the logic of Friston’s argument? The fictional reviewer in point 4 is claiming a) small sample size = insufficient power – which is untrue in the sense that a properly controlled significance test indicates power was sufficient. I think this kind of poor argument from the fictional reviewer is worth rebutting. Though I agree with your points that Friston’s rebuttal is also problematic for the reasons you describe.

Thanks for raising the debate. I found that the Friston paper was not helping at all to have a sane reviewing situation. It seems to pitch the conversation between reviewers and investigators as some sort of formal game, like a political debate.

But, I often wonder: is there any truth in reviewers’ comments against small samples? I think that they are correct if it raises the suspicion that the authors added participants until they found a significant effect, which inflates the alpha. Is there any other good reason for criticizing small samples with significant effects?

I tend to pace a lot as well. I’ll write a few hundred words then get up and wander round for a bit. I’ll often do that when I just want to think about something for a while. I’ll probably do it right after sending this post.

I’m always amazed by people who write essay plans and the like, since, with the exception of the barest of intro/methods/results/discussion frameworks, I can only ever structure my writing on the fly, writing a few paragraphs and then moving them around to get the right flow.