Comments on: the capricious nature of p < .05, or why data peeking is evil

By: Crimes and Misdemeanors: Reforming Social Psychology » Random Assignment

Crimes and Misdemeanors: Reforming Social Psychology » Random Assignment — Tue, 10 Jul 2012 19:01:46 +0000

[...] data substantially raises the probability of a false positive finding (for more discussion read Why Data Peeking is Evil by [...]

By: [citation needed]» Blog Archive » The psychology of parapsychology, or why good researchers publishing good articles in good journals can still get it totally wrong

Mon, 10 Jan 2011 07:53:04 +0000

[...] significant result. This may seem like a harmless little foible, but as I’ve discussed elsewhere, is actually a very bad thing, as it can substantially inflate Type I error rates (i.e., false [...]

By: John

John — Tue, 28 Dec 2010 22:15:33 +0000

Great entry! Could you please give us some references or keywords to the literature on how to adjust p values for data peeking? Thanks.

By: [citation needed]» Blog Archive » the ‘decline effect’ doesn’t work that way

Thu, 16 Dec 2010 02:25:43 +0000

[...] written a bit about these issues myself in a number of papers (1, 2, 3) and blog posts (1, 2, 3, 4, 5), so I’m partial to such concerns. Still, important as the role of the various [...]

By: Tal Yarkoni

Tal Yarkoni — Wed, 12 May 2010 23:54:02 +0000

Yigal, I think the idea of pre-registering studies is a fine one in principle, but I’m not sure it’s really viable in practice for the vast majority of psychology research. There are just too many studies, most of them very small, and the stakes are too small for it to be worth anyone’s while to encourage, let alone mandate, using a database of this sort.

Neuroskeptic, I’m not sure how widespread the practice is; I don’t know that every clinical trial properly accounts for peeking. But I do know that there’s a sizable statistical literature on this issue specifically in the context of clinical studies, so I assume that at least a reasonable proportion of clinical trials do this…

By: Neuroskeptic

Neuroskeptic — Wed, 12 May 2010 20:48:37 +0000

“The other alternative is to explicitly correct for data peeking. This is a common approach in large clinical trials…”

I did not know that. I mean I knew they peeked, but I didn’t know there were specific corrections. Thanks!

By: Yigal

Yigal — Fri, 07 May 2010 14:22:53 +0000

Nice post! These are tough issues for all of us, and the pressure to publish certainly doesn’t help. Maybe the solution is pre-registration of studies like they do for clinical trials, so the reader can know the original number of subjects the authors intended to recruit.