How to think scientifically about scientists’ proposals for fixing science

I wrote this article for a sociology journal: Science is in crisis. Any doubt about this status has surely been been dispelled by the loud assurances to the contrary by various authority figures who are deeply invested in the current system and have written things such as, “Psychology is not in crisis, contrary to popular rumor . . . Crisis or no crisis, the field develops consensus about the most valuable insights . . . National panels will convene and caution scientists, reviewers, and editors to uphold standards.” (Fiske, Schacter, and Taylor, 2016). When leaders go to that much trouble to insist there is no problem, it’s only natural for outsiders to worry . . . When I say that the replication crisis is also an opportunity, this is more than a fortune-cookie cliche; it is also a recognition that…
Original Post: How to think scientifically about scientists’ proposals for fixing science

My review of Duncan Watts’s book, “Everything is Obvious (once you know the answer)”

My review of Duncan Watts’s book, “Everything is Obvious (once you know the answer)” Posted by Andrew on 18 May 2017, 3:37 pm We had some recent discussion of this book in the comments and so I thought I’d point you to my review from a few years ago. Lots to chew on in the book, and in the review.
Original Post: My review of Duncan Watts’s book, “Everything is Obvious (once you know the answer)”

“P-hacking” and the intention-to-cheat effect

“P-hacking” and the intention-to-cheat effect Posted by Andrew on 10 May 2017, 5:53 pm I’m a big fan of the work of Uri Simonsohn and his collaborators, but I don’t like the term “p-hacking” because it can be taken to imply an intention to cheat. The image of p-hacking is of a researcher trying test after test on the data until reaching the magic “p less than .05.” But, as Eric Loken and I discuss in our paper on the garden of forking paths, multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. I worry that the widespread use term “p-hacking” gives two wrong impressions: First, it implies that the many researchers who use p-values incorrectly are cheating or “hacking,” even though I suspect they’re mostly…
Original Post: “P-hacking” and the intention-to-cheat effect

“Everybody Lies” by Seth Stephens-Davidowitz

Seth Stephens-Davidowitz sent me his new book on learning from data. As is just about always the case for this sort of book, I’m a natural reviewer but I’m not really the intended audience. That’s why I gave Dan Ariely’s book to Juli Simon Thomas to review; I thought her perspective would be more relevant than mine for the potential reader. I took the new book by Stephens-Davidowitz and passed it along to someone else, a demanding reader who I thought might like it, and he did: he kept coming to me with new thought-provoking bits that he’d found in it. So that’s a pretty solid endorsement. I couldn’t convince him to write a review so you’ll have to take my word that he liked it. The thing I found most appealing about the book was that, in addition to…
Original Post: “Everybody Lies” by Seth Stephens-Davidowitz

Honesty and transparency are not enough

[cat picture] From a recent article, Honesty and transparency are not enough: This point . . . is important for two reasons. First, consider the practical consequences for a researcher who eagerly accepts the message of ethical and practical values of sharing and openness, but does not learn about the importance of data quality. He or she could then just be driving very carefully and very efficiently into a brick wall, conducting transparent experiment after transparent experiment and continuing to produce and publish noise. The openness of the work may make it easier for later researcher to attempt—and fail—to replicate the resulting published claims, but little if any useful empirical science will be done by anyone concerned. I don’t think we’re doing anybody any favors by having them work more openly using data that are inadequate to the task. The…
Original Post: Honesty and transparency are not enough

“An anonymous tip”

“An anonymous tip” Posted by Andrew on 9 May 2017, 12:27 am [cat picture] I and a couple others received the following bizarre email: **’s research is just the tip of the iceberg. If you want to expose more serious flaws, look at research by his co-authors – ** at ** and ** at **. I won’t be checking this disposable e-mail address again. People send me tips all the time where they ask to maintain anonymity, and I do this. This one’s just strange with all the cloak-and-dagger stuff, also because the correspondent didn’t supply any details. If there’s a problem with some published work, you could point to the publication in question and say what the problems are, no? I would not be surprised if **’s co-authors’ work has serious flaws; at this point I suppose that any readers…
Original Post: “An anonymous tip”

Discussion with Lee Jussim and Simine Vazire on eminence, junk science, and blind reviewing

Lee Jussim pointed me to the recent article in Psychological Science by Joseph Simmons and Uri Simonsohn, expanding on their blog post on flaws in the notorious power pose article. Jussim then commented: I [Jussim] think that Cuddy/Fiske world is slowly shrinking. I think your “What Has Happened Here…” post was: 1. A bit premature ONLY in its claim that Fiske was working in a dead paradigm BUT 2. Inspirationally aspirational — that is Exactly what most of us are shooting for — to kill that paradigm. and, therefore 3. Perhaps that claim in your post, though not descriptive, may actually be prescient. In that spirit, you may enjoy this killer unpubbed paper by Vazire: Against Eminence And you can find a great turn of phrase in this Freakonomics Podcast: “We were practicing eminence-based medicine” which I have used to…
Original Post: Discussion with Lee Jussim and Simine Vazire on eminence, junk science, and blind reviewing

A completely reasonable-sounding statement with which I strongly disagree

From a couple years ago: In the context of a listserv discussion about replication in psychology experiments, someone wrote: The current best estimate of the effect size is somewhere in between the original study and the replication’s reported value. This conciliatory, split-the-difference statement sounds reasonable, and it might well represent good politics in the context of a war over replications—but from a statistical perspective I strongly disagree with it, for the following reason. The original study’s estimate typically has a huge bias (due to the statistical significance filter). The estimate from the replicated study, assuming it’s a preregistered replication, is unbiased. I think in such a setting the safest course is to use the replication’s reported value as our current best estimate. That doesn’t mean that the original study is “wrong,” but it is wrong to report a biased estimate…
Original Post: A completely reasonable-sounding statement with which I strongly disagree

7th graders trained to avoid Pizzagate-style data exploration—but is the training too rigid?

7th graders trained to avoid Pizzagate-style data exploration—but is the training too rigid? Posted by Andrew on 5 May 2017, 9:51 am [cat picture] Laura Kapitula writes: I wanted to share a cute story that gave me a bit of hope. My daughter who is in 7th grade was doing her science project. She had designed an experiment comparing lemon batteries to potato batteries, a 2×4 design with lemons or potatoes as one factor and number of fruits/vegetables as the other factor (1, 2, 3 or 4). She had to “preregister” her experiment with her teacher and had basically designed her experiment herself and done her analysis plan without any help from her statistician mother. Typical scientist not consulting the statistician until she was already collecting data. She was running the experiment and after she had done all her batteries and…
Original Post: 7th graders trained to avoid Pizzagate-style data exploration—but is the training too rigid?

When do stories work, Process tracing, and Connections between qualitative and quantitative research

When do stories work, Process tracing, and Connections between qualitative and quantitative research Posted by Andrew on 11 January 2017, 9:45 am Jonathan Stray writes: I read your “when do stories work” paper (with Thomas Basbøll) with interest—as a journalist stories are of course central to my field. I wondered if you had encountered the “process tracing” literature in political science? It attempts to make sense of stories as “case studies” and there’s a nice logic of selection and falsification that has grown up around this. This article by David Collier is a good overview of process tracing, with a neat typology of story-based theory tests. Besides being a good paper generally, section 6 of this paper by James Mahoney and Gary Goertz discusses why you want non-random case/story selection in certain types of qualitative research. This paper by Jack Levy…
Original Post: When do stories work, Process tracing, and Connections between qualitative and quantitative research