The real world is full of misinformation, bad intentions, and lies. That’s what drew me to science. To be sure, there were mistakes. My middle school textbook discussed the impending ice age, and the man-made hole in the ozone layer. Apparently the Aqua Net favored by the theater department for turning normal hair into a plastic facsimile was also favored by hominids forty thousand years ago. But over time, science comes closer and closer to the truth.

And those involved in the creation of knowledge (pretentiousness alert) try. We really do. But we get it wrong sometimes. I remember the first time I encountered this. I was devastated. At fourteen years old a lot of things devastate you. This instance was cold fusion. Now a footnote in history, at the time reasonable people all across the world thought that our energy problems were over. A team at some university or other had created a fusion reaction at room temperature. If this could be replicated then eventually every home in the world could have a table top reactor creating virtually unlimited energy, giving off only helium as a byproduct. But it couldn’t be replicated, and so, after several attempted replications, scientists moved on.

The point is, there was a process. It worked because of the free and accurate sharing of information. Years later, as a grad student I was sitting at a table with (I’m pretty sure, memory is funny though) Benson Honig. We were chatting about data availability and replicating research. He had very strong opinions, opinions that I agreed with as a grad student. I promised that I personally would provide my data to anyone who asked. (As a result of that promise I haven’t published out of my dissertation because I was robbed and my data disappeared. Such is life.) We agreed that data should be available, the analysis should be spelled out, and everything should be made available, upon request, to other scientists who might want to extend the work or even just check it.

It was with that background I had the good fortune to hook up with a talented bunch of authors working on a large project using the PSED. This thing could be a dissertation or two, and will end up being several papers. The first was recently accepted. I’m thrilled that academe has found our work interesting enough to share. I’m disappointed in what we found.

We attempted first to reproduce a bunch of analysis from published papers, then replicate and extend the research. We assumed that the reproduction was going to be simple and we’d start having problems when we attempted to replicate and extend. Nope. I personally was unsuccessful with every reproduction I attempted. I was unable to get to the same N as any of the studies I was examining. This was frustrating and I thought I was going to be kicked out of the club for my incompetence. Fortunately others had similar problems and our failure became the story.

But our failure was different than the cold fusion failure. We didn’t fail the experiment, we couldn’t even set the experiment up. We couldn’t figure out from the published research what beakers, batteries, and solution we should be using to reproduce the experiment. This is a real problem, and one we describe in great detail in the paper, along with making several suggestions about how to limit this problem in the future.

What we discovered was, at least to me, interesting. That may be why it’s being published. But what we found wasn’t really new. Much has been written about problems with replication, first in psychology I believe, then in other fields. The term ‘replication crisis’ was coined in the ‘early 2010’s’ according to everyone’s favorite secret research buddy, Wikipedia. Stories of faked data ending careers and invalidating research are not uncommon. Even completely tongue-in-cheek articles are shepherded through the publication process with the journal editors missing the joke. There are problems. A fantastic part of the scientific process is becoming aware of these problems and working to fix them. Our article is an attempt to do part of that work and I hope that it makes a small contribution.