Twitter | Search | |
michael_nielsen May 6
Replying to @michael_nielsen
In 2000 I was visiting the laboratory of an atomic physicist. His main project was to produce BEC. He'd spent 4 years, perhaps 10,000 hours of staff time (including much of his own), & hundreds of thousands of dollars on this. He'd seen absolutely nothing.
Reply Retweet Like
michael_nielsen May 6
Replying to @michael_nielsen
He was somewhat mournful about his failure to see BEC. But the lesson he drew wasn't that the original papers were wrong. It's that he still had more work to do.
Reply Retweet Like
michael_nielsen May 6
Replying to @michael_nielsen
I asked him what he thought was wrong. "I don't know. I think it might be a problem with the power supply." He described all the (many, many!) things he was doing to get cleaner current, as well as half a dozen other issues it might be.
Reply Retweet Like
michael_nielsen May 6
Replying to @michael_nielsen
Two years later I saw him again, a few months after he'd achieved BEC. I asked what had been the problem. "Turns out it was the power supply!" he said. Not quite beaming -- he's not a beamer -- but as close as he got.
Reply Retweet Like
michael_nielsen May 6
Replying to @michael_nielsen
Actually, when I dug down into details, he'd changed or fixed a _lot_ of things in the intervening two years. And it's hard to be sure. Maybe some of those other things were essential, too. Hard to test the counterfactual.
Reply Retweet Like
michael_nielsen May 6
Replying to @michael_nielsen
This type of story is very common. Often, failure to replicate means the experimentalist needs to do more work. The source of the trouble is often tacit knowledge or uncontrolled elements in the original paper.
Reply Retweet Like
michael_nielsen May 6
Replying to @michael_nielsen
This most emphatically doesn't mean the original paper is bad. Indeed, as with BEC, the original paper may be extremely good. Instead, it may mean more work is needed to understand exactly what's required to see the effect. The original paper is merely an important first step.
Reply Retweet Like
michael_nielsen May 6
Replying to @michael_nielsen
Another good example: measurements of the quality factor of sapphire (basically, how good a lasing substance is it) differed by _orders of magnitude(!)_ between Russia and the West during the cold war.
Reply Retweet Like
michael_nielsen May 6
Replying to @michael_nielsen
It took more than 20 years to sort this out! Turns out it was due to tacit knowledge available in the Russian lab that wasn't known in the West. Story is told here:
Reply Retweet Like
michael_nielsen May 6
Replying to @michael_nielsen
It's worth noting: some Western scientists thought this meant the Russian results were wrong. Turns out it was the Westerners who were wrong. (The Q of sapphire was a hot topic, as it was thought to be relevant for the detection of gravitational waves. So, not small stakes.)
Reply Retweet Like
michael_nielsen May 6
Replying to @michael_nielsen
A tempting response is to say "Oh, the paper should have included more detail." But first-rate experiments often include a mindboggling number of details that have to be gotten right. Figuring those out is (rightly) the decades-long task of an entire community doing followup work
Reply Retweet Like
michael_nielsen May 6
Replying to @michael_nielsen
If you don't believe this, look at the miniscule details Collins paper on the Q of sapphire. Or write out a list of all the possible noise sources in your power supply that might muck up an experiment. (I'll be waiting when your list passes 100 items.)
Reply Retweet Like
michael_nielsen
The "failure to replicate = bad" narrative is tempting. But it's a dramatic misunderstanding & oversimplification of how science works. I wish people had better mental models, to understand that failure to replicate is often instead merely a step along the way to understanding.
Reply Retweet Like More
Jacob D Biamonte May 6
Replying to @michael_nielsen
I’m biological sciences “failure to replicate” to me (an outsider) seems a clear argument against the experiment. In physics the most I’ve seen is “we didn’t observe this effect”.
Reply Retweet Like
Within Without May 6
Replying to @michael_nielsen
Failure to replicate is valuable because it teaches that there is an uncontrolled variable that is affecting the outcome. Identification of this uncontrolled variable using the scientific method is how knowledge and understanding is advanced.
Reply Retweet Like
M Stone May 6
True, but in too many instances flawed experimental design & analysis, confirmation bias, pressure to publish (esp. ‘novel’ results), please peers (especially powerful senior ones), and requirements for professional advancement are major contributors to lack of replicability.
Reply Retweet Like
Ben Reinhardt May 6
Replying to @michael_nielsen
What's the difference between the BEC and the unreplicable priming studies where the original author said effectively "you didn't do it right"? In the former it feels right to put burden of proof on the replicator, the former it feels right to put it on the OA.
Reply Retweet Like
infosik May 6
Replying to @michael_nielsen
Thank you. Balls thread.
Reply Retweet Like
David W. Locke May 6
Replying to @michael_nielsen
In a factor analysis, the first three factors are quick and cheap. The 200th factor is expensive and minute in the quantity of variation for which it accounts. That factor is a long way down the tail. Do we communicate the whole tail?
Reply Retweet Like
Hazem A. M. Awad,PhD May 7
Replying to @michael_nielsen
True but if multiple teams fail to replicate or, worse, if the original paper authors fail to properly replicate then we have real issues with the original result. This is far more acute a problem in the non hard sciences, e.g. humanities hence current replication crisis
Reply Retweet Like