|The beauty of being wrong: A plea for post-publication revision||
October 18 2012
By Sebastiaan Mathôt 2184 reads
I realize that most of what I will ever write as a scientist is wrong. Not because it is obviously wrong when I write it, or because I don't believe in my own research. But simply because virtually every insight is eventually replaced by some newer and generally more accurate insight. This is true even for the great theories of the likes of Darwin, Einstein, and Newton. And it is certainly true for the minor contributions of the remaining 99.99% of the research population, for whom supersession generally occurs quite rapidly.
Perhaps this sounds depressing, but it's really not. It's just a (negative) way of describing progress. It's a good thing. Scientists should be encouraged to acknowledge the wrongness of their theories, to find out what's so wrong about them, and to replace them by newer theories that are still wrong, but less so. You could even argue that the possibility of being wrong is what differentiates science from opinionist fields such as philosophy, art, and economics.
But the ideal model of a scientist who continually seeks to disprove himself is not very compatible with human nature. Being human after all, scientists hate to be proven wrong, and will go through great lengths (including plain denial) to avoid this from happening. Nevertheless, given a healthy environment, scientists can, to some extent, overcome their innate dislike of being wrong. But only given a healthy scientific environment. And this is, in my opinion, where things go, well... wrong.
Lately, there has been much ado about reforms of academic publishing. Most of the debate has focused on the slow, but steady shift towards open access models (i.e. making scientific papers publicly accessible). I'm a big proponent of open access, but I also think that there are deeper problems that go beyond copyright issues. More specifically, I think that the way in which scientific results are presented is detrimental to a healthy scientific climate.
Scientists produce papers that ostensibly describe definite, proven results. Once a paper goes into the massive archives, there is no going back. The results, the interpretation given to these results, the opinion of the author, and the general zeitgeist, are all frozen in time, with no real chance of revision. And once a paper has been published, the author is assumed, by colleagues, to be committed to it. There is even a chance that it will be frowned upon when an author openly distances him or herself from a previous work: Are you saying that you made a mistake? Why did you publish, if you weren't sure? Or worst of all: If you don't believe the paper anymore, why don't you retract it? No wonder, therefore, that scientists are reluctant to revise a theory or question results that they have presented in a published paper.
Phrased differently: Rather than fostering a healthy environment in which scientists seek to disprove themselves, the format in which results are presented (as individual, seemingly definite publications) strengthens the already firm human tendency to avoid disproval.
One result of this is that there are many unnecessary (except perhaps for the sake of entertainment) disputes between scientists. Most commonly, this happens when two researchers independently publish results that (appear to) conflict. This can happen for any number of reasons, and in many cases there is no obvious reason to prefer one result over the other. (For example, imagine Huygens and Newton feuding over whether light consists of waves or particles, before scientists arrived at the conclusion that it's a bit of both.) So both researchers will fall back to the default position of preferring their own result. What is obviously lacking here, is an incentive for these researchers to engage in a constructive dialogue, because that would require them to acknowledge the dangerous possibility that they may have published something that was wrong. (Some researchers nevertheless do engage in a constructive dialogue, but the point is that they are hardly encouraged to do so.)
In this respect, I think that science can learn a lot from how software is developed. There are striking parallels between science and software development: Both are gradual, never-ending processes that are characterized by continuous refinement. Just as a scientist publishes a paper, a developer will release a new version of his or her software. And, just like a paper will be eventually superseded by a new paper, a software release will be superseded by a new release.
But the attitude towards progress is markedly different between scientists and developers. Scientists tend to mistakenly perceive their papers as finished products, in large part because of the definite nature of the publication process: Once a paper has been published, it's there to stay. New papers appear frequently, of course, but the paper itself is never updated. And when a paper turns out to be (partly) wrong, this is perceived to some extent as a failure on the part of the author. In contrast, software developers tend to see releases as snapshots of a particular moment during a continuous process. Consequently, developers are not overly committed to a particular release. If a user reports a bug, the developer will take note and fix it for the next release. New releases are celebrated, and the fact that updates are necessary is not perceived as failure.
Before the internet era, there was no real alternative to disseminating scientific results in the form of discrete, published papers, so this model of academic publishing made sense. But nowadays there is much more flexibility (thanks to computers) and the old model is outdated to the point of being dysfunctional.
But what can be done to improve it? Perhaps it goes to far (or does it?) to suggest that scientists, like many developers, conduct their work on publicly accessible versioning systems (kind of like an open DropBox with infinite history). But at the very least, journals should allow authors to revise and extend papers after they have been published. Now that the internet is the primary way in which papers are distributed, there is no reason why this cannot be done. In fact, this is already implemented on so-called pre-print servers, such as ArXiv and the soon-to-be-launched PeerJ PrePrints (the latter is an assumption).
You might argue that post-publication revision is unnecessary, because you can always publish a new paper to express your new insights. But this is inadequate for a number of reasons. Firstly, the old paper will still be around without any obvious sign that 'updates' are available. Consequently, outdated papers have a tendency to keep surfacing, long after their expiration date. Secondly, the threshold for a new publication is high. Imagine that you merely want to revise your discussion in light of new findings. Or perhaps you'd like to present an additional analysis. You will not be able to publish such tidbits as a new paper, even though they may be worth sharing with colleagues. Finally, and perhaps most importantly, the possibility to revise published work takes away the false suggestion that science is ever finished, and, by extension, it fosters a more constructive dialogue between scientists.