OPINION - It is not surprising that a case of scientific fraud occasionally comes to light, according to associate professor Empirical Political Science Alex Lehr. ‘Many of us are doing our stinking best to maintain scientific integrity and produce rigorous research. But we do that mostly despite the incentives created by the academic system, not because of them.’

A recent case of scientific fraud by a – now dismissed – colleague at Radboud University prompted me to go off on a long rant on social media. Vox asked me to share it here in slightly edited form. This is not something I would normally be inclined to do, but in light of current trends in science and society, I guess it is good to speak openly and clearly about these things. So, here it goes, with the provision that I am not a methodologist and not a philosopher of science, just a practicing researcher.

Don’t hate the player, hate the game

A very understandable reaction to academic integrity violations is to view them as idiosyncratic, malicious actions of individuals. We shrug our shoulders and continue business as usual. I think it’s the wrong reaction.

The real issue here is the misalignment of incentives with the desired outcomes. We get rewarded for telling simple and clean stories – preferably the kind of stories others want to hear. For applying a credible looking veneer of rigorous science to those stories. For theoretical novelty. For demonstrating productivity with lots of publications, preferable “top-publications”. That’s what’s getting us the research grants, the career promotions, the media attention and the policy influence.

What we are not getting rewarded for, is being open about how complex, messy and uncertain research results can be. For articulating all those pesky assumptions our inferences are based on – or even for worrying about them too much ourselves. For showing how variable our results are, depending on the models we choose apply to our data. For being transparent about all the decisions we make as part of our scientific workflow, at best based on reasoned and reasonable trade-offs between practical feasibility and maximum rigor rather then self-serving opportunism.

For spending the time, effort and money needed for collecting high-quality data when there are cheaper short-cuts available. For showing the results of our analyses that don’t fit so nicely with the story. For admitting that the data are simply too noisy to provide a clear signal. For taking the time that is necessary to complete research projects that adhere to FAIR principles, rather than just pumping out another publication. For doing the kind of theoretical and empirical grunt work that is needed to contribute to the accumulation of knowledge instead of just telling interesting “new” stories.

By no means universally, with large variations between different areas of study and between academic communities, and with incremental improvements over time. But still way too pervasively.

I do believe that many of us are doing our stinking best to maintain scientific integrity and produce rigorous research. But let’s not kid ourselves, we do that mostly despite the incentives created by the academic system, not because of them.

So, by all means, hold individual scientists accountable. But also recognize that they might merely be responding to the incentives that we, as a scientific community, are creating. Be willing to consider that the known cases of academic fraud may well be the symptom of a much more systemic problem.

A small victory for open science

As much as we would like to take it for granted that science is always self-correcting, in many areas of study where repeated, independent replication is uncommon or difficult, corrections can only happen when errors in existing research are identified. This requires transparency. And while transparency alone is not sufficient for producing scientific integrity, it sure does create some helpful guardrails.

The fraud case at Radboud University came to light in part because at least some information was openly archived in the Open Science Framework (OSF) repository. Not so long ago, it likely would never have come to light at all. So, as painful as what happened may be, there is a real silver lining here. I hope that the lesson we take from this is to push for more open science. Because in many areas of study, it is still not the norm.

Light-switch

‘Any sufficiently crappy research is indistinguishable from fraud’. This quote is not mine, it’s Andrew Gelman’s variant of Clarke’s third law. But it highlights an important insight: scientific rigor is not a light-switch that’s either on or off. Some research is not outright fraudulent with malicious intent, it’s simply so shoddy that the difference becomes meaningless.

Yet, standards are so variable between researchers, and subject to so much change over time, that it’s unavoidable that what one considers perfectly valid research practices, others may view as hanging offences. This appears to have happened here, going by the reaction of the researcher involved. Science is hard, science is messy, and science is always changing.

How come someone felt compelled and justified to do science in this way?

So, someone violated scientific integrity and got dismissed (though for other reasons). The discussion we should be having is what this researchers’ scientific community, its culture, and the standing research and career policies, did to prevent this situation from even occurring in the first place. If what happened in this case is not in line with desired practices, how come someone felt compelled and justified to do science in this way? Are we sure it’s an extreme and idiosyncratic case, or are there now quite a few others out there thinking ‘ouch, this could have been me?’.

And to avoid misunderstanding, I’m certainly not singling out a specific research group, department, research institute or even our university. These issues are pervasive and systemic.

I don’t think firing someone without also having some critical reflection on the role of the community and the institute in the wider sense, is the best course of action. In the worst case, it might create an atmosphere of fear. Fear of being found out, fear of falling short of scientific standards, fear of being punished for honest mistakes. That kind of fear might just create more incentives for not providing full transparency on one’s research, and for creating plausible deniability and outsourcing risk (“it wasn’t me, it was my research assistant…”).

Deliberate and outright fraud is a no-go, but admitting to honest mistakes, being open to learn and revise, and being willing to retract publications when necessary, should be stimulated, and never be career-ending offences (just as “prestigious” research grants or “top-publications” based on shoddy research should never be career-making acts). That message should be a lot clearer.

Growth will be painful

I think one of the most frustrating issues for those of us that view these things as being problematic can be the inertia of academia. At the same time, it’s perfectly understandable. To a large extent, science is governed by scientists. Scientists that gained their positions by being academically successful. Leading change when the implication of that change is that practices that brought you success may no longer be acceptable can create quite a bit of cognitive dissonance. Good academic leaders are able to overcome themselves, but it takes some guts, humility and self-reflection.

Science is also a social endeavor, and that, too, makes change difficult. You will not always make friends by being the critical voice in the room, especially not if the academic culture is not one that takes kindly to criticism. Power dynamics make this even more difficult. Not everyone dares to openly criticize the more senior figures, especially not when they hold powerful positions in one’s academic network. And criticism towards more junior colleagues and students always creates a difficult balancing act between being critical and being motivating.

I think some pain will be unavoidable if we want to do better. But we can do some things to mitigate it. We can be open about our own limitations and our own need to grow, especially those of us that are already well-established scientists. We can make clear that is not and will never be about doing perfect research or being the perfect scientist, but about the willingness to be open and to learn. We can strive for an academic culture that values scientific criticism above all else, while maintaining a sharp boundary between scientific criticism and personal criticism.

Never a good time to rock the boat

“The best time to plant a tree was 20 years ago. The second best time is now”. That old proverb will likely ring true in general to most. But probably not so much to everyone inside academia when it specifically comes to criticizing the scientific system right now.

These are times of greater uncertainty about the future of science, with threats from outside and from within. Discussing these problems openly may seem to make us even more vulnerable, and I think some would much prefer a ‘closing of the ranks’. This is a completely understandable position, it just happens to be one I disagree with. For reasons related to philosophical foundations of science that I won’t even try to go into here – the debates are old, probably fundamentally unresolvable, and mutual understanding has been hard to come by even under the best circumstances. But more simply also for strategic reasons.

If science wants to maintain its position as an autonomous institution that critically engages with knowledge and power, it might be especially important nowadays to show that it deserves that position. To me, that means being self-reflective, recognizing our own problems and being willing to learn from them. It means showing a demonstratable, continued commitment to seeking truth. Even if we don’t like that truth, even if that truth is riddled with complexity, ambiguity and uncertainty. This won’t be enough to safeguard our scientific autonomy by itself, but I believe not doing so will make us even more vulnerable.

Be the change you want to see

Structural problems require structural solutions. What we need first and foremost are research policies, grant allocation schemes and career reward systems that are better aligned with stimulating scientific integrity. But individual scientists can also show some agency. Don’t wait around for the system to change, enact the changes that you would like to see if you can, where you can. Especially if you are already well-established. Publish less, but publish better research. Put time and effort into transparency. Share everything you can share, as openly as you can share it. Use your privileged position to do research in the way you think it ought to be done, even if that’s not the quickest way to achieve academic success.

‘Nurture young scholars that struggle with the perverse incentives of the system’

Don’t hide uncertainty, embrace it, and show it. If you are sitting on hiring/award committees, don’t just look at where and how much people publish, look at what they publish and how they got their results. Be aware of the implicit signal you might be giving those you supervise when you say things like ‘you need to get a result’ or ‘we need to make this publishable’.

Nurture young scholars that struggle with the perverse incentives of the system, help them channel their frustration into something productive. Show them that to some extent, the rules of the game are also a social construct we create ourselves, and we can bend, break and change them in the interest of better science. Call out bad research practices, even if it won’t make you popular. Don’t pretend to be perfect, but try to be better.