An 85% computational reproducibility rate sounds depressingly low to me, I'm flabbergasted that this is presented as "high" in the original paper. Only an 85% chance that your conclusions match the data you collected does not bode well for the overall correctness rate of studies when you take into account data collection issues. In a sane world this would be >99%.
Ah, ok, the 15% included papers where code to reproduce the results was not provided by the authors, even if the results actually were correct. That's a little better. Also makes the headline number rather misleading, that no longer seems like as useful a statistic. I want to know whether papers are *correct*, not just how helpful the authors felt like being.
I was wondering about that, thanks. Did those authors promise to share code/data though? It's often required. I also wonder how anyone can check their conclusions if the code isn't provided unless there described the algorithm/model carefully in the paper.
Also wow, that authors list. How many people does it take to check the calculations in a paper?
The comments from Sudhof are just so unfathomably mercenary, I'm a little shocked. As far as I can see, this is still very much in the "okay, could be fraud, could be sloppiness" territory, but the way he's responded makes me suspect fraud much more than I otherwise would.
It's just such a rote, half-assed attempt to play some sort of identity politics card, claiming that having errors pointed out in his team's work is particularly bad for the women in his team, somehow, for no specific reason, then refusing to give any examples of that being the case or provide anything to back that up. It's just a sort of automatic, knee-jerk "oh, I'm under attack, this is, um [spins wheel] sexist - what's that you say - I'm a man - erm I guess it's bad for some women who are involved, no I don't know whom, but, you know demanding data not be faked or contain sloppy errors is, erm, bad for women in some generalised way because, erm, because accuracy and honesty are more male traits or something, as I said you're the sexist here".
It's the laziest attempt to draw some sort of conceptual line between someone criticising him and some bad thing I can remember seeing, and it's just so offensive itself, to just assert as if it's obvious that having scientific standards, expecting errors to be corrected, acting against fraud, is particularly bad for women. This is implicitly a claim that women are less honest and/or more sloppy than men!
Your vigilance and others are keeping science honest. Thank you for that. **but**
I was a bench scientist/researcher in biotech and have never met a chief scientists/CEO that didn't spin pushing the envelope to get the funding necessary to advance science. Some very good discoveries have come from data showing promise but needing spinning to get the funding to make that breakthrough discovery.
An 85% computational reproducibility rate sounds depressingly low to me, I'm flabbergasted that this is presented as "high" in the original paper. Only an 85% chance that your conclusions match the data you collected does not bode well for the overall correctness rate of studies when you take into account data collection issues. In a sane world this would be >99%.
Ah, ok, the 15% included papers where code to reproduce the results was not provided by the authors, even if the results actually were correct. That's a little better. Also makes the headline number rather misleading, that no longer seems like as useful a statistic. I want to know whether papers are *correct*, not just how helpful the authors felt like being.
I was wondering about that, thanks. Did those authors promise to share code/data though? It's often required. I also wonder how anyone can check their conclusions if the code isn't provided unless there described the algorithm/model carefully in the paper.
Also wow, that authors list. How many people does it take to check the calculations in a paper?
The comments from Sudhof are just so unfathomably mercenary, I'm a little shocked. As far as I can see, this is still very much in the "okay, could be fraud, could be sloppiness" territory, but the way he's responded makes me suspect fraud much more than I otherwise would.
It's just such a rote, half-assed attempt to play some sort of identity politics card, claiming that having errors pointed out in his team's work is particularly bad for the women in his team, somehow, for no specific reason, then refusing to give any examples of that being the case or provide anything to back that up. It's just a sort of automatic, knee-jerk "oh, I'm under attack, this is, um [spins wheel] sexist - what's that you say - I'm a man - erm I guess it's bad for some women who are involved, no I don't know whom, but, you know demanding data not be faked or contain sloppy errors is, erm, bad for women in some generalised way because, erm, because accuracy and honesty are more male traits or something, as I said you're the sexist here".
It's the laziest attempt to draw some sort of conceptual line between someone criticising him and some bad thing I can remember seeing, and it's just so offensive itself, to just assert as if it's obvious that having scientific standards, expecting errors to be corrected, acting against fraud, is particularly bad for women. This is implicitly a claim that women are less honest and/or more sloppy than men!
Criticizing science because human scientists commit error and fraud reminds me of Winston Churchill's comment about democracy:
"'It has been said that democracy is the worst form of government except all the others that have been tried."
Show us another field of human endeavor that has produced even half the beneficial results of the scientific method.
Criticising corruption in science isn't criticising science, it's criticising corruption 🙄
Your vigilance and others are keeping science honest. Thank you for that. **but**
I was a bench scientist/researcher in biotech and have never met a chief scientists/CEO that didn't spin pushing the envelope to get the funding necessary to advance science. Some very good discoveries have come from data showing promise but needing spinning to get the funding to make that breakthrough discovery.