How does the open science movement address flaws in peer review?

By increasing transparency and involving more stakeholders in the review process. Open data, pre-prints, and public commentary allow for more eyes on research, potentially catching errors or flawed methodologies that might be missed by a small, opaque peer review committee.

What are the common arguments against open science?

Criticisms include the concern that the public may misinterpret complex scientific findings, the theoretical risk of data theft, increased workload for researchers, and potential damage to reputations if minor errors are exposed.

How can open data prevent policy errors, like the Reinhart and Rogoff paper?

If the data and analysis methods are open, errors in calculations or data selection (like the Reinhart and Rogoff Excel error) can be identified and corrected quickly by other researchers, preventing potentially harmful policies based on flawed findings from being implemented.

What are registered reports, and how do they combat publication bias?

Registered reports involve submitting study protocols and analysis plans for review *before* data collection. Journals accept the paper in principle, which guarantees publication regardless of the results, thereby eliminating publication bias towards positive or statistically significant findings.

Why is peer review often uncompensated, and what's the impact?

Peer review is largely uncompensated because major publishers operate on high profit margins by leveraging unpaid academic labor. This lack of compensation can lead to superficial reviews, as reviewers lack incentives and time, impacting the quality and rigor of published science.

What is the historical shift in scientific publishing that led to current profit models?

Before World War II, scientific publishing was often handled by scholarly societies without significant profit motives. Post-WWII, entrepreneurs like Robert Maxwell recognized the potential to profit by using academic journals and unpaid reviewer labor, establishing the dominant for-profit model seen today.

What are the main barriers preventing science from becoming more open?

Primary barriers include institutional inertia, where established practices are hard to change, and fear among researchers. This fear stems from imposter syndrome and the anxiety that making their data publicly available will expose minor imperfections or errors in their work.

Can noise or errors in data collection be accepted in science?

Yes, errors and noise are inherent in data collection, whether from equipment malfunctions or human factors like misgrooved reps in strength training. Recognizing this, open science aims to be transparent about these imperfections, which is more realistic than presenting data as perfectly clean.

Can you share an example of a destructive gym fail?

One speaker described a rack pull session with excessive band tension that caused the entire power rack to tip backward, shattering a mirror and damaging a cinder block wall. This happened because the rack wasn't bolted down and the band tension created an upward lifting force.

What was a dangerous gym fail involving a max bench press?

One individual experienced a near-disaster attempting a max bench press in a basement gym late at night. After missing the rep, they were pinned under the weight and experienced an intense fight-or-flight response, realizing the potential danger of lifting alone without safety measures.

What happened during an eccentric bench press that was a near-fail?

While performing eccentric bench presses with weight exceeding their max, one speaker tore a pectoral muscle. They were caught off guard without a spotter, and the quarter-second descent to the safety pins felt like an eternity, highlighting the critical importance of safety precautions.

Key Moments

Fireside Chat #7: Open science and biggest gym fails

Stronger By Science

Education5 min read72 min video

Aug 6, 2020|3,926 views|130|32

Stronger By Science Podcast

Save to Pod

Key Moments

TL;DR

Open science aims for transparency, facing challenges like added work and potential misuse, but offers significant benefits.

Key Insights

The open science movement advocates for transparency in research by making data, methods, and analysis publicly accessible.

Current peer review processes can be opaque, leading to the publication of questionable research, highlighting a need for greater scrutiny.

Open science can improve research quality and accessibility, breaking down paywalls that limit public access to scientific findings.

Criticisms of open science include concerns about misinterpretation by the public, potential data 'theft,' and increased workload for researchers.

The economics paper by Reinhart and Rogoff, which influenced austerity policies, serves as a cautionary tale of how simple errors in opaque systems can have widespread negative consequences.

Registered reports and pre-registration of study protocols are proposed solutions to mitigate publication bias and increase research integrity, though they add complexity.

The current scientific publishing model, largely driven by for-profit publishers, relies heavily on uncompensated peer review, creating inefficiencies and high profit margins.

Arguments against compensating peer reviewers, such as the fear of superficial work, are largely dismissed as illogical in business contexts.

Institutional inertia and fear of reputational damage due to potential errors in data collection are significant barriers to adopting open science practices.

Despite challenges, the benefits of open science, such as enhanced transparency and accuracy, are believed to outweigh the drawbacks, driven by organizations like STORK in exercise science.

Humorous and potentially dangerous gym fails, like tipping a power rack or getting pancaked during a near-maximal bench press, highlight the risks and sometimes absurdities of the fitness world.

THE OPEN SCIENCE MOVEMENT EXPLAINED

The open science movement is less an organization and more a philosophical shift in how scientific research is conducted and shared. Traditionally, research processes have been opaque, with data, peer reviews, and published articles often hidden behind paywalls or inaccessible. Open science aims to change this by promoting transparency through practices like pre-registering study methods and analysis plans, making data and statistical code publicly available, and utilizing preprint servers for early dissemination of research. The core idea is to make the scientific process more accessible, verifiable, and collaborative for everyone involved.

CRITIQUES OF THE CURRENT PEER REVIEW SYSTEM

The conversation highlights significant flaws within the current academic peer review system. Both hosts express alarm at recently encountered papers with outlandish premises that were nevertheless indexed in reputable databases like PubMed. The fact that these ridiculous studies went viral and were soon retracted or flagged for retraction raises serious questions about the effectiveness of traditional peer review. It suggests that relying on a small number of reviewers, whose identities and reviews are often hidden, allows significant errors and absurd claims to slip through, underscoring the need for more robust and transparent evaluation methods.

BENEFITS AND DRAWBACKS OF OPEN SCIENCE

Proponents of open science argue that increased transparency leads to higher quality research, better consensus-building, and greater ethical considerations, especially for health-related discoveries. It breaks down paywalls that prevent public access to vital information. However, criticisms exist, including concerns that the public might misinterpret complex research, potential for data to be 'stolen' or misused by others, and the added workload and 'busy work' it imposes on researchers. While these concerns are acknowledged, the consensus leans towards the benefits of openness significantly outweighing the potential drawbacks.

CASE STUDIES ILLUSTRATING OPEN SCIENCE'S IMPORTANCE

The discussion features compelling examples illustrating the need for open science. The Reinhart and Rogoff paper, influential in austerity policies, contained a simple Excel error that significantly altered its conclusions when discovered years later. This highlights how errors in opaque systems with uncompensated labor can have profound societal impacts. Another example involves reviewers potentially stealing unpublished work, a risk exacerbated by the current system's opacity. These instances underscore the value of open data and transparent review processes in catching and correcting errors before they cause widespread harm.

ADDRESSING OBJECTIONS: INCREASED WORKLOAD AND COMPENSATION

A common objection to open science is the perceived increase in workload, such as pre-registering study protocols or submitting work as registered reports. While these steps add complexity, they aim to mitigate issues like publication bias and p-hacking by locking in methods before data collection. The conversation also critically examines the lack of compensation for peer reviewers, who perform a significant portion of the journal's work. Arguments against paying reviewers, such as the fear of superficial work, are largely dismissed. The current model, often operated by for-profit publishers with high profit margins, is described as a 'racket' where unpaid labor of researchers fuels substantial corporate gains.

OVERCOMING INSTITUTIONAL INERTIA AND FEAR

The primary barrier to wider adoption of open science appears to be institutional inertia, particularly among more experienced academics who are comfortable with the existing system. There's also a degree of fear, driven by imposter syndrome or concerns about revealing imperfections in data collection. Researchers may worry that open data will expose minor errors, making them appear less competent. However, the argument is made that most datasets have some 'noise,' and transparency can actually foster a more honest scientific environment. Organizations like STORK are actively working to promote open science practices within exercise science and kinesiology.

HUMOROUS YET HAZARDOUS GYM FAILS

Shifting to a lighter tone, the hosts share their most memorable gym fails. Eric recounts an incident involving excessive band tension on rack pulls that caused the power rack to tip backward, shattering a mirror and damaging the wall. Greg shares a harrowing experience of being 'pancaked' by a near-maximal bench press attempt in a basement gym with no spotter, triggering an intense fight-or-flight response. He also recounts tearing his pec during eccentric bench press, saved only by safety pins, highlighting the critical importance of spotters and safety precautions, even when implementing advanced training techniques.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

●Books

●People Referenced

Common Questions

The open science movement aims to make the scientific process more transparent and accessible. This includes making research data, analysis code, and findings openly available, moving away from traditional opaque and paywalled academic publishing.

Topics

Technology & Innovation Science & Mathematics Society & Philosophy Peer Review Scientific Publishing Academic Institutions Research Integrity Publication Bias Open Science Data Transparency Gym Safety

Mentioned in this video

Organizations

PubMed

A database of biomedical literature that indexes journals, mentioned in the context of ridiculous papers being published in indexed journals.

STORK

The Society for Transparency, Openness, and Replication in Kinesiology, an organization leading efforts to promote open science methods in exercise science and kinesiology.

Westside Barbell

A renowned powerlifting gym known for its unique training methods, mentioned as the origin of the extreme use of band tension that led to a gym fail.

Books

British Medical Journal

A significant medical journal where Richard Smith, its former editor, published an article critiquing the peer review process.

Nature

A prominent scientific journal, mentioned in a hypothetical example of an ambiguous name requiring disambiguation.

Companies

Springer Nature

A scientific publishing company with high profit margins, discussed in the context of who profits from academic publishing.

Taylor & Francis

A major academic publisher with high profit margins, discussed in the context of the profitability of the scientific publishing industry.

Wolters Kluwer

A global provider of professional information, software solutions, and services; a publisher with high profit margins, discussed in the context of industry profitability.

Elsevier

A major publisher of scientific, technical, and medical information, noted for its high profit margins, which could be used to pay peer reviewers.

People

Louie Simmons

Founder of Westside Barbell, whose training methods were extremized by some of his followers, leading to the gym fail involving excessive band tension.

Robert Maxwell

A businessman who built an empire by applying magazine publishing models to academic journals, exploiting unpaid academic labor and generating high profits.

Reinhardt and Rogoff

Researchers who published a paper in economics suggesting high government debt negatively impacts GDP growth, later found to contain a calculation error.

Software & Apps

SPSS

A statistical software package that uses a point-and-click interface, noted for its limitations in performing certain advanced statistical analyses, making it a potential source of misinterpretation or error.