Why did John Ioannidis pursue medicine in addition to mathematics?

Ioannidis was attracted to medicine because of its potential to save lives and improve human quality of life, which he saw as a worthwhile pursuit. He aimed to combine the rigor of mathematics and scientific method to get reliable evidence in medicine.

What was the main conclusion of Ioannidis's 2005 paper, 'Why Most Published Research Findings Are False'?

The paper, using a mathematical model, concluded that under most realistic circumstances in biomedical research, if a study reports a statistically significant result (p-value < 0.05), the chances of it being a false positive are higher than 50%. This implies that a vast majority of published research findings are unreliable.

What is the difference between statistical significance and clinical significance?

Statistical significance (often based on a p-value) indicates if a result is unlikely due to chance, but doesn't necessarily mean it's important. Clinical significance, conversely, refers to whether a finding is practically meaningful and makes a real difference to patients or public health, in terms of benefits, costs, and harms.

How do underpowered and overpowered studies contribute to unreliable results?

Underpowered studies (too small) are likely to produce false positives or greatly exaggerate the magnitude of real effects. Overpowered studies (with 'big data') can find statistically significant results that have no clinical meaning and are highly susceptible to even tiny biases, effectively measuring bias itself.

Why has genetics adapted better research practices than nutritional epidemiology?

Genetics benefits from more accurate measurement tools (like whole genome sequencing) and lacks strong prior beliefs or expert opinions that can bias findings. It has adopted strategies like very large studies, stringent p-value thresholds (e.g., 10^-8), and large-scale data sharing and standardized analysis across consortia, which nutritional epidemiology has been slower to implement.

What was the intent behind Ioannidis's 'Systematic Cookbook Review' paper?

The 2012 paper was a sarcastic commentary on the flawed nature of nutritional epidemiology. It showed that by applying common methodologies to randomly selected cookbook ingredients, almost all could be spuriously linked to cancer risk, highlighting the prevalence of false positives and methodological issues in the field.

What lessons did Dr. Ioannidis draw from the PREDIMED trial controversy?

The PREDIMED trial, which initially showed strong benefits for the Mediterranean diet, later faced scrutiny and retraction due to subverted randomization and recruitment issues. Ioannidis notes that even with good intentions, such fundamental flaws undermine the credibility of a study, emphasizing the need for rigorous adherence to design principles.

How did early COVID-19 seroprevalence studies challenge initial assumptions about the virus?

Early seroprevalence studies in locations like Santa Clara and LA County (led by Ben David and Bhattacharya) revealed that the virus was far more widely spread than indicated by documented PCR cases, suggesting a significantly lower infection fatality rate than initially feared by some models (e.g., 5-10%).

What challenges do scientists face when communicating findings in a polarized environment?

Scientists like Ioannidis and Anthony Fauci faced intense animosity, criticism, and personal attacks due to the highly polarized political environment during scientific discourse, especially concerning COVID-19. This environment makes it difficult for scientists to disseminate objective findings without fear of misinterpretation or personal harm, potentially silencing younger researchers.

Key Moments

#143 - John Ioannidis, M.D., D.Sc.: Why most biomedical research is flawed, and how to improve it

Q: What is meta-research, and what does Dr. John Ioannidis study?

Meta-research is the study of scientific research itself, focusing on improving its quality and credibility. Dr. John Ioannidis is one of the world's foremost experts in this field, primarily in clinical medicine and social sciences, as co-director of the Meta-Research Innovation Center at Stanford.

Q: What is the difference between statistical significance and clinical significance?

Statistical significance (often based on a p-value) indicates if a result is unlikely due to chance, but doesn't necessarily mean it's important. Clinical significance, conversely, refers to whether a finding is practically meaningful and makes a real difference to patients or public health, in terms of benefits, costs, and harms.

Q: How do underpowered and overpowered studies contribute to unreliable results?

Underpowered studies (too small) are likely to produce false positives or greatly exaggerate the magnitude of real effects. Overpowered studies (with 'big data') can find statistically significant results that have no clinical meaning and are highly susceptible to even tiny biases, effectively measuring bias itself.

Q: Why has genetics adapted better research practices than nutritional epidemiology?

Genetics benefits from more accurate measurement tools (like whole genome sequencing) and lacks strong prior beliefs or expert opinions that can bias findings. It has adopted strategies like very large studies, stringent p-value thresholds (e.g., 10^-8), and large-scale data sharing and standardized analysis across consortia, which nutritional epidemiology has been slower to implement.

Q: What was the intent behind Ioannidis's 'Systematic Cookbook Review' paper?

The 2012 paper was a sarcastic commentary on the flawed nature of nutritional epidemiology. It showed that by applying common methodologies to randomly selected cookbook ingredients, almost all could be spuriously linked to cancer risk, highlighting the prevalence of false positives and methodological issues in the field.

Q: What lessons did Dr. Ioannidis draw from the PREDIMED trial controversy?

The PREDIMED trial, which initially showed strong benefits for the Mediterranean diet, later faced scrutiny and retraction due to subverted randomization and recruitment issues. Ioannidis notes that even with good intentions, such fundamental flaws undermine the credibility of a study, emphasizing the need for rigorous adherence to design principles.

Q: How did early COVID-19 seroprevalence studies challenge initial assumptions about the virus?

Early seroprevalence studies in locations like Santa Clara and LA County (led by Ben David and Bhattacharya) revealed that the virus was far more widely spread than indicated by documented PCR cases, suggesting a significantly lower infection fatality rate than initially feared by some models (e.g., 5-10%).

Q: What challenges do scientists face when communicating findings in a polarized environment?

Scientists like Ioannidis and Anthony Fauci faced intense animosity, criticism, and personal attacks due to the highly polarized political environment during scientific discourse, especially concerning COVID-19. This environment makes it difficult for scientists to disseminate objective findings without fear of misinterpretation or personal harm, potentially silencing younger researchers.

Peter Attia MD

People & Blogs4 min read113 min video

Jan 4, 2021|10,398 views|289|22

Save to Pod

Key Moments

TL;DR

Most biomedical research is flawed due to biases. John Ioannidis explains why and offers solutions for improvement.

Key Insights

A significant portion of published biomedical research is flawed, with false positive rates often exceeding 50%.

Low statistical power, publication bias, selective reporting, and various other biases contribute to unreliable research findings.

Statistical significance (p-values) should not be confused with clinical significance, which focuses on real-world patient impact.

Nutritional epidemiology, compared to genetics, has struggled with methodological rigor, leading to a proliferation of questionable findings.

Reforming research requires better training, transparency, pre-registration of studies, and a shift towards collaborative, large-scale research.

Philanthropy and public funding play crucial roles in supporting high-risk, high-reward research, but also need careful alignment of priorities.

THE PROBLEM OF UNRELIABLE RESEARCH

John Ioannidis, a professor at Stanford and expert in meta-research, highlights a staggering finding: the majority of published biomedical research is incorrect. This issue is compounded by factors such as low statistical power, publication bias, selective reporting, and numerous other biases that inflate false positive rates, often exceeding 50%. These systemic flaws undermine the credibility of scientific evidence, which is crucial for medical decision-making and public health.

UNDERSTANDING STATISTICAL VS. CLINICAL SIGNIFICANCE

The discussion emphasizes the critical distinction between statistical significance and clinical significance. While statistical significance (often determined by p-values) indicates whether a result is likely due to chance, clinical significance addresses whether an observed effect is large enough to matter in a real-world patient or population context. Confusion between these can lead to the adoption of interventions with negligible real-world benefits, despite appearing statistically robust.

CHALLENGES IN NUTRITIONAL EPIDEMIOLOGY

Nutritional epidemiology is presented as a prime example of a field struggling with research integrity. Unlike genetics, which has adopted more rigorous methods like genome-wide association studies and large consortia, nutrition research often relies on imprecise measurements, lacks strong priors, and suffers from deep-seated beliefs. This has led to a cascade of unreliable findings, often influenced by expert opinion and cultural biases rather than robust evidence.

REFORMING RESEARCH PRACTICES

Ioannidis advocates for substantial reforms, including increased methodological training, pre-registration of study protocols, and greater transparency. He champions collaborative research models, such as large-scale meta-analyses of primary data, and suggests expanding exposure-wide association tests to analyze multiple factors simultaneously. These approaches aim to counteract biases and enhance the reproducibility and reliability of scientific findings across various fields.

THE ROLE OF DIFFERENT FUNDING SOURCES

The conversation touches on the complex interplay of research funding. While industry funding is often tied to product development and can create conflicts of interest, public funding is essential for unbiased testing of these products. Philanthropy is highlighted as a vital source for supporting high-risk, exploratory research that might not fit traditional funding models, promoting scientific innovation and discovery beyond immediate commercial or political pressures.

NAVIGATING SCIENCE COMMUNICATION AND POLITICS

The polarized political climate has significantly impacted the dissemination and acceptance of scientific findings, as seen during the COVID-19 pandemic. Ioannidis stresses the importance of dissociating science from politics, advocating for honest, clear communication that acknowledges uncertainty rather than succumbing to political agendas. Protecting scientists from vitriol and smearing is crucial for fostering an environment where objective research can thrive and contribute to societal well-being.

THE PERILS OF BIG DATA AND OVERPOWERED STUDIES

While small, underpowered studies are a common issue, the rise of 'big data' presents a new challenge: overpowered studies. With massive datasets, even minuscule effects can reach statistical significance, potentially leading to findings that lack clinical meaning. This can also exacerbate problems with bias, making it difficult to distinguish genuine signals from measurement artifacts or ingrained biases within the data.

THE CASE OF PREDI-MED AND RESEARCH INTEGRITY

The Predimed study, initially lauded as a landmark trial for the Mediterranean diet, serves as a cautionary tale. Subversion of randomization and subsequent re-analysis revealed significant issues with its foundational design. This case underscores how even well-intentioned research can falter due to methodological flaws, highlighting the need for extreme caution in interpreting results and the importance of maintaining the integrity of research protocols.

EMBRACING UNCERTAINTY AND CONTINUOUS LEARNING

Ioannidis expresses optimism for the future of science, driven by the realization that there is always more to learn and that previous assumptions may need correction. He views the pursuit of knowledge as an ongoing process of discovery and self-correction. This continuous learning, marked by humility and a willingness to update beliefs in the face of new evidence, is fundamental to scientific progress.

Mentioned in This Episode

●Organizations

●Books

●Studies Cited

●Concepts

●People Referenced

Common Questions

Meta-research is the study of scientific research itself, focusing on improving its quality and credibility. Dr. John Ioannidis is one of the world's foremost experts in this field, primarily in clinical medicine and social sciences, as co-director of the Meta-Research Innovation Center at Stanford.

Topics

Mental Health & Psychology Science & Mathematics Nutritional Science Science Communication Clinical Trials Medicine & Clinical Statistical Methods Research Credibility Scientific Bias COVID-19 Research

Mentioned in this video

People

David Sackett

Part of the McMaster team who coined the term 'evidence-based medicine'.

Gordon Wyatt

Part of the McMaster team who coined the term 'evidence-based medicine'.

Pat Brown

A Stanford professor credited as one of the forces behind the journal PLOS.

Anthony Fauci

John Ioannidis's former supervisor at NIAID/NIH, described as a brilliant scientist who was ferociously attacked during the pandemic, highlighting the dangers scientists face.

Dimitri Drakopoulos

Professor of epidemiology and chair of epidemiology at Harvard, he was a great teacher for John Ioannidis in medical school.

Tom Chalmers

A pioneer in evidence-based medicine and one of the first in the U.S. to design a randomized trial and perform meta-analysis, he was a revelation for Ioannidis.

Brian Wansink

A prominent professor at Cornell University who was found to have urged students to 'cut corners and torture the data' to get appealing results, leading to fraudulent research practices.

David Allison

Contacted by Peter Attia to assess seropositivity in New York City, who then suggested reaching out to Ioannidis.

Jay Bhattacharya

A lead investigator for the seroprevalence studies in Santa Clara and LA County that John Ioannidis was part of.

Peter Attia

Host of The Drive podcast and interviewee of Dr. Ioannidis.

Bob Malloring

Physician-in-chief and professor at Harvard, he was a great physician scientist who shaped Ioannidis's clinical acumen and approach to patients.

Joseph Lau

Worked with Tom Chalmers at Tufts Medical Center, advancing the frontiers of evidence-based medicine which greatly influenced Ioannidis.

Richard Feynman

Nobel laureate physicist, quoted by Ioannidis for his emphasis on scientific honesty, stating that the most important rule in science is not to fool yourself, as you are the easiest person to fool.

Aaron Ben David

A lead investigator for the seroprevalence studies in Santa Clara and LA County that John Ioannidis was part of.

Organizations

NIH

The institution where Anthony Fauci was John Ioannidis's supervisor.

Meta-Research Innovation Center at Stanford

John Ioannidis is the co-director of this center, which studies scientific research itself, primarily in clinical medicine and social sciences.

PLOS Medicine

An open-source journal where John Ioannidis published his influential 2005 paper, 'Why Most Published Research Findings Are False.'

Cornell University

The institution where Brian Wansink, a professor accused of research fraud, conducted his studies.

University of Southern California

Colleagues from USC collaborated on the seroprevalence study in LA County.

Stanford University

Dr. John Ioannidis is a professor at Stanford University and co-director of its Meta-Research Innovation Center.

Tufts Medical Center

John Ioannidis completed his fellowship training in infectious diseases at Tufts Medical Center, where he also met Tom Chalmers and Joseph Lau.

NIAID

The institution where Anthony Fauci was John Ioannidis's supervisor.

Concepts

P-value

A statistical concept indicating the probability of obtaining results as extreme as, or more extreme than, the observed results, assuming the null hypothesis is true. Ioannidis warns against misinterpreting it as clinical significance.

Statistical significance

A concept tied to p-values, often set at a threshold like 0.05, that determines if a result is unlikely to have occurred by chance. Ioannidis argues it's often confused with clinical significance and can be misleading, especially with large datasets or bias.

Evidence-Based Medicine

A term advanced by the McMaster team (David Sackett and Gordon Wyatt), which Ioannidis initially believed would provide reliable evidence but soon realized its limitations.

Feynman's Principle

The idea that the most important rule in science is not to fool yourself, as you are the easiest person to fool. This highlights the importance of honesty and rigor in scientific inquiry.

Clinical Significance

The practical importance of a treatment effect or measured outcome, distinguishing it from merely statistically significant findings. Ioannidis emphasizes that this is what truly matters in medicine.

Mediterranean diet

A dietary pattern that John Ioannidis personally prefers and initially believed was shown to have significant benefits in the PREDIMED trial, but whose definitive evidence was complicated by the trial's issues.

Greece

John Ioannidis was born in the U.S. but grew up in Athens, Greece, where he also completed his medical school and postgraduate training.

Studies & Research

Mendelian randomization

A method using genetic instruments to create study designs that are more equivalent to randomized trials, offering more credible estimates than pure observational evidence.

Harvard University

John Ioannidis completed his residency training at Harvard, and his epidemiology professor, Dimitri Drakopoulos, was also chair of epidemiology there. Harvard also has a strong team of nutritional epidemiologists.

The Lancet

A medical journal that published a national survey on hemodialysis patients, whose infection rate translated into an infection fatality rate identical to what Ioannidis's team observed for COVID-19 in early April.

New England Journal of Medicine

Tom Chalmers and Joseph Lau published an influential paper on cumulative meta-analysis in this journal.

PREDIMED Trial

A clinical trial on the Mediterranean diet, initially lauded for showing significant benefits on clinical outcomes, but later found to have randomization issues and was retracted and republished, leading to questions about its credibility.

Books

Why Most Published Research Findings Are False

A seminal theoretical paper by Ioannidis published in PLOS Medicine, arguing through a mathematical model that most published biomedical research findings are likely incorrect.

The Systematic Cookbook Review

A paper by Ioannidis that sarcastically demonstrated how most common cookbook ingredients show spurious associations with cancer risk in observational nutritional epidemiology.

The Boston Cookbook

A cookbook that has been published since the 19th century, used by Ioannidis and his team to randomly select 50 ingredients for their 'Systematic Cookbook Review' paper.

Locations

New York City

John Ioannidis was born in New York City.

Los Angeles County

One of the locations where seroprevalence studies were conducted by Ioannidis's team, revealing a much higher spread of the virus than initially thought.

Santa Clara County

One of the locations where seroprevalence studies were conducted by Ioannidis's team, revealing a much higher spread of the virus than initially thought.