What is the typical process for a scientific study from idea to publication?

A study typically begins with a hypothesis (often a null hypothesis). This leads to experimental design, determining subject numbers via power analysis, and obtaining Institutional Review Board (IRB) approval if human/animal subjects are involved. Primary and secondary outcomes are set, a statistical plan is developed, and the study is pre-registered. Finally, after execution, results are analyzed, a manuscript is written, and submitted for peer review to a journal.

What are the main types of scientific studies?

Scientific studies are broadly categorized into three types: observational studies (e.g., individual case reports, case series, cohort studies), experimental studies (e.g., non-randomized and randomized controlled trials), and summative analyses (e.g., systemic reviews and meta-analyses). Each type has different strengths and limitations regarding establishing causality and generalizability.

What are the phases of human clinical trials for a new drug?

Human clinical trials typically involve four phases. Phase 1 focuses on dose escalation and safety in a small group. Phase 2 continues safety evaluation and begins to look for efficacy, often in an open-label manner. Phase 3 is a large, rigorous, often placebo-controlled and blinded trial to confirm efficacy and safety before approval. Phase 4 occurs post-approval to gather more data on side effects, new indications, or specific populations.

What is 'healthy user bias' in observational studies and why is it important?

Healthy user bias occurs when people who make healthy choices (e.g., not eating meat, exercising) also tend to engage in other healthy behaviors that are difficult to control for in a study. This can falsely attribute benefits or harms to a specific behavior, making it challenging to establish true causality. For example, people who avoid bacon might also smoke less, exercise more, and manage stress better.

What is performance bias, also known as the Hawthorne effect?

Performance bias, or the Hawthorne effect, is when subjects in a study change their behavior simply because they know they are being observed. This can influence outcomes, for example, causing individuals to adhere more strictly to a diet when they know their food intake is being logged. A related concept is the 'Avis effect,' where a control group tries harder because they perceive themselves as the 'underdog.'

Why is it important to differentiate between primary and secondary outcomes in a study?

Studies are designed and statistically 'powered' to detect a difference in their primary outcome(s). If a study fails to meet its primary outcome, it's generally considered a null study, even if secondary (exploratory) outcomes show significant results. News reports often fail to make this distinction, misleading readers about the study's true conclusions.

How does 'statistical power' relate to study design and results?

Statistical power (typically 80-90%) is the probability of correctly finding a difference between groups if one truly exists. An underpowered study means there weren't enough subjects to detect a real difference, leading to a false negative result. An overpowered study might find statistically significant but clinically irrelevant effects.

What are hazard ratios and number needed to treat (NNT) and why are they useful?

Hazard ratios (HR) measure how likely an event is to occur in one group compared to another over time. An HR of 0.82 means an 18% reduced risk. NNT (Number Needed to Treat) is calculated as 1 divided by the absolute risk reduction, indicating how many people need to receive a treatment to prevent one adverse event. They provide a more practical understanding of effect size beyond relative risk.

What is a 'confidence interval' and how should it be interpreted?

A confidence interval (often 95% CI) provides a range in which the true population parameter is likely to lie. While commonly misunderstood as the probability that a specific interval contains the true effect, it more accurately means that if you repeated the study 100 times, 95 of those intervals would contain the true mean value. A wide interval suggests greater uncertainty.

What are the main reasons why a clinical trial might be stopped before its scheduled completion?

Clinical trials can be stopped prematurely for three primary reasons: safety (if one group experiences significantly more adverse events), benefit (if one treatment is so clearly superior that continuing the control arm would be unethical), or futility (if statisticians determine that no matter what happens, the study will not reach a significant outcome, making further continuation pointless).

What is 'publication bias' and how is it being addressed?

Publication bias refers to the tendency for studies with positive or 'interesting' findings to be published more readily than those with negative or null results. This distorts the scientific literature, hindering comprehensive knowledge. It is being combatted through pre-registration of studies (e.g., on clinicaltrials.gov) and publishing formats like 'registered reports,' where the study protocol is peer-reviewed and provisionally accepted for publication before data collection.

How is a journal's prestige or reputation determined?

A journal's reputation is often measured by its 'impact factor,' which is the ratio of total citations received by articles published in that journal to the total number of articles published by the journal over a previous period (typically a year). Higher impact factors generally indicate more influential and cited research.

What is an effective strategy for reading and understanding a scientific paper?

A recommended strategy involves starting with the abstract to gauge interest, then reading the introduction if unfamiliar with the topic, but often proceeding directly to the methods section for intervention details. Next, focus on figures and tables (which should be standalone with clear legends) before reading the prose of the results section. Finally, read the discussion, forming your own conclusions before comparing them to the authors'.

Key Moments

269 - Good vs. bad science: how to read and understand scientific studies

Peter Attia MD

Science & Technology3 min read123 min video

Sep 4, 2023|36,653 views|673|42

Peter Attia MD Dr. Peter Attia Early Medical The Drive Podcast The Drive Longevity Zone 2

Save to Pod

Key Moments

TL;DR

How to critically evaluate scientific studies: understanding study types, biases, and statistical significance.

Key Insights

Scientific studies progress from hypothesis to design, execution, and publication, with rigorous steps to minimize bias.

Studies are categorized into observational, experimental, and meta-analyses, each with unique strengths and limitations.

Key biases such as healthy user bias, recall bias, and performance bias can significantly impact study outcomes.

Randomized controlled trials (RCTs) are the gold standard, but careful examination of blinding, control groups, and sample size is crucial.

Interpreting study results requires understanding statistical concepts like p-values, confidence intervals, and effect sizes (e.g., absolute vs. relative risk, hazard ratios, NNT).

Studies can be stopped pre-emptively for safety concerns, demonstrated benefit, or futility.

Publication bias, where negative or inconclusive studies are less likely to be published, is a significant issue, partially mitigated by pre-registration.

Journal impact factors provide a metric for a journal's influence, and a structured approach to reading papers (abstract, methods, figures, results, discussion) is recommended.

THE SCIENTIFIC METHOD: FROM IDEA TO PUBLICATION

The journey of a scientific study begins with a hypothesis, ideally a "null hypothesis" stating no relationship between phenomena. This is followed by designing an experiment, determining sample size through power analysis, and obtaining ethical approval (e.g., IRB). Key outcomes (primary and secondary) are defined, a statistical plan is developed, and the study is pre-registered before execution. Funding is secured throughout this process. This rigorous framework aims to ensure scientific integrity and minimize bias from the outset.

CATEGORIES OF SCIENTIFIC STUDIES

Studies generally fall into three main categories: observational, experimental, and summations of existing studies (reviews, meta-analyses). Observational studies include individual case reports and case series, offering hypothesis-generating insights but lacking generalizability. Cohort studies, which can be retrospective or prospective, observe groups over time. Experimental studies, including randomized controlled trials (RCTs), actively intervene. Meta-analyses statistically combine data from multiple studies, offering a broad overview but are only as good as the studies they aggregate ('garbage in, garbage out').

UNDERSTANDING BIASES IN STUDY DESIGN

Several biases can skew study results. Healthy user bias occurs when individuals who adopt healthy behaviors also engage in other positive lifestyle choices, making it hard to isolate the effect of a single behavior. Recall bias is prevalent in nutritional epidemiology, where participants struggle to accurately remember past food consumption. Performance bias, also known as the Hawthorne effect, arises when participants alter their behavior because they know they are being observed or are in a competitive situation, especially in lifestyle intervention studies where differential attention can influence outcomes.

EVALUATING EXPERIMENTAL STUDIES AND DATA INTERPRETATION

Robust experimental studies, particularly RCTs, are crucial. Randomization, blinding (single or double), control groups, and adequate sample size (power) are key. When interpreting results, it's vital to distinguish between statistical significance and clinical relevance. Effect sizes should be considered in absolute terms (e.g., absolute risk reduction) rather than solely relative terms. Metrics like hazard ratios capture temporal risk, while the Number Needed to Treat (NNT) provides a practical measure of intervention impact. P-values indicate the probability of rejecting a true null hypothesis, and confidence intervals quantify the uncertainty around an estimate.

THE PUBLICATION PROCESS AND COMBATING BIAS

Once a study is completed, it undergoes peer review before journal publication. Reviewers assess originality, methodology, and statistical rigor. However, publication bias, where studies with positive or statistically significant findings are more likely to be published than negative or inconclusive ones, is a significant challenge. This can distort the scientific literature. Measures like pre-registration of studies (e.g., on clinicaltrials.gov) and novel publishing formats like 'registered reports' aim to mitigate this by making study protocols transparent and encouraging the publication of all results, regardless of outcome.

JOURNAL REPUTATION AND READING STRATEGIES

Journal impact factors, a measure of citation frequency, often indicate a journal's perceived prestige and influence, guiding authors on where to submit their work. When reading a scientific paper, a strategic approach is recommended: start with the abstract to gauge interest, then delve into the methods and figures/tables for critical details. The results section provides the findings, and the discussion section offers interpretation and limitations. By understanding the types of studies, potential biases, statistical measures, and the publication process, individuals can become more discerning consumers of scientific information.

Mentioned in This Episode

●Products

●Software & Apps

●Tools

●Companies

●Organizations

●Books

●Studies Cited

●Concepts

●People Referenced

Clinical Trial Phases and Their Goals

Data extracted from this episode

Phase	Primary Goal	Typical Sample Size	Key Characteristics
Phase 1	Determine toxicity and safe dose range (dose escalation)	Less than 100 people	Small, done in cohorts, often in patients with advanced disease, not focused on efficacy
Phase 2	Continue safety evaluation, start to look for efficacy	20-300 people	Open label (often no randomization), compare to natural history, can have bias
Phase 3	Measure efficacy and effectiveness rigorously, further safety monitoring	Thousands of patients	Randomized, placebo-controlled (or standard of care vs. new agent), blinded, longer studies, basis for drug approval
Phase 4 (Post-Marketing)	Gather additional information post-approval (e.g., new indications, rare side effects)	Large, broad population	Expands indication, looks for missed side effects in wider populations and different risk profiles

Study Stopping Rules

Data extracted from this episode

Reason for Stopping	Description	Example Study
Safety	A statistically significant difference in an important safety metric between groups, indicating harm.	CTEP Inhibitor (Torcetrapib) trial
Benefit	Such a clear and substantial benefit in one group that it would be unethical to continue the study for the control group.	PREDIMED trial (initially)
Futility	No benefit observed, and statistically, no future events in the study can change the outcome, making continuation purposeless.	Look AHEAD trial

Top Scientific Journals by Impact Factor (2019)

Data extracted from this episode

Journal Name	Citations (approx.)	Impact Factor
New England Journal of Medicine	347,000	74.7
CA: A Cancer Journal for Clinicians	40,000	292.0 (outlier)
The Lancet	250,000	60.0
JAMA (Journal of the American Medical Association)	200,000	56.3
WHO Technical Report Series	3,500	59.0 (outlier)

Common Questions

To better understand studies, recognize that scientific reporting in the news can be biased. Pay attention to a study's design (e.g., randomized controlled trial vs. observational), watch for biases like 'healthy user bias' or 'recall bias,' understand the difference between primary and secondary outcomes, and critically assess effect sizes and funding sources. Also, be aware of how statistics like p-values and confidence intervals are presented.