Heretic
Method · Drugs & hormones · Supplements · Diet & nutrition

Trials, observational studies and n-of-1

Also RCT vs observational, evidence hierarchy, n-of-1 trial, single-patient trial, self-experimentation, healthy user bias.

The evidence hierarchy is a rule of thumb. Here is what the record says about when it holds.

Updated 1 Nov 202612 sources6 min read
Short answer

On average, well-run observational studies give answers close to randomised trials (Cochrane ratio of ratios 1.08). When they diverge they can diverge badly: hormone therapy looked protective in cohorts and harmful in the WHI trial, and beta-carotene raised lung cancer in smokers. A trial average can also hide individual responses. Blinded n-of-1 trials test one person repeatedly; in StatinWISE they showed most reported statin muscle pain was not caused by the drug.

Summary drafted from this entry’s sources; editorial review pending.

01Key questions

Do observational studies exaggerate treatment effects?

Not on average. A 2024 Cochrane review of 34 methodological reviews found no difference or a very small difference between observational and randomised estimates, with low certainty. Database studies of mortality were an exception, running 31% more favourable than later trials.38

Why did hormone therapy look protective in cohorts but harmful in the WHI trial?

When the Nurses' Health Study was reanalysed like a trial, comparing starters with non-starters, its estimates were similar to the WHI's. The gap was largely explained by time since menopause and length of follow-up.456

What is an n-of-1 trial?

A double-blind randomised trial in one person, alternating active treatment and placebo in random order with a pre-chosen outcome, described by Guyatt and Sackett in 1986.11

What did n-of-1 trials show about statins and muscle pain?

In StatinWISE, 200 patients who blamed statins for muscle symptoms had no difference in symptom scores between blinded statin and placebo periods, and two thirds of completers restarted statins.12

Are randomised trials always needed?

No. When an effect is large and rapid compared with a condition's usual course, often with rate ratios above 10, bias cannot plausibly explain it.10

The usual rule says randomised trials count and observational studies don't. The record is more interesting. On average, well-run observational studies land close to trials on the same question. When they miss, they can miss badly, usually because the people who chose a treatment differed from those who didn't. And a trial's average can hide what happens to you. The tools that fill that gap, n-of-1 trials, already exist and have overturned a common belief about statins. This site uses all three kinds of evidence and labels each one.

02Observational studies usually agree with trials

  • The 2000 comparisons. Across 136 reports on 19 treatments, observational and randomised estimates were usually similar; in only 2 of 19 did the pooled observational estimate fall outside the trials' 95% confidence interval (Benson and Hartz 2000, PMID 10861324). A companion study of 99 reports on five topics found well-designed cohort and case-control studies did not systematically overstate effects. For BCG vaccine against tuberculosis, 13 trials gave a relative risk of 0.49 and 10 case-control studies gave an odds ratio of 0.50 (Concato 2000, PMID 10861325).
  • The Cochrane answer. A 2024 Cochrane review pooled 34 methodological reviews covering 2,869 trials and 3,924 observational studies. It found no difference or a very small difference between designs (ratio of ratios 1.08, 95% CI 1.01 to 1.15), and 23 of 34 reviews reported agreement on average. Differences were somewhat larger for drug interventions and when observational studies did not use propensity scores. Certainty was rated low (Toews 2024, PMID 38174786).

03When they disagree, they can disagree badly

  • Hormone therapy. In the Nurses' Health Study, women taking estrogen had a relative risk of major coronary disease of 0.56 (Stampfer 1991, PMID 1870648). The Women's Health Initiative trial of estrogen plus progestin was stopped early, with a coronary hazard ratio of 1.29 and more breast cancer, stroke and blood clots (Rossouw 2002, PMID 12117397). The twist: when the cohort data were reanalysed the way a trial would be, comparing women who started therapy with women who didn't, the estimates were similar to the trial's, and most of the gap was explained by time since menopause and length of follow-up (Hernán 2008, PMID 18854702). Both studies were partly right about different women. See hormone optimisation.
  • Beta-carotene. Observational links between diet and lower cancer risk led to a trial in 29,000 Finnish male smokers. Beta-carotene supplements raised lung cancer incidence by 18% (ATBC 1994, PMID 8127329).
  • Database studies. Propensity-score studies of routinely collected health data, published before any trial, gave mortality estimates 31% more favourable than the trials that followed (summary relative odds ratio 1.31), and the direction of effect differed in 5 of 16 questions (Hemkens 2016, PMID 26858277).
  • Highly cited claims. Of 6 highly cited non-randomised studies, 5 were later contradicted or found to have overstated their effects, against 9 of 39 highly cited trials (Ioannidis 2005, PMID 16014596). Trials are not immune: small ones were the most likely to be overturned.

04Some effects need no trial

When an effect is large and fast compared with the usual course of a condition, bias cannot plausibly explain it. Glasziou and colleagues listed historical treatments accepted without trials, with rate ratios often above 10 (Glasziou 2007, PMID 17303884). The reverse also holds: the smaller the claimed effect, the more a randomised comparison matters, because modest benefits are exactly what confounding produces.

05The n-of-1 trial

  • The idea. In 1986 Guyatt, Sackett and colleagues described double-blind randomised trials in a single patient: paired periods of active treatment and placebo in random order, continued until benefit is shown or ruled out, with the patient's own symptoms or test results as the outcome (Guyatt 1986, PMID 2936958). They built it because large trials had not been run, or could not be run, for many conditions, and because trial averages are hard to apply to one person.
  • The statin result. StatinWISE ran a series of n-of-1 trials in 200 UK patients who had stopped or were thinking of stopping statins because of muscle pain. Each had six blinded two-month periods of statin or placebo. Muscle symptom scores did not differ between statin and placebo periods (mean difference minus 0.11, 95% CI minus 0.36 to 0.14), and two thirds of those who finished restarted statins (Herrett 2021, PMID 33627334). Patients' experience was real; the cause, for most, was not the drug.
  • What it needs. A placebo or blinded comparison, a random order, repeated switches, enough time for the effect to come and go, and an outcome chosen in advance. Without those, a self-experiment mostly measures expectation and the natural ups and downs of symptoms.
  • Where it doesn't work. It suits stable, chronic conditions and treatments that act and wear off quickly. It cannot answer questions about rare events or long-term outcomes such as heart attacks, cancer or lifespan; those need large groups.

06How this site uses each kind of evidence

  • A trial answers what happens on average to people like those enrolled. Check who they were.
  • A cohort answers what happens to people who chose something. Ask why they chose it, and whether the analysis compared starters with non-starters.
  • Agreement across designs with different biases is stronger than any single design.
  • User reports and case series are labelled as such. They point to questions; they do not settle them.
  • Who paid matters for all of them; see industry funding and trial results and off-patent compounds and missing trials.

This page explains how to read evidence; it is not medical advice.

07Who paid, and how it got here

Who funded the key studies on this page, the favourable and the unfavourable alike, and how its legal status came about. From each paper’s funding and conflict-of-interest statements, trial registries and official documents; checked 1 Nov 2026. Why we show this

Who paid for the key studies

  • StatinWISE, series of n-of-1 trials, n=200Statins did not worsen muscle symptoms versus placebo. Conflict-of-interest statement
  • Hemkens 2016, 16 database studies vs 36 later trialsDatabase studies ran 31% more favourable on mortality. Conflict-of-interest statement

08Connections

Neighborhood · 6 linked entriesOpen full map →
Biohacking and optimizationGuide · 26 links→ Seed oils and inflammationDiet · 15 links→ Hormone optimization (TRT, HRT)Therapy · 14 links→ Carnivore dietDiet · 9 links→ Industry funding and trial resultsMethod · 9 links→ Off-patent compounds and missing trialsMethod · 9 links→

09Sources

  1. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342(25):1878-86. PMID 10861324PubMed 10861324
  2. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342(25):1887-92. PMID 10861325PubMed 10861325
  3. Toews I, Anglemyer A, Nyirenda JL et al. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst Rev. 2024;1:MR000034. PMID 38174786PubMed 38174786
  4. Stampfer MJ et al. Postmenopausal estrogen therapy and cardiovascular disease: ten-year follow-up from the Nurses' Health Study. N Engl J Med. 1991. PMID 1870648PubMed 1870648
  5. Rossouw JE et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal women. JAMA. 2002. PMID 12117397PubMed 12117397
  6. Hernán MA et al. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology. 2008. PMID 18854702PubMed 18854702
  7. Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group. The effect of vitamin E and beta carotene on the incidence of lung cancer and other cancers in male smokers. N Engl J Med. 1994. PMID 8127329PubMed 8127329
  8. Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials. BMJ. 2016;352:i493. PMID 26858277PubMed 26858277
  9. Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005. PMID 16014596PubMed 16014596
  10. Glasziou P, Chalmers I, Rawlins M, McCulloch P. When are randomised trials unnecessary? Picking signal from noise. BMJ. 2007;334:349-51. PMID 17303884PubMed 17303884
  11. Guyatt G, Sackett D, Taylor DW et al. Determining optimal therapy: randomized trials in individual patients. N Engl J Med. 1986;314(14):889-92. PMID 2936958PubMed 2936958
  12. Herrett E, Williamson E, Brack K et al. Statin treatment and muscle symptoms: series of randomised, placebo controlled n-of-1 trials. BMJ. 2021;372:n135. PMID 33627334PubMed 33627334

10Entry history

Latest content update.

Heretic informs and never prescribes. Where amounts appear, they describe what published studies used. Talk to a qualified clinician before acting on anything here.