# Trials, observational studies and n-of-1

Rating: Proven — Replicated results in human randomized trials
Type: Method
Updated: 2026-11-01
Canonical: https://heretic.bio/articles/trials-vs-observational-data-and-n-of-1.html

> On average, well-run observational studies give answers close to randomised trials (Cochrane ratio of ratios 1.08). When they diverge they can diverge badly: hormone therapy looked protective in cohorts and harmful in the WHI trial, and beta-carotene raised lung cancer in smokers. A trial average can also hide individual responses. Blinded n-of-1 trials test one person repeatedly; in StatinWISE they showed most reported statin muscle pain was not caused by the drug.

> The usual rule says randomised trials count and observational studies don't. The record is more interesting. On average, well-run observational studies land close to trials on the same question. When they miss, they can miss badly, usually because the people who chose a treatment differed from those who didn't. And a trial's average can hide what happens to you. The tools that fill that gap, n-of-1 trials, already exist and have overturned a common belief about statins. This site uses all three kinds of evidence and labels each one.

## Observational studies usually agree with trials

  * **The 2000 comparisons.** Across 136 reports on 19 treatments, observational and randomised estimates were usually similar; in only 2 of 19 did the pooled observational estimate fall outside the trials' 95% confidence interval (Benson and Hartz 2000, [PMID 10861324](https://pubmed.ncbi.nlm.nih.gov/10861324/)). A companion study of 99 reports on five topics found well-designed cohort and case-control studies did not systematically overstate effects. For BCG vaccine against tuberculosis, 13 trials gave a relative risk of 0.49 and 10 case-control studies gave an odds ratio of 0.50 (Concato 2000, [PMID 10861325](https://pubmed.ncbi.nlm.nih.gov/10861325/)).
  * **The Cochrane answer.** A 2024 Cochrane review pooled 34 methodological reviews covering 2,869 trials and 3,924 observational studies. It found no difference or a very small difference between designs (ratio of ratios 1.08, 95% CI 1.01 to 1.15), and 23 of 34 reviews reported agreement on average. Differences were somewhat larger for drug interventions and when observational studies did not use propensity scores. Certainty was rated low (Toews 2024, [PMID 38174786](https://pubmed.ncbi.nlm.nih.gov/38174786/)).



## When they disagree, they can disagree badly

  * **Hormone therapy.** In the Nurses' Health Study, women taking estrogen had a relative risk of major coronary disease of 0.56 (Stampfer 1991, [PMID 1870648](https://pubmed.ncbi.nlm.nih.gov/1870648/)). The Women's Health Initiative trial of estrogen plus progestin was stopped early, with a coronary hazard ratio of 1.29 and more breast cancer, stroke and blood clots (Rossouw 2002, [PMID 12117397](https://pubmed.ncbi.nlm.nih.gov/12117397/)). The twist: when the cohort data were reanalysed the way a trial would be, comparing women who started therapy with women who didn't, the estimates were similar to the trial's, and most of the gap was explained by time since menopause and length of follow-up (Hernán 2008, [PMID 18854702](https://pubmed.ncbi.nlm.nih.gov/18854702/)). Both studies were partly right about different women. See [hormone optimisation](hormone-optimization-trt-hrt-perimenopause.html).
  * **Beta-carotene.** Observational links between diet and lower cancer risk led to a trial in 29,000 Finnish male smokers. Beta-carotene supplements raised lung cancer incidence by 18% (ATBC 1994, [PMID 8127329](https://pubmed.ncbi.nlm.nih.gov/8127329/)).
  * **Database studies.** Propensity-score studies of routinely collected health data, published before any trial, gave mortality estimates 31% more favourable than the trials that followed (summary relative odds ratio 1.31), and the direction of effect differed in 5 of 16 questions (Hemkens 2016, [PMID 26858277](https://pubmed.ncbi.nlm.nih.gov/26858277/)).
  * **Highly cited claims.** Of 6 highly cited non-randomised studies, 5 were later contradicted or found to have overstated their effects, against 9 of 39 highly cited trials (Ioannidis 2005, [PMID 16014596](https://pubmed.ncbi.nlm.nih.gov/16014596/)). Trials are not immune: small ones were the most likely to be overturned.



## Some effects need no trial

When an effect is large and fast compared with the usual course of a condition, bias cannot plausibly explain it. Glasziou and colleagues listed historical treatments accepted without trials, with rate ratios often above 10 (Glasziou 2007, [PMID 17303884](https://pubmed.ncbi.nlm.nih.gov/17303884/)). The reverse also holds: the smaller the claimed effect, the more a randomised comparison matters, because modest benefits are exactly what confounding produces.

## The n-of-1 trial

  * **The idea.** In 1986 Guyatt, Sackett and colleagues described double-blind randomised trials in a single patient: paired periods of active treatment and placebo in random order, continued until benefit is shown or ruled out, with the patient's own symptoms or test results as the outcome (Guyatt 1986, [PMID 2936958](https://pubmed.ncbi.nlm.nih.gov/2936958/)). They built it because large trials had not been run, or could not be run, for many conditions, and because trial averages are hard to apply to one person.
  * **The statin result.** StatinWISE ran a series of n-of-1 trials in 200 UK patients who had stopped or were thinking of stopping statins because of muscle pain. Each had six blinded two-month periods of statin or placebo. Muscle symptom scores did not differ between statin and placebo periods (mean difference minus 0.11, 95% CI minus 0.36 to 0.14), and two thirds of those who finished restarted statins (Herrett 2021, [PMID 33627334](https://pubmed.ncbi.nlm.nih.gov/33627334/)). Patients' experience was real; the cause, for most, was not the drug.
  * **What it needs.** A placebo or blinded comparison, a random order, repeated switches, enough time for the effect to come and go, and an outcome chosen in advance. Without those, a self-experiment mostly measures expectation and the natural ups and downs of symptoms.
  * **Where it doesn't work.** It suits stable, chronic conditions and treatments that act and wear off quickly. It cannot answer questions about rare events or long-term outcomes such as heart attacks, cancer or lifespan; those need large groups.



## How this site uses each kind of evidence

  * A trial answers what happens on average to people like those enrolled. Check who they were.
  * A cohort answers what happens to people who chose something. Ask why they chose it, and whether the analysis compared starters with non-starters.
  * Agreement across designs with different biases is stronger than any single design.
  * User reports and case series are labelled as such. They point to questions; they do not settle them.
  * Who paid matters for all of them; see [industry funding and trial results](industry-funding-and-trial-results.html) and [off-patent compounds and missing trials](off-patent-compounds-and-missing-trials.html).



This page explains how to read evidence; it is not medical advice.

## Who paid, and how it got here

### Who paid for the key studies

- StatinWISE, series of n-of-1 trials, n=200: Statins did not worsen muscle symptoms versus placebo. Funding and interests: NIHR and LSHTM (public funding). ([Conflict-of-interest statement](https://pubmed.ncbi.nlm.nih.gov/33627334/))
- Hemkens 2016, 16 database studies vs 36 later trials: Database studies ran 31% more favourable on mortality. Funding and interests: Commonwealth Fund support for one author (charitable funding). ([Conflict-of-interest statement](https://pubmed.ncbi.nlm.nih.gov/26858277/))

Checked 2026-11-01.

## Sources

1. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342(25):1878-86. PMID 10861324 (PubMed 10861324)
2. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342(25):1887-92. PMID 10861325 (PubMed 10861325)
3. Toews I, Anglemyer A, Nyirenda JL et al. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst Rev. 2024;1:MR000034. PMID 38174786 (PubMed 38174786)
4. Stampfer MJ et al. Postmenopausal estrogen therapy and cardiovascular disease: ten-year follow-up from the Nurses' Health Study. N Engl J Med. 1991. PMID 1870648 (PubMed 1870648)
5. Rossouw JE et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal women. JAMA. 2002. PMID 12117397 (PubMed 12117397)
6. Hernán MA et al. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology. 2008. PMID 18854702 (PubMed 18854702)
7. Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group. The effect of vitamin E and beta carotene on the incidence of lung cancer and other cancers in male smokers. N Engl J Med. 1994. PMID 8127329 (PubMed 8127329)
8. Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JP. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials. BMJ. 2016;352:i493. PMID 26858277 (PubMed 26858277)
9. Ioannidis JP. Contradicted and initially stronger effects in highly cited clinical research. JAMA. 2005. PMID 16014596 (PubMed 16014596)
10. Glasziou P, Chalmers I, Rawlins M, McCulloch P. When are randomised trials unnecessary? Picking signal from noise. BMJ. 2007;334:349-51. PMID 17303884 (PubMed 17303884)
11. Guyatt G, Sackett D, Taylor DW et al. Determining optimal therapy: randomized trials in individual patients. N Engl J Med. 1986;314(14):889-92. PMID 2936958 (PubMed 2936958)
12. Herrett E, Williamson E, Brack K et al. Statin treatment and muscle symptoms: series of randomised, placebo controlled n-of-1 trials. BMJ. 2021;372:n135. PMID 33627334 (PubMed 33627334)