How to Read a Scientific Study

'A Study Found...'

These three words appear in thousands of news articles every day, and they're often where critical reading breaks down. Readers see 'study' and assume scientific rigor. But studies vary enormously in quality, and the gap between what a study actually found and what the headline claims it found can be enormous.

In 2015, a team led by Brian Nosek at the University of Virginia attempted to replicate 100 published psychology studies. Only 36% produced the same results. This 'replication crisis' shook the scientific world and should shake your trust in any single study reported in the news.

You don't need a PhD to evaluate scientific claims in news. You need a checklist of questions that separate strong evidence from weak evidence. The goal isn't to become a scientist — it's to become a reader who knows when to be skeptical.

The Seven-Question Study Checklist

1. Who conducted it? Is it from a university, a government agency, or a company selling a product? Industry-funded studies are 3-4 times more likely to produce results favorable to the sponsor, according to systematic reviews.

2. Where was it published? Peer-reviewed journals (Nature, The Lancet, JAMA) have higher standards than preprint servers (medRxiv, arXiv) or press releases. Preprints haven't been peer-reviewed — they're preliminary.

3. How big was the sample? A study of 15 people cannot tell you much about a population of millions. Look for the 'n=' number. For medical trials, hundreds to thousands is typical. For surveys, at least 1,000 for national claims.

4. Was there a control group? Without comparing to a group that didn't receive the treatment or intervention, you can't attribute the outcome to the treatment.

5. Was it randomized? Random assignment to treatment and control groups eliminates many confounding variables. Observational studies (no randomization) can show correlation but not causation.

6. What was the effect size? 'Statistically significant' doesn't mean 'large.' A drug that reduces symptoms by 2% might be statistically significant with a large enough sample but practically meaningless.

7. Has it been replicated? One study is a hypothesis. Multiple studies showing the same result are evidence. Systematic reviews and meta-analyses (which combine many studies) are the gold standard.

Red Flags in Science Reporting

Be extra skeptical when you see:

• 'Breakthrough' or 'miracle.' Real scientists almost never use these words. If the journalist uses them, they're editorializing.

• No link to the actual study. Good science journalism links to or names the specific paper. If the article says 'researchers found' without naming the researchers, the journal, or the institution, something is wrong.

• Results from a single study treated as settled science. Science is cumulative. One study doesn't prove anything — it contributes to a body of evidence.

• Animal studies presented as human results. 'Coffee cures cancer' might be based on a study where mice were given the equivalent of 50 cups per day. The leap from mice to humans is enormous.

• Relative risk without absolute risk. 'Doubles your risk' of a disease that affects 1 in 10 million people means your risk went from 0.00001% to 0.00002%. Not quite the panic the headline implies.