How Science Tests a Claim: Thirteen Concepts From Observation to Replication

Every test starts with someone noting what happens, by eye or by instrument. This entry also carries the warning that the noting is never perfectly neutral, since what counts as a relevant observation is already shaped by the observer's assumptions.

Norwood Russell Hanson argued in 1958 that two scientists holding different theories can read the same instrument and, in a meaningful sense, see different things. Whether that blocks evidence from ever settling a dispute is contested, and the weaker, more widely accepted reading says observation is coloured by theory but can still correct it.

The loop of observing, forming a testable hypothesis, predicting what follows and checking the prediction is the frame for everything after this stop. Historians and philosophers still argue that no single fixed method is what scientists actually follow, so read the later stops as working parts of a family of practices.

Karl Popper's 1934 criterion says a theory is scientific only if it forbids some possible observation, so a real result could in principle show it false. Imre Lakatos and the Duhem-Quine thesis complicate this, because an anomalous result can often be blamed on an auxiliary assumption instead.

When two explanations fit the evidence equally well, prefer the one that assumes the least. It is named for William of Ockham, though he never phrased it as one maxim, and it recommends a better starting point rather than proving a simpler theory true.

James Lind's 1747 shipboard comparison of six pairs of sailors with scurvy, each pair given a different treatment, is one of the earliest documented examples. The pair given citrus fruit recovered, which shows why a comparison group lets a difference be attributed to the treatment.

Patients given a treatment with no active ingredient can still improve through expectation alone. That is the baseline a new drug must beat, and Henry Beecher's 1955 paper The Powerful Placebo is credited with giving the effect its modern place in trial design.

Keeping the participant unaware is not enough, because a researcher who knows who is on the real drug can read symptoms differently without meaning to. In a double-blind design neither side knows, and the Nuremberg salt test of 1835 is an early example of the protocol.

Randomization assigns people to groups by chance so that known and unknown differences between them spread evenly. The Medical Research Council's 1948 streptomycin trial for tuberculosis, designed by Austin Bradford Hill, is credited as the first to allocate patients with a formal random-numbers procedure.

Ronald Fisher's 1925 book formalised the p-value and the conventional threshold of 0.05 for judging whether a result is worth taking seriously. The cutoff is a convention he did not intend as a rigid rule, and a 2016 American Statistical Association statement warned against using it in place of judgment.

Independent experts examine a claimed finding before it is published. The Royal Society's Philosophical Transactions, begun in 1665, had its Council review submissions before printing them, and the practice is still contested for reviewer bias and for missing fraud.

Reproducibility asks whether an independent researcher with the original data and method gets the same result, while replicability asks whether a fresh study with new data finds the same thing. A result nobody else can reproduce has not really been confirmed, whatever its authors believed.

The Open Science Collaboration's 2015 project tried to replicate 100 psychology studies and reproduced the original result in well under half of them. A 2016 Nature survey of over 1,500 scientists found more than 70 percent had failed to reproduce another scientist's experiment, and researchers still debate how far the problem reaches.