On this page
A result can be clear enough to read and still impossible to rebuild. The data file is missing. A manual exclusion was made in a spreadsheet. The analysis used a library version nobody recorded. The reported chart has no rule for which rows entered it.
Reproducibility asks whether another analyst can reconstruct a result from the materials and procedure specified for it. The object is the path from question to output, not the confidence or importance of the conclusion.
The evidence-review stopping-rule article asks when a claim has earned decision weight. The measurement-invariance article asks whether a result can travel to a named target. This page asks the earlier question: can another analyst rerun the result in the first place?
What does reproducibility mean?
The practical contract names:
| Component | Required question | Example failure |
|---|---|---|
| Question or estimand | What was the analysis trying to calculate or compare? | The chart has no target quantity |
| Inputs | Which data files, versions, filters, and access conditions were used? | A current table replaces the historical snapshot |
| Procedure | Which transformations, exclusions, and decisions occurred in what order? | A manual step is remembered but not recorded |
| Code | Which analysis script and parameters produced the output? | A notebook cell or formula is missing |
| Environment | Which runtime, libraries, configurations, and random state matter? | A package update changes the result |
| Output | What exact table, estimate, chart, or file should appear? | “Same result” has no tolerance or comparison rule |
Table 1What does reproducibility mean?
Source: Table from this essay. Sources and interpretation are given in the article.
Banzi et al. (2026) present a consensus list of core reproducibility items across planning, methods, data collection, management and analysis, and dissemination. The source is a research-consensus view, not a validated business scorecard. Its value here is the breadth of the reconstruction contract: reproducibility begins before the final calculation.
Is reproducibility the same as replication?
No. Keep the tests separate:
| Test | Data and procedure | Question answered | What it does not answer |
|---|---|---|---|
| Reproducibility | Same declared materials and procedure | Can the result be reconstructed? | Whether the result is true or general |
| Replication | New observation or data, with an intendedly similar design | Does a result recur under a new observation? | Whether the original workflow was fully reconstructable |
| External validity | Source result and a named target | Does the conclusion transfer? | Whether the source analysis can be rerun |
Table 2Is reproducibility the same as replication?
Source: Table from this essay. Sources and interpretation are given in the article.
The Open Science Collaboration re-ran 100 psychology studies and reported more than one replication success measure. The lesson for a commercial review is not a percentage to import. It is that the question “did it replicate?” needs a declared criterion, population, outcome, and comparison.
What does a reconstruction worksheet look like?
The six rows below are synthetic. They illustrate readiness states, not a reproducibility rate for a company, discipline, or dataset.
| ID | Question or output | Inputs and data package | Code and environment | Expected output | Rerun state and disposition |
|---|---|---|---|---|---|
| R-01 | Cohort rate at day 14 | Snapshot V3 and eligibility rule held | Script S-12; runtime recorded | 50.0% | Confirmed within stated tolerance |
| R-02 | Revenue bridge by line | Invoice file held; product map version unclear | Script S-18; package lock missing | Five-effect bridge | Mismatch risk; hold release |
| R-03 | Survey mean by locale | Data package and codebook held | Analysis script held; exclusion manual | Table of means and counts | Rerun pending exclusion log |
| R-04 | Forecast error by horizon | Baseline and actual files held | Notebook and library versions held | Error table and plot | Confirmed; no claim of causality |
| R-05 | Customer segment model | Access restricted; derived file only | Script available; random seed absent | Model metrics | Unavailable input; cannot reconstruct |
| R-06 | Treatment contrast | Preregistered dataset and treatment rule held | Script and environment held | Effect estimate and interval | Reconstructed; replication still open |
Figure 1The synthetic reconstruction worksheet
The rows are illustrative. Confirmed reconstruction, replication, and causal validity remain separate statuses.
Source: Author's synthetic worksheet grounded in Banzi et al. (2026) and the Open Science Collaboration (2015); all rows and states are illustrative.
R-02 is not “almost reproducible” because its final chart looks plausible. A missing product-map version can change the line population. R-05 is not solved by publishing model metrics when the input cannot be reconstructed. R-06 is reproducible as a workflow, while the replication question remains open because it needs new observation.
Which decisions belong in the workflow record?
Keep these decisions visible rather than burying them in prose:
- the unit, target quantity, and analysis window;
- the source file, snapshot, version, and access condition;
- every filter, exclusion, join, transformation, and manual edit;
- the analysis script, parameter file, runtime, library versions, and random seed where relevant;
- the expected output, comparison tolerance, and known nondeterminism;
- the person or process that approved a change and the date it took effect;
- the result of the rerun, including mismatch and unavailable states.
Preregistration can record selected questions or analysis decisions before observation. It does not make an analysis immune to change. A changed decision can be legitimate when it is dated, explained, and distinguished from the original plan.
What can a successful rerun prove?
It can show that the specified path produces the specified output under the specified environment. It cannot, by itself, prove that the input is accurate, the construct is valid, the model is correctly specified, the comparison is causal, or the result transfers to a new target.
That boundary is why a reconstruction worksheet should contain both a status and a disposition. A confirmed rerun can release the method for review. A mismatch can open a debugging question. An unavailable input can block a stronger claim. None of those statuses is a substantive outcome.
How should a team test reproducibility?
- Freeze the question, unit, outcome, and observation window.
- Assemble the exact input versions and data package.
- Replay every transformation, exclusion, and manual step.
- Run the declared script in the declared environment.
- Compare the output to the stated result with a declared tolerance or exact-match rule.
- Record mismatch, unavailable input, environment drift, or successful reconstruction.
- Treat a new-data test as replication and a target change as external-validity work.
Reproducibility is evidence about the reconstruction path. It is not a universal score, a guarantee of truth, or a substitute for a new study.
References
- Banzi, R., Varga, M., Gelsleichter, Y. A., Vinatier, C., Moher, D., & Naudet, F. (2026). An international consensus on core reproducibility items in research. PLoS Biology, 24(4), e3003726. https://doi.org/10.1371/journal.pbio.3003726
- Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
