What must another researcher receive to reproduce a numerical claim?
14-week course · 3 credits · Week 13
Motivation
What must another researcher receive to reproduce a numerical claim?
A result depends on code, parameters, data, versions, random state, and execution conditions. A clean environment and deterministic inputs support repeatability, while independent checks and provenance make the result auditable.
Today’s result must be interpretable as a physical statement with declared data, algorithm, uncertainty, and a check that could fail.
Learning objectives
After this lecture, you can
What must another researcher receive to reproduce a numerical claim?
Pin dependencies
Reproducibility is not validity. Repeating the same bug or approximation exactly does not provide physical verification.
Core language
Terms for This Lecture
Reproducibility
Definition: The ability to regenerate a result from recorded code, data, parameters, environment, and random state.
Why it matters here: It helps answer What must another researcher receive to reproduce a numerical claim? within the numerical model described here.
Relationships: Connect it to Provenance, the governing expression \(\mathrm{claim}=f(\mathrm{code},\mathrm{parameters},\mathrm{data},\mathrm{versions},\mathrm{random\ state})\), and the computed diagnostic developed later in the lecture.
Boundary or common confusion: Reproducibility is not validity. Repeating the same bug or approximation exactly does not provide physical verification.
Provenance
Definition: The documented origin and transformation history of every input and derived artifact.
Why it matters here: It helps answer What must another researcher receive to reproduce a numerical claim? within the numerical model described here.
Relationships: Connect it to Dependency pinning, the governing expression \(\mathrm{claim}=f(\mathrm{code},\mathrm{parameters},\mathrm{data},\mathrm{versions},\mathrm{random\ state})\), and the computed diagnostic developed later in the lecture.
Boundary or common confusion: Reproducibility is not validity. Repeating the same bug or approximation exactly does not provide physical verification.
Dependency pinning
Definition: Recording exact software versions needed to recreate an execution environment.
Why it matters here: It helps answer What must another researcher receive to reproduce a numerical claim? within the numerical model described here.
Relationships: Connect it to Checksum, the governing expression \(\mathrm{claim}=f(\mathrm{code},\mathrm{parameters},\mathrm{data},\mathrm{versions},\mathrm{random\ state})\), and the computed diagnostic developed later in the lecture.
Boundary or common confusion: Reproducibility is not validity. Repeating the same bug or approximation exactly does not provide physical verification.
Checksum
Definition: A content digest used to detect whether a file has changed.
Why it matters here: It helps answer What must another researcher receive to reproduce a numerical claim? within the numerical model described here.
Relationships: Connect it to Regression test, the governing expression \(\mathrm{claim}=f(\mathrm{code},\mathrm{parameters},\mathrm{data},\mathrm{versions},\mathrm{random\ state})\), and the computed diagnostic developed later in the lecture.
Boundary or common confusion: Reproducibility is not validity. Repeating the same bug or approximation exactly does not provide physical verification.
Regression test
Definition: An automated check that protects a previously established scientific or software behavior.
Why it matters here: It helps answer What must another researcher receive to reproduce a numerical claim? within the numerical model described here.
Relationships: Connect it to Reproducibility, the governing expression \(\mathrm{claim}=f(\mathrm{code},\mathrm{parameters},\mathrm{data},\mathrm{versions},\mathrm{random\ state})\), and the computed diagnostic developed later in the lecture.
Boundary or common confusion: Reproducibility is not validity. Repeating the same bug or approximation exactly does not provide physical verification.
Physical and mathematical model
State what is being evolved
A result depends on code, parameters, data, versions, random state, and execution conditions. A clean environment and deterministic inputs support repeatability, while independent checks and provenance make the result auditable.
Define the state, coordinates, units, initial/boundary conditions, and domain before interpreting solver output.
Numerical method
Algorithm choice is a modeling decision
Pin dependencies; record runtime and data checksums; separate raw inputs from generated outputs; run tests and convergence checks from a clean checkout; capture commands, seeds, and expected ranges.
Write the model and its assumptions.
Choose a solver and identify its stability, accuracy, and conservation properties.
Record configuration, tolerances or timestep, and output sampling.
Compare against an analytic limit, independent method, or controlled refinement.
Visual reasoning
Reproducible workflow from source to claim
Distinct workflow graph showing provenance and review gates; each edge represents an auditable artifact dependency.
Computed example
One result, with a check
The executed environment reports numpy 2.5.3, scipy 1.18.1, astropy 8.0.1, rebound 5.2.0; the same pins are recorded in requirements.txt.
Inputs and method are generated by src/astr474_computations.py; the machine-readable values are in data/generated/computed_results.json.
Reproducibility is not validity. Repeating the same bug or approximation exactly does not provide physical verification.
Verification
What would falsify this result?
Repeat with a smaller timestep, tighter tolerance, or finer spatial grid.
Track conservation or residual structure in addition to visual agreement.
Check units, coordinate frame, boundary conditions, and invariants independently.
Change a random seed when the method is stochastic; report the resulting sampling uncertainty.
Do not report a smoother curve as proof of convergence.
Failure modes
Numerical agreement has a domain
Reproducibility is not validity. Repeating the same bug or approximation exactly does not provide physical verification.
Solver error, discretization error, stochastic error, parameter uncertainty, and model discrepancy are different quantities. Diagnose and report them separately.
Active reasoning
Make a prediction before rerunning
What must another researcher receive to reproduce a numerical claim?
State a measurable outcome, a competing explanation, and the smallest computation that distinguishes them. Include an expected direction or scaling, then test it.
Workflow
From parameters to auditable output
Pin dependencies; record runtime and data checksums; separate raw inputs from generated outputs; run tests and convergence checks from a clean checkout; capture commands, seeds, and expected ranges.
Record the exact command, package versions, data path, random seed if applicable, and output checksum. Keep raw inputs unchanged and generated files separate.
Transfer
Lab and problem-set connection
Lab 07 rebuilds the package from its pinned requirements and audits outputs against independent invariants.
Submit plots with labeled axes/units, a concise uncertainty argument, and enough code or intermediate values for a peer to reproduce the claim.
Synthesis
Three durable checks
Does the implemented equation match the intended physical model?
Does the numerical result converge under a controlled method change?
Does the interpretation stay inside the validity and provenance limits?
Reproducibility is not validity. Repeating the same bug or approximation exactly does not provide physical verification.