Lecture 13: Reproducible Simulation Workflows

Terms Developed in Context

The terms below connect the mathematical model, implementation choices, computed diagnostics, and limits of the scientific claim.

Reproducibility

The ability to regenerate a result from recorded code, data, parameters, environment, and random state. In this lecture it is used to address What must another researcher receive to reproduce a numerical claim?

Model connection: Read the term against Provenance and the governing relation \(\mathrm{claim}=f(\mathrm{code},\mathrm{parameters},\mathrm{data},\mathrm{versions},\mathrm{random\ state})\). The relevant computational evidence is a controlled output, residual, invariant, or refinement sequence rather than visual smoothness alone.

Interpretive limit: Reproducibility is not validity. Repeating the same bug or approximation exactly does not provide physical verification. This distinction separates numerical behavior from physical accuracy and model adequacy.

Provenance

The documented origin and transformation history of every input and derived artifact. In this lecture it is used to address What must another researcher receive to reproduce a numerical claim?

Model connection: Read the term against Dependency pinning and the governing relation \(\mathrm{claim}=f(\mathrm{code},\mathrm{parameters},\mathrm{data},\mathrm{versions},\mathrm{random\ state})\). The relevant computational evidence is a controlled output, residual, invariant, or refinement sequence rather than visual smoothness alone.

Dependency pinning

Recording exact software versions needed to recreate an execution environment. In this lecture it is used to address What must another researcher receive to reproduce a numerical claim?

Model connection: Read the term against Checksum and the governing relation \(\mathrm{claim}=f(\mathrm{code},\mathrm{parameters},\mathrm{data},\mathrm{versions},\mathrm{random\ state})\). The relevant computational evidence is a controlled output, residual, invariant, or refinement sequence rather than visual smoothness alone.

Checksum

A content digest used to detect whether a file has changed. In this lecture it is used to address What must another researcher receive to reproduce a numerical claim?

Model connection: Read the term against Regression test and the governing relation \(\mathrm{claim}=f(\mathrm{code},\mathrm{parameters},\mathrm{data},\mathrm{versions},\mathrm{random\ state})\). The relevant computational evidence is a controlled output, residual, invariant, or refinement sequence rather than visual smoothness alone.

Regression test

An automated check that protects a previously established scientific or software behavior. In this lecture it is used to address What must another researcher receive to reproduce a numerical claim?

Model connection: Read the term against Reproducibility and the governing relation \(\mathrm{claim}=f(\mathrm{code},\mathrm{parameters},\mathrm{data},\mathrm{versions},\mathrm{random\ state})\). The relevant computational evidence is a controlled output, residual, invariant, or refinement sequence rather than visual smoothness alone.

Problem and physical meaning

A result depends on code, parameters, data, versions, random state, and execution conditions. A clean environment and deterministic inputs support repeatability, while independent checks and provenance make the result auditable.

What must another researcher receive to reproduce a numerical claim? Begin from the scientific observable or state variable, and name what is directly specified versus inferred. The model should be simple enough to validate, but not quietly presented as the entire astrophysical system.

Model and derivation

The governing expression for this lecture is:

\[\mathrm{claim}=f(\mathrm{code},\mathrm{parameters},\mathrm{data},\mathrm{versions},\mathrm{random\ state})\]

Define the symbols and the independent variable before numerical evaluation. Dimensional analysis checks that both sides have matching units; if the model is dimensionless, state the adopted scale factors and the transformation back to physical units.

Numerical implementation

Pin dependencies; record runtime and data checksums; separate raw inputs from generated outputs; run tests and convergence checks from a clean checkout; capture commands, seeds, and expected ranges.

Keep physical quantities and numerical arrays conceptually separate. Convert units at well-defined boundaries, preserve raw inputs, and record dtype, solver, tolerances/timestep, grid, and software versions. A numerical method is part of the model evidence, not an invisible implementation detail.

Worked, programmatically computed example

The executed environment reports numpy 2.5.3, scipy 1.18.1, astropy 8.0.1, rebound 5.2.0; the same pins are recorded in requirements.txt.

These values are generated in src/astr474_computations.py, then shared with lecture/lab/assignment renderers. Re-run the generator after changing an input; do not hand-edit a displayed result without recomputing dependent artifacts.

At least one independent check is required: an analytic solution, a second numerical method, a conservation law, or a controlled convergence sequence. When these disagree, diagnose units, frame, discretization, source terms, and model scope before selecting a result.

Error budget and limitations

Reproducibility is not validity. Repeating the same bug or approximation exactly does not provide physical verification.

Classify each uncertainty before combining terms: local/global solver error, timestep or spatial discretization, Monte Carlo sampling, parameter uncertainty, measurement uncertainty, and model discrepancy are not interchangeable. Conservation can test internal consistency but does not prove the model is physically complete.

Connection to evidence and research practice

Lab 07 rebuilds the package from its pinned requirements and audits outputs against independent invariants.

For real parameters, the source record distinguishes the fetched product and epoch from derived values. For synthetic experiments, the initial conditions, generator, seed, and exact file are recorded. A simulation is not an observation, even when its parameters resemble a real object.

Misconceptions to avoid

More displayed digits do not imply more physical accuracy.
A successful solver exit or smooth plot does not establish convergence.
Agreement between two solvers with the same wrong model is not independent physical validation.
A formal covariance or Monte Carlo interval is conditional on the adopted model and noise process.

Study and review questions

What must another researcher receive to reproduce a numerical claim?
Which numerical parameter would you vary first, what observable would you track, and how would you know the sequence reached an asymptotic regime?
Design one independent check and state what failure would change your scientific interpretation.

Preparation for the next lecture: explain how this method's assumptions change when the problem becomes more coupled, stochastic, or data-driven.

Lecture-specific derivation and assumptions

A computational claim is a function of source code, parameters, data, software environment, and execution state. Repeatability requires preserving these inputs; auditability additionally needs provenance and independent checks. SHA-256 maps file bytes to a 256-bit digest, making accidental changes detectable, though a checksum does not establish authorship, authenticity, or scientific validity. Dependency pins constrain package versions but not compiler, BLAS, operating system, CPU arithmetic, or all random-stream guarantees.

Algorithm trace and computed-output interpretation

From a clean environment, install the exact requirements, record interpreter/package versions, run the one-command build, and compare every generated data file with its SHA256SUMS entry. Change one input deliberately and confirm the digest check fails; restore it by rebuilding from source. Then independently recompute one analytic limit and one numerical convergence result instead of treating matching hashes as validation.

Computed result: The executed environment reports numpy 2.5.3, scipy 1.18.1, astropy 8.0.1, rebound 5.2.0; the same pins are recorded in requirements.txt.

A deterministic archive allows a peer to identify changed inputs and reproduce expected outputs; it cannot show that the model is complete or the chosen precision is justified. Keep raw source data separate from generated files and list the data level beside each file. A checksum manifest should include files but not hash itself.

Two-meeting teaching arc

Each weekly module is taught in two 75-minute meetings. The sequence below includes a formative check in each meeting and is intended to use the lecture's worked computation, not merely present it.

Meeting 1 (75 min): 10 min reproducibility case study; 20 min dependency/input graph; 20 min provenance levels and raw/generated separation; 20 min checksum semantics; 5 min state one authenticity limitation.
Meeting 2 (75 min): 10 min recap; 20 min clean rebuild; 25 min inspect validator/checksum failures; 15 min peer audit design; 5 min distinguish repeatability from validity.

Targeted study questions

Which environmental dependencies remain outside a Python requirements file?
Why must the checksum manifest not include its own digest?
What independent scientific check would catch a perfectly reproducible sign error?

References and provenance: course reference log. Generated 2026-09-25.