Learning objectives
- Recognize p-hacking, multiple comparisons, leakage, and uncertainty laundering.
- Explain pre-registration, multiplicity, train-test separation, and effect sizes with the formal object.
- Separate evidence, transformation, inference, and limitation.
Terms Developed in Context
These terms form the lecture's reasoning chain from observable through model to a bounded physical conclusion.
- Multiple comparisons
The inflation of false-positive risk when many hypotheses or analysis choices are tested. In this lecture it supports the evidence-to-model chain for pre-registration, multiplicity, train-test separation, and effect sizes.
Use and relationship: Read it together with Preregistration and P-hacking. The governing relation \(p_adjusted=min(1,m p)\) shows where the concept enters quantitatively; the student should state whether its value is measured, calibrated, adopted, or inferred.
Boundary: Archive choice is part of scientific design. This caveat prevents the term from being treated as a unique explanation or a context-free number.
- P-hacking
Selecting analyses or stopping rules after inspecting results to obtain a preferred significance claim. In this lecture it supports the evidence-to-model chain for pre-registration, multiplicity, train-test separation, and effect sizes.
Use and relationship: Read it together with Multiple comparisons and Data leakage. The governing relation \(p_adjusted=min(1,m p)\) shows where the concept enters quantitatively; the student should state whether its value is measured, calibrated, adopted, or inferred.
Boundary: Archive choice is part of scientific design. This caveat prevents the term from being treated as a unique explanation or a context-free number.
- Data leakage
The unintended use of evaluation information during model training, tuning, or feature construction. In this lecture it supports the evidence-to-model chain for pre-registration, multiplicity, train-test separation, and effect sizes.
Use and relationship: Read it together with P-hacking and Effect size. The governing relation \(p_adjusted=min(1,m p)\) shows where the concept enters quantitatively; the student should state whether its value is measured, calibrated, adopted, or inferred.
Boundary: Archive choice is part of scientific design. This caveat prevents the term from being treated as a unique explanation or a context-free number.
- Effect size
The magnitude of a relationship or difference, reported separately from its statistical significance. In this lecture it supports the evidence-to-model chain for pre-registration, multiplicity, train-test separation, and effect sizes.
Use and relationship: Read it together with Data leakage and Preregistration. The governing relation \(p_adjusted=min(1,m p)\) shows where the concept enters quantitatively; the student should state whether its value is measured, calibrated, adopted, or inferred.
Boundary: Archive choice is part of scientific design. This caveat prevents the term from being treated as a unique explanation or a context-free number.
- Preregistration
A time-stamped declaration of hypotheses, data rules, and analyses made before examining outcomes. In this lecture it supports the evidence-to-model chain for pre-registration, multiplicity, train-test separation, and effect sizes.
Use and relationship: Read it together with Effect size and Multiple comparisons. The governing relation \(p_adjusted=min(1,m p)\) shows where the concept enters quantitatively; the student should state whether its value is measured, calibrated, adopted, or inferred.
Boundary: Archive choice is part of scientific design. This caveat prevents the term from being treated as a unique explanation or a context-free number.
Method in context
Statistical ethics demands disclosure of search degrees of freedom and the full analysis path. Ask what a second researcher needs to reproduce the claim: raw input, processing state, software or database version, and inferential step.
Derivation and assumptions
Define every symbol and explain the failure mode. A method becomes scientific when assumptions are visible enough to be tested.
Worked example
MAST serves mission products, Gaia astrometric catalogs, SDSS imaging/spectroscopy, and Exoplanet Archive curated tables; product scope is the discriminant.
Archive choice is part of scientific design. Require an independent recomputation and one perturbation.
Misconceptions and study guidance
Do not confuse a data service with a measurement, a catalog identifier with physical identity, a formal error with total uncertainty, or a polished figure with a validated inference. Reconstruct the evidence chain before interpreting.
Verified reading
NASA Open Science Overview; Astropy Statistics documentation; ASA ethical guidance.
Use the exact documentation anchor in the reference log and record any access mismatch.