Multiple comparisons
Definition: The inflation of false-positive risk when many hypotheses or analysis choices are tested.
Why it matters here: In Statistical Ethics and Inference, Multiple comparisons is needed to reason about pre-registration, multiplicity, train-test separation, and effect sizes rather than merely label the phenomenon.
Relationship: It is interpreted alongside P-hacking and the lecture relation below; identify which quantities are observed, calibrated, or model-derived.
Limitation or common confusion: Archive choice is part of scientific design.
\(p_adjusted=min(1,m p)\)
P-hacking
Definition: Selecting analyses or stopping rules after inspecting results to obtain a preferred significance claim.
Why it matters here: In Statistical Ethics and Inference, P-hacking is needed to reason about pre-registration, multiplicity, train-test separation, and effect sizes rather than merely label the phenomenon.
Relationship: It is interpreted alongside Data leakage and the lecture relation below; identify which quantities are observed, calibrated, or model-derived.
Limitation or common confusion: Archive choice is part of scientific design.
Data leakage
Definition: The unintended use of evaluation information during model training, tuning, or feature construction.
Why it matters here: In Statistical Ethics and Inference, Data leakage is needed to reason about pre-registration, multiplicity, train-test separation, and effect sizes rather than merely label the phenomenon.
Relationship: It is interpreted alongside Effect size and the lecture relation below; identify which quantities are observed, calibrated, or model-derived.
Limitation or common confusion: Archive choice is part of scientific design.
Effect size
Definition: The magnitude of a relationship or difference, reported separately from its statistical significance.
Why it matters here: In Statistical Ethics and Inference, Effect size is needed to reason about pre-registration, multiplicity, train-test separation, and effect sizes rather than merely label the phenomenon.
Relationship: It is interpreted alongside Preregistration and the lecture relation below; identify which quantities are observed, calibrated, or model-derived.
Limitation or common confusion: Archive choice is part of scientific design.
Preregistration
Definition: A time-stamped declaration of hypotheses, data rules, and analyses made before examining outcomes.
Why it matters here: In Statistical Ethics and Inference, Preregistration is needed to reason about pre-registration, multiplicity, train-test separation, and effect sizes rather than merely label the phenomenon.
Relationship: It is interpreted alongside Multiple comparisons and the lecture relation below; identify which quantities are observed, calibrated, or model-derived.
Limitation or common confusion: Archive choice is part of scientific design.