Causal Inference & Experimentation

Potential outcomes

A causal effect compares what a unit's outcome would be with treatment, Y(1), versus without, Y(0). Each unit reveals only one of the two — the fundamental problem of causal inference. Every method here is a strategy for estimating averages of effects despite half of each comparison being forever missing.

Selection bias

The naive contrast equals the causal effect plus a bias term: how the treated would have differed even without treatment. Randomization is the one design that deletes this term outright — which is why experiments are the default and everything else is the fallback.

Estimands: ATE, ATT, LATE

Decide what you're estimating before how: the average effect for everyone (ATE), for the treated (ATT), for units an instrument can move (LATE), or at a threshold (RDD). Under heterogeneous effects these are different numbers for the same program — say which one you're reporting, always.

Confounders, mediators, colliders

Causal graphs (DAGs) give the adjustment rule: control for confounders (common causes), never for mediators (the mechanism) if you want the total effect, and never for colliders (common consequences) — conditioning on a collider manufactures bias. "Control for everything" is not a safe default.

Difference-in-differences

Compare the treated group's before–after change with an untreated group's change; stable group differences cancel out.

Key assumption Parallel trends — probeable with pre-period data.

Use when A policy or launch hits one group at a known time.

Synthetic control

Build a weighted blend of untreated markets that tracks the treated market pre-launch; the post-launch gap is the effect. Inference by placebo permutation. The engine behind geo experiments.

Key assumption Pre-period fit predicts the counterfactual.

Use when Rollouts are market-by-market and users interfere.

Instrumental variables (2SLS)

A variable that nudges treatment but touches the outcome only through treatment (a randomized encouragement letter, distance to a clinic) identifies the effect for compliers — the LATE — even with unobserved confounding.

Key assumption Exclusion (untestable); relevance is testable — report the first-stage F.

Use when Take-up is self-selected but the nudge was (as-good-as) random.

Regression discontinuity

When a threshold rule assigns treatment (risk score ≥ 65 → auto-enroll), units just above and below the cutoff are locally comparable; the jump in the outcome at the cutoff is the effect.

Key assumption No precise manipulation of the score; check placebos, bandwidths, density.

Use when Eligibility is score- or cutoff-based. Estimate is local to the threshold.

Propensity scores: IPW & matching

Model the probability of treatment given observed covariates, then reweight (IPW) or match units to mimic a randomized comparison. Balance and overlap diagnostics are most of the job.

Key assumption No unmeasured confounding — untestable.

Use when Selection runs through covariates you actually observe.

Doubly robust (AIPW)

Combine an outcome model with propensity weights: consistent if either model is right. The same construction underlies double/debiased machine learning.

Key assumption Same as propensity methods — but one wrong nuisance model is survivable.

Use when You're adjusting on observables and want model-misspecification insurance.

Sensitivity analysis (E-values)

"No unmeasured confounding" can't be verified — only quantified. The E-value is the minimum strength an unmeasured confounder would need, with both treatment and outcome, to fully explain away your estimate.

Use when Always — next to every observational estimate you report.

Fixed & random effects

Fixed effects (the within estimator) demeans each unit's data, removing all stable confounding; identification comes from within-unit change. Random effects is more efficient but assumes the unit effect is uncorrelated with the regressors; the Hausman test arbitrates. DiD is FE's two-period special case.

Caveat FE can't fix time-varying confounding, and can't estimate time-constant predictors.

Cross-lagged panel models

Two processes co-evolve — engagement and study time, strain and health. Which leads? Regress each wave on the previous wave of both. The classic caveat (RI-CLPM): without modeling stable trait differences, cross-lags mix within-person influence with "high-x people are high-y people".

Use when The question is temporal precedence, not a single treatment effect.

Latent growth curves

Give every unit its own trajectory (intercept + slope) and model the distribution: average growth, how much growth varies, and whether units that start high grow faster. Individual change becomes the estimand instead of a nuisance.

Use when Development over time is itself the outcome — learning, adherence, careers.

SEM & latent variables

Structural equation models chain regressions into path models and add measurement models: latent constructs measured by multiple indicators, correcting the attenuation that measurement error causes in any regression. Growth curves and RI-CLPMs are special cases. The causal content still comes from the assumed graph — not the fitting machinery.

Marginal structural models

With time-varying treatment, a time-varying confounder can also be a mediator — plain regression and FE both fail. The g-methods (IPW over treatment histories, g-computation) are the standard solution from epidemiology.

Use when Treatment and confounders feed back on each other over time.

Power analysis & MDE

Size the test before collecting data, for the smallest effect that would change the ship decision — not the effect you hope for. Underpowered tests are the most common experimentation failure.

Randomization & bucketing

Salted-hash bucketing: hash (salt, user id) to a variant. Deterministic — same user, same variant, no storage — and different salts give independent assignments, which is what lets many experiments overlap safely. Geo and switchback designs handle interference.

Integrity: SRM & A/A tests

Sample-ratio mismatch (a chi-square on the observed split) means assignment or logging is broken — results must not be read. A/A tests validate the whole pipeline: null experiments should produce uniform p-values. Check integrity before metrics, always.

Variance reduction (CUPED)

Subtract the part of the outcome predictable from pre-experiment behavior. Same effect estimate, much tighter interval — routinely worth 2–3× the effective sample size.

Ratio metrics & clustering

CTR-style metrics are ratios of correlated within-user events — treating each event as independent understates the variance (delta method fixes it). Same trap when analysis is finer than randomization: cluster-robust SEs on the assignment unit.

Sequential testing

Stakeholders will peek daily; repeatedly checking a fixed-horizon p-value inflates false positives badly. Always-valid p-values (mSPRT) and confidence sequences keep the guarantee under continuous monitoring and optional stopping.

Multiple testing

A metric scorecard is many tests read together: with enough null metrics, something "wins" by chance. Benjamini–Hochberg controls the false-discovery rate for exploratory metrics; Bonferroni protects a short pre-registered guardrail list.

Bayesian readouts

Ship decisions ask "what's the probability treatment is better, and what do we lose if we're wrong?" — P(better) and expected loss, the quantities a launch review actually uses.