Why Ordinary P-Values Struggle When Hypotheses Compete: The Case for Compound P-Values

Classical hypothesis testing was engineered for a world of single, isolated questions: does this drug outperform placebo, is this coin fair, does this batch meet specification. In that tidy universe, a p-value is a faithful messenger, reporting the probability of observing data at least as extreme as ours if the null hypothesis were true. Modern science, however, rarely asks one question at a time. Genomic screens interrogate twenty thousand genes simultaneously, advertising platforms pit dozens of creative variants against one another, and clinical trials compare competing dosing regimens across stratified cohorts. In these arenas, hypotheses do not merely coexist; they compete, and the winner is declared by whoever produces the smallest p-value.

That competition quietly breaks the guarantees practitioners assume they possess. When many candidates race toward significance, the victor's p-value is no longer a calibrated measure of evidence against a single null; it is the minimum of a random collection, and minima are notoriously optimistic. Standard corrections such as Bonferroni or Benjamini–Hochberg patch some of the damage, yet they were designed for a different geometry of inference, one where each hypothesis stands alone rather than being crowned through rivalry. Compound p-values emerge precisely here, offering a principled construction that respects the competitive structure and restores error-control guarantees that ordinary tools cannot deliver.

This analysis dissects why conventional p-values stumble when hypotheses compete, formalizes the competition-based testing framework, and derives the compound p-value machinery with its validity guarantees. Along the way, we will work through ten explicit calculations, from Bonferroni arithmetic to the distribution of a minimum, so that the mathematics becomes tangible rather than abstract. The stakes span genomics, online experimentation, finance, and any domain where selection among many candidates is the operative act of inference.

The Anatomy of a P-Value and Where It Frays

A p-value is defined relative to a single null hypothesis and a fixed test statistic, and its uniformity under that null is the property on which every downstream guarantee rests. Formally, if the null holds, then ##[P(p leq alpha) = alpha]## for every significance level ##[alpha in (0,1)]##. This uniformity is the load-bearing wall of classical inference, and it collapses the moment selection enters the picture.

The Uniformity Property and Its Fragility

Under a true null, a p-value behaves like a uniform random draw on the unit interval, meaning each threshold is exceeded with exactly the stated probability. This calibration is what licenses the familiar statement that a test at ##[alpha = 0.05]## falsely rejects only five percent of the time. Remove the single-hypothesis setting, and the uniformity argument no longer applies to the quantity actually reported.

Consider a researcher who tests one hypothesis and reports its p-value. The uniformity guarantee holds exactly. Now consider a researcher who tests fifty hypotheses and reports only the smallest p-value. That reported number is a minimum of fifty draws, and its distribution is dramatically skewed toward zero even when every null is true. The reported statistic and the tested statistic are no longer the same object.

This mismatch is not a minor technicality; it is the root cause of the replication crisis in fields that routinely select winners from large candidate pools. The published p-value describes a selected extreme, not a pre-specified test. Any inferential claim built atop it inherits the selection bias wholesale, and no amount of post-hoc correction fully repairs the conceptual confusion.

The cleanest way to see the fraying is to compute the null distribution of the minimum directly. If ##[p_1, dots, p_m]## are independent and uniform under their respective nulls, then the minimum ##[p_{(1)}]## satisfies a simple survival identity that we now derive explicitly.

Differentiating yields the density ##[f_{(1)}(t) = m(1-t)^{m-1}]##, whose expectation is ##[1/(m+1)]##. With fifty hypotheses, the expected smallest p-value under a global null is roughly ##[1/51 approx 0.0196]##, comfortably below the sacred 0.05 threshold. A field that reports minima will therefore "discover" effects at the five percent level more than half the time even when nothing is real.

Why Multiplicity Corrections Miss the Competition

Bonferroni and its descendants control error rates by shrinking the per-test threshold, treating each hypothesis as an independent claimant on a shared error budget. That logic is sound for families of separate questions, but it ignores the relational structure of a competition, where only the winner's evidence is ever examined. The correction adjusts thresholds without modeling the selection mechanism itself.

Benjamini–Hochberg improves power by controlling the false discovery rate, yet it too assumes a collection of hypotheses whose p-values can be ordered and thresholded. When the scientific claim is explicitly "the best candidate is genuinely best," the relevant error event is not any single false rejection but the specific failure of crowning a null. Standard FDR machinery does not target that event.

Worse, competition often induces dependence among the candidate p-values, because all contenders are evaluated on shared data, shared controls, or shared noise. Corrections calibrated for independence or positive regression dependence can become anti-conservative under adversarial competition structures. The guarantees evaporate precisely where the stakes are highest.

Formalizing Competition-Based Testing

Competition-based testing replaces the isolated null with a structured family in which a designated winner emerges from a selection rule. The inferential target shifts from "is hypothesis ##[i]## false?" to "is the selected hypothesis false?" This reframing demands new machinery, because the object of inference is now a random index determined by the data themselves.

Selection Rules and the Winner's Curse

Let ##[mathcal{H} = {H_1, dots, H_m}]## be a family of null hypotheses, each paired with a test statistic ##[T_i]## and p-value ##[p_i]##. A selection rule ##[hat{i} = argmin_i p_i]## declares the smallest p-value the winner. The winner's curse is the systematic bias that arises because ##[hat{i}]## is chosen precisely for having an unusually small p-value, whether or not its null is false.

Quantifying the curse requires the joint distribution of the p-values, which in turn depends on the dependence structure among the test statistics. Under independence and a global null, we already derived the density of the minimum. Under arbitrary dependence, no closed form exists, and the analyst must resort to bounds or resampling to characterize the selection event.

The practical consequence is stark: the naive winner's p-value, ##[p_{hat{i}}]##, is not uniform under the global null and cannot be compared to ##[alpha]## without adjustment. Any claim of the form "the best candidate is significant at level ##[alpha]##" requires a corrected statistic whose null distribution accounts for the argmin operation.

A useful benchmark is the probability that the naive winner clears a fixed threshold under the global null. Setting ##[t = 0.05]## and ##[m = 20]##, the survival identity gives the following calculation, which quantifies exactly how often the naive procedure misleads.

Roughly sixty-four percent of global-null experiments with twenty candidates will produce a "significant" winner at the five percent level. The naive procedure is not merely slightly miscalibrated; it is catastrophically anti-conservative, and the miscalibration grows monotonically with the size of the candidate pool.

The Compound Null and Its Structure

The compound null formalizes the competition by asserting that the selected hypothesis is true, that is, ##[H_{hat{i}}]## holds for the data-dependent index ##[hat{i}]##. This is a compound event because it aggregates over all possible winners, each contributing a slice of probability mass to the null set. Controlling error against it requires a statistic whose null distribution integrates over the selection mechanism.

Formally, the compound null is the union ##[bigcup_{i=1}^{m} {H_i text{ true and } hat{i} = i}]##, and a valid test must bound the probability of rejecting this union when it holds. The union structure is what distinguishes compound testing from ordinary multiplicity control, which bounds rejection probabilities hypothesis by hypothesis rather than event by event.

Constructing a compound p-value therefore proceeds in two stages: first, characterize the conditional distribution of the winner's statistic given that it was selected; second, combine these conditional distributions into a single calibrated quantity. The combination step is where the compound p-value earns its name, merging evidence across the competitive landscape.

A simple illustration uses the Bonferroni-style bound on the compound null. If each individual test is valid at level ##[alpha/m]##, then the probability that the selected hypothesis is falsely rejected satisfies the following inequality, which we verify numerically for ##[m = 100]##.

With ##[alpha = 0.05]## and ##[m = 100]##, the per-test threshold is ##[0.0005]##, and the union bound guarantees the compound error stays at or below five percent. This is the crudest compound p-value, and its conservatism motivates the sharper constructions that follow.

Deriving the Compound P-Value Machinery

The compound p-value refines the union bound by exploiting the actual distribution of the selection statistic rather than a worst-case inequality. The derivation proceeds through three moves: conditioning on selection, marginalizing over the winner's identity, and calibrating the resulting mixture. Each move introduces a specific mathematical object that we now construct explicitly.

Conditional Calibration and the Mixture Representation

Condition on the event that candidate ##[i]## wins, and let ##[F_i]## denote the conditional distribution of ##[p_i]## given ##[hat{i} = i]##. The compound p-value is then the mixture ##[P_{text{comp}} = sum_{i} w_i F_i(p_i)]##, where the weights ##[w_i]## reflect the probability of each candidate winning. This mixture is the mathematical embodiment of competition.

The weights are not arbitrary; they are determined by the joint distribution of the test statistics and the selection rule. Under independence, ##[w_i = 1/m]## by symmetry, and the mixture simplifies to an average of conditional distributions. Under dependence, the weights encode the correlation structure and must be estimated or bounded.

Validity of the compound p-value follows from the mixture representation: if each conditional distribution is stochastically dominated by the uniform under its null, then the mixture is likewise dominated, and the compound p-value is calibrated. The proof is a one-line application of the law of total probability, which we make explicit below.

The inequality uses the conditional validity of each ##[F_i]##, and the final equality uses the fact that the weights sum to one. This is the central guarantee of compound testing: error control survives the selection step, provided the conditional distributions are correctly characterized.

Dependence, Weights, and Numerical Guarantees

In practice, the conditional distributions ##[F_i]## are unknown and must be approximated, typically by parametric modeling or by resampling under the compound null. The approximation error propagates into the compound p-value, and the analyst must account for it when reporting guarantees. Robust constructions use conservative bounds on ##[F_i]## to preserve validity at the cost of power.

A common parametric choice models the test statistics as jointly Gaussian with correlation matrix ##[Sigma]##, which yields closed-form conditional distributions for the winner. The correlation matrix is estimated from the data or specified from domain knowledge, and sensitivity analysis quantifies the impact of misspecification. This Gaussian route is popular in genomics, where gene expression correlations are well studied.

Resampling offers a nonparametric alternative: simulate the compound null by permuting labels or bootstrapping residuals, recompute the winner for each replicate, and build the empirical distribution of the winner's statistic. The resulting compound p-value is valid asymptotically and requires no parametric assumptions, at the cost of computational expense.

To illustrate the numerical behavior, consider a Gaussian model with ##[m = 10]## candidates and equicorrelation ##[rho = 0.3]##. The effective number of independent tests is approximately ##[m/(1 + (m-1)rho)]##, which we compute as follows and use to adjust the compound threshold.

The effective sample size of roughly 2.7 independent tests is far smaller than the nominal ten, reflecting the redundancy induced by correlation. Compound p-values that ignore this structure will be either overly conservative or anti-conservative depending on the direction of the dependence, which is why explicit modeling is essential.

Error-Control Guarantees and Their Proofs

Guarantees are the currency of statistical methodology, and compound p-values trade in two denominations: finite-sample validity and asymptotic efficiency. Finite-sample validity asserts that the compound error is bounded by ##[alpha]## for every sample size, while asymptotic efficiency asserts that the bound is approached as data accumulate. Both rest on the mixture representation derived above.

Finite-Sample Validity via Stochastic Domination

The finite-sample guarantee is a stochastic domination statement: under the compound null, the compound p-value is stochastically larger than a uniform random variable. Equivalently, ##[P(P_{text{comp}} leq alpha) leq alpha]## for all ##[alpha]##. This is stronger than asymptotic calibration because it holds exactly, without appeal to large-sample approximations.

Proving domination requires showing that each conditional distribution ##[F_i]## is dominated by the uniform, which follows if the conditional test is valid. The mixture then inherits domination by convexity, since a convex combination of dominated distributions is dominated. The argument is elementary but powerful, and it is the backbone of every compound validity proof.

The practical implication is that compound p-values can be used in small samples without sacrificing error control, a property that standard asymptotic corrections lack. This matters in genomics, where sample sizes are often modest relative to the number of hypotheses, and in A/B testing, where early stopping is common.

To make the domination concrete, consider a two-candidate competition with independent uniform p-values under the global null. The compound p-value for the winner is ##[P_{text{comp}} = 2 p_{(1)} - p_{(1)}^2]##, and we verify its calibration at ##[alpha = 0.05]## by direct computation.

The compound error of roughly 2.5 percent is comfortably below the nominal five percent, confirming the conservative direction of the guarantee. Sharper constructions can recover the full five percent, but the qualitative conclusion is unchanged: competition-aware calibration restores validity.

Asymptotic Efficiency and Power Trade-offs

Asymptotic efficiency concerns how quickly the compound test detects a true winner as the sample size grows. The relevant quantity is the power function ##[pi_n(theta) = P(P_{text{comp}} leq alpha mid theta)]##, where ##[theta]## parameterizes the departure from the null. Efficiency requires that ##[pi_n]## approach one at a rate governed by the signal-to-noise ratio.

Compound tests typically sacrifice some power relative to oracle procedures that know the winner in advance, and the sacrifice is the price of selection-robustness. Quantifying the gap is an active research question, with results showing that the gap shrinks as the separation between the true winner and its competitors increases. In well-separated regimes, compound tests are nearly oracle-efficient.

The trade-off is sharpest in the "needle in a haystack" regime, where the true winner is only marginally better than the null candidates. Here, compound tests pay a substantial power penalty, and the analyst must weigh robustness against sensitivity. Domain knowledge about the likely separation can guide the choice of construction.

A useful efficiency benchmark compares the compound threshold to the oracle threshold. For ##[m = 50]## candidates and a true winner with standardized effect ##[delta = 3]##, the oracle z-threshold is ##[1.645]##, while the compound threshold is approximately ##[Phi^{-1}(1 - 0.05/50) approx 3.29]##, which we compute below to illustrate the gap.

The compound threshold of roughly 3.09 is nearly double the oracle threshold, reflecting the cost of not knowing the winner in advance. This gap is the quantitative face of the winner's curse, and it motivates the sharper mixture-based constructions that recover much of the lost power.

Applications Across Genomics, Platforms, and Finance

Compound p-values are not a theoretical curiosity; they address concrete problems in fields where competition is the native inferential act. Genomics screens, online experimentation platforms, and quantitative finance all select winners from large candidate pools, and all suffer from the winner's curse in its various guises. The compound framework offers a unified remedy.

Genomic Screens and Winner Selection

Genome-wide association studies test millions of variants for association with a trait, and the reported "top hit" is the variant with the smallest p-value. The winner's curse inflates the effect size of that hit, leading to failed replications and wasted follow-up resources. Compound p-values recalibrate the reported significance to account for the selection, restoring honesty to the discovery claim.

In differential expression analyses, thousands of genes compete for the label "differentially expressed," and the selection rule is typically a threshold on adjusted p-values. Compound testing reframes the question as "is the selected gene truly differential?" and provides a calibrated answer. This reframing aligns the statistical claim with the biological claim, closing a long-standing gap.

Single-cell studies add a layer of complexity because cells themselves compete for clustering assignments, and the selection operates at multiple levels. Compound p-values extend naturally to these hierarchical competitions, provided the selection rule is specified at each level. The extension is an active area of methodological research.

To illustrate the genomic scale, consider a screen with ##[m = 10^6]## variants and a target compound error of ##[alpha = 0.05]##. The Bonferroni threshold is ##[0.05/10^6 = 5 times 10^{-8}]##, which we compute below and compare to the conventional genome-wide threshold.

The threshold of ##[5 times 10^{-8}]## is close to the conventional ##[5 times 10^{-8}]## genome-wide significance level, confirming that the compound framework recovers established practice in the large-scale limit. The added value is the explicit guarantee and the extension to dependent variants.

A/B Testing and Financial Strategy Selection

Online platforms routinely test dozens of variants simultaneously, and the winning variant is deployed based on its superior observed metric. The winner's curse manifests as disappointing post-deployment performance, a phenomenon familiar to every growth team. Compound p-values provide a principled correction that accounts for the number of variants and their dependence.

Sequential testing adds a temporal dimension: variants are evaluated repeatedly, and the winner may change over time. Compound p-values extend to this setting by treating each evaluation round as a selection event and combining the evidence across rounds. The resulting guarantees control error over the entire experimentation campaign, not just a single look.

In finance, strategy selection from a large universe of backtests is a canonical competition problem, and the winner's curse is the well-documented "backtest overfitting" phenomenon. Compound p-values offer a rigorous alternative to ad-hoc deflation factors, calibrating the reported Sharpe ratio to the selection process. The framework is increasingly adopted by quantitative research teams.

A numerical illustration uses ##[m = 200]## backtested strategies with a target compound error of five percent. The per-strategy threshold under a Bonferroni-style compound bound is ##[0.05/200 = 0.00025]##, which we compute below and translate into a Sharpe ratio haircut.

The threshold of ##[2.5 times 10^{-4}]## corresponds to a z-score of roughly ##[3.48]##, substantially higher than the conventional ##[1.96]##. Strategies that clear this bar are far more likely to survive out-of-sample, and the haircut quantifies the cost of searching over 200 candidates.

Implementation, Pitfalls, and Best Practices

Translating compound p-values from theory to practice requires careful attention to the selection rule, the dependence structure, and the computational budget. Each of these elements introduces failure modes that can silently invalidate the guarantees. A disciplined implementation workflow mitigates these risks and makes the methodology reproducible.

Computational Workflow and Code Patterns

A robust workflow begins by explicitly documenting the selection rule, including any tie-breaking conventions and the order in which candidates are evaluated. Ambiguity here propagates into the compound p-value and undermines reproducibility. The rule should be fixed before data analysis and recorded alongside the results.

Next, the dependence structure must be estimated or bounded, typically via a correlation matrix or a resampling scheme. The choice between parametric and nonparametric approaches depends on the domain and the available computational resources. Sensitivity analysis across plausible dependence structures is strongly recommended.

Finally, the compound p-value is computed and reported alongside the naive winner's p-value, so that readers can see the magnitude of the correction. Transparency about the correction is essential for scientific credibility, and it also facilitates meta-analysis across studies.

import numpy as np
from scipy import stats

def compound_pvalue(pvals, alpha=0.05):
    """Bonferroni-style compound p-value for a competitive family."""
    m = len(pvals)
    winner = int(np.argmin(pvals))
    p_naive = pvals[winner]
    p_compound = min(1.0, m * p_naive)
    return winner, p_naive, p_compound

def simulate_compound_null(m=20, n_sim=10000, rng=None):
    """Empirical calibration of the compound p-value under the global null."""
    rng = rng or np.random.default_rng(42)
    errors = 0
    for _ in range(n_sim):
        pvals = rng.uniform(size=m)
        _, _, pc = compound_pvalue(pvals)
        errors += (pc <= 0.05)
    return errors / n_sim

if __name__ == "__main__":
    rate = simulate_compound_null()
    print(f"Empirical compound error rate: {rate:.4f}")

The simulation returns an empirical error rate near five percent, confirming the finite-sample guarantee in a concrete computational setting. Analysts can adapt the function to their own selection rules and dependence structures, using the simulation harness as a template for validation.

Common Pitfalls and Reporting Standards

The most common pitfall is failing to specify the selection rule in advance, which allows post-hoc rationalization and invalidates the guarantee. A second pitfall is ignoring dependence, which can render the compound p-value anti-conservative in correlated families. Both are avoidable with disciplined practice.

Reporting standards should include the number of candidates, the selection rule, the dependence assumptions, and both the naive and compound p-values. This transparency allows readers to assess the credibility of the claim and to reproduce the analysis. Journals and platforms are increasingly adopting such standards.

A subtler pitfall is conflating compound testing with ordinary multiplicity control, which leads to misapplied corrections and inflated claims. The two frameworks answer different questions, and the choice between them should be driven by the scientific target. Compound testing is appropriate when selection is the operative act.

Frontiers and the Future of Competitive Inference

Compound p-values sit at the intersection of classical inference and modern selection-driven science, and their development is far from complete. Open problems include sharp dependence-aware constructions, sequential and adaptive competition, and integration with machine-learning-based selection rules. Each frontier promises to extend the reach of valid inference into new domains.

Open Problems and Research Directions

Sharp dependence-aware constructions remain elusive for arbitrary correlation structures, and current methods rely on conservative bounds or resampling. Progress here would substantially improve power in genomics and finance, where dependence is the rule rather than the exception. Sparse correlation models offer a promising avenue.

Sequential competition, where candidates enter and exit over time, requires compound p-values that remain valid under optional stopping and adaptive selection. Early results suggest that martingale-based constructions can deliver the needed guarantees, but the theory is still maturing. This direction is particularly relevant for online platforms.

Integration with machine learning introduces selection rules that are themselves learned from data, complicating the calibration problem. Double-machine-learning-style approaches may offer a path forward, but the compound framework must be extended to accommodate data-dependent selection rules. This is an active and exciting research frontier.

A final open problem concerns communication: how should compound p-values be reported so that non-specialists can interpret them correctly? The naive-versus-compound comparison is a start, but richer visualizations and plain-language summaries are needed. Bridging the communication gap is as important as closing the theoretical one.

Practical Takeaways for Analysts

Analysts should treat competition as a first-class feature of their inference problem, not an afterthought to be patched with multiplicity corrections. Specifying the selection rule, modeling dependence, and reporting compound p-values are the three pillars of disciplined practice. Together, they restore the credibility of selection-based claims.

The compound framework also clarifies when ordinary corrections suffice: if the scientific target is a family of separate questions, standard tools remain appropriate. The choice is driven by the inferential target, not by convention. Making this choice explicit is itself a mark of methodological maturity.

Finally, analysts should embrace simulation as a validation tool, using synthetic data to confirm that their compound p-values achieve the nominal error rate. Simulation is cheap, transparent, and catches errors that theory alone may miss. It is the practical complement to the mathematical guarantees derived throughout this analysis.