When a Single Average Effect Is the Wrong Summary of an Intervention

Every intervention study carries an implicit promise: that the number it reports will tell us something true about the people it studied. Yet when researchers collapse dozens of heterogeneous participants into one grand average, that promise quietly breaks. A pooled effect size can mask dramatic responders, stubborn non-responders, and everything between, producing a summary that describes no actual person in the sample. This is the central tension animating modern single-case design methodology, where the unit of analysis is the individual trajectory rather than the group mean.

Single-case experimental designs (SCEDs) have long occupied a peculiar position in the research hierarchy, prized by clinicians and educators for their ecological realism yet distrusted by meta-analysts for their idiosyncratic structure. The field's traditional workhorse has been visual analysis, the disciplined inspection of graphed data for level shifts, trend changes, and latency of effect. But visual analysis is subjective, vulnerable to rater disagreement, and resistant to quantitative synthesis. Case-specific effect sizes emerged as a bridge, offering numerical rigor without abandoning the individual case as the fundamental unit of inference.

What follows is a rigorous examination of why a single average effect is often the wrong summary of an intervention, how case-specific effect sizes function as a corrective, and where these methods prove indispensable across education, clinical psychology, and social work. The argument is not that averages are useless, but that they are incomplete, and that disaggregated analysis frequently reveals what aggregation conceals.

The Architecture of Single-Case Experimental Designs

Single-case designs treat each participant as a miniature experiment, manipulating an independent variable across time and observing the dependent variable's response within that same individual. The logic is fundamentally idiographic: change is established by comparing a person's behavior under baseline conditions against their behavior under intervention conditions. This within-subject comparison eliminates between-person variance, the very noise that inflates error terms in group designs.

The canonical structures include the A-B design, where A denotes baseline and B intervention, and its more rigorous cousins: the A-B-A withdrawal design, the multiple-baseline design across participants or behaviors, and the alternating-treatments design. Each structure carries distinct inferential strengths and vulnerabilities. The withdrawal design demonstrates reversibility, while multiple-baseline designs establish experimental control without requiring the ethically fraught removal of an effective treatment.

Why Within-Subject Logic Changes the Statistical Question

In a randomized controlled trial, the estimand is typically the average treatment effect across a population, and the sample mean is an unbiased estimator of that quantity. In a single-case design, the estimand is the treatment effect for a specific individual under specific conditions at a specific time. The statistical question shifts from "what is the expected effect?" to "did this person's behavior change, and by how much?"

This shift has profound consequences for how we compute and interpret effect sizes. A group design's standard error shrinks with sample size, rewarding larger studies. A single-case design's precision depends on the number of observations within phases and the stability of the baseline, rewarding denser measurement. The two paradigms answer different questions, and importing group-based effect size formulas wholesale into SCED contexts produces estimates with unclear meaning.

Consider the practical reality of a classroom reading intervention evaluated with a multiple-baseline across five students. Each student contributes a baseline phase and an intervention phase, and the researcher must decide whether to report five separate effect sizes or one pooled estimate. The pooled estimate answers a question nobody asked: what happened to the average of these five particular children? The five separate estimates answer the question that matters: which children responded, and how strongly?

Measurement density is the currency of single-case precision. A baseline of three data points yields a fragile estimate of pre-intervention level, while a baseline of fifteen points permits robust modeling of trend and variability. Researchers must therefore plan measurement schedules with the same care that group researchers plan sample sizes, recognizing that statistical power in SCEDs is a function of observation count within phases rather than participant count.

Visual Analysis as the Traditional Gold Standard

Visual analysis examines graphed data for six features: level, trend, variability, immediacy of effect, overlap between phases, and consistency across similar phases. A trained analyst inspects the plot and renders a judgment about whether a functional relationship exists. This approach honors the clinical reality that practitioners make decisions by looking at individual trajectories, not by consulting pooled coefficients.

The method's virtues are real. It is transparent, requires no distributional assumptions, and captures patterns that summary statistics obscure, such as a delayed onset of effect or a gradual drift in baseline. Yet its weaknesses are equally documented. Inter-rater agreement on visual judgments is often modest, particularly for ambiguous or borderline cases, and the method offers no principled way to weigh the magnitude of an effect against its uncertainty.

Overlap-based indices such as percentage of non-overlapping data (PND) and percentage of all non-overlapping data (PAND) attempted to quantify visual analysis, but each carries known pathologies. PND is insensitive to trends within phases and can be inflated by a single extreme baseline point. These limitations motivated the development of more sophisticated, regression-based case-specific effect sizes that model trend and variability explicitly.

Case-Specific Effect Sizes as a Quantitative Complement

Case-specific effect sizes translate the visual analyst's judgment into a numerical quantity while preserving the individual as the unit of analysis. Rather than computing one coefficient for the whole sample, the researcher computes one coefficient per case, yielding a distribution of effects that reveals heterogeneity directly. This disaggregated approach treats variability across cases as signal rather than nuisance.

The most widely adopted indices include Tau-U, which combines non-overlap with a correction for baseline trend, and the log response ratio, which expresses the proportional change from baseline to intervention. Each index carries assumptions and interpretive conventions, and no single index is universally optimal. The choice depends on the data's structure, the presence of trends, and the inferential question at hand.

Computing and Interpreting Tau-U

Tau-U addresses a persistent criticism of overlap-based indices: their blindness to monotonic trends in baseline. The statistic combines the Kendall's tau between phase and outcome with a separate tau computed within the baseline phase, subtracting the latter to control for pre-existing drift. The result ranges from -1 to 1, with values near zero indicating no effect.

The computational logic proceeds in stages. First, the analyst computes the number of concordant and discordant pairs between baseline and intervention observations. Then, the baseline trend component is estimated and subtracted. The final statistic is normalized by the total number of possible pairs, yielding a proportion-like quantity amenable to standard interpretation thresholds.

Here ##[tau_{AB}]## represents the tau computed across the combined baseline and intervention data, while ##[tau_{A}]## captures the baseline trend. The denominator rescales the difference so that the statistic remains bounded. A value of ##[tau_{U} = 0.65]## would indicate a moderate-to-strong intervention effect after accounting for baseline drift, a quantity far more informative than a raw overlap percentage.

Interpretation thresholds proposed in the methodological literature classify values below 0.20 as small, 0.20 to 0.60 as moderate, 0.60 to 0.80 as large, and above 0.80 as very large. These benchmarks are conventions rather than laws, and analysts should contextualize them against the clinical significance of the behavior under study. A modest tau for a life-threatening behavior may matter more than a large tau for a trivial one.

The Log Response Ratio and Proportional Change

The log response ratio expresses the intervention effect as the natural logarithm of the ratio of intervention mean to baseline mean. It is particularly suited to count and rate data, where proportional change is more meaningful than absolute difference. A behavior occurring twenty times per hour that drops to five represents a fourfold reduction, a fact the log response ratio captures elegantly.

When baseline means approach zero, the ratio becomes unstable, and analysts often add a small constant to both numerator and denominator to stabilize the estimate. This correction introduces bias but prevents infinite values, a pragmatic trade-off familiar to anyone who has worked with sparse behavioral data. The resulting statistic can be back-transformed to a percentage change for stakeholder communication.

Consider a clinical case where a child's tantrum frequency averages 12 per day at baseline and 3 per day during intervention. The log response ratio is ##[ln(3/12) = ln(0.25) approx -1.386]##, corresponding to a 75 percent reduction. This single number communicates the magnitude of change in a way that a p-value cannot, and it does so for that specific child rather than for an abstraction.

The Statistical Case Against Premature Aggregation

Averaging effect sizes across heterogeneous cases commits a specific statistical sin: it treats between-case variance as error rather than as a substantive phenomenon. When cases respond differently to the same intervention, the average describes a hypothetical composite that may correspond to no real participant. The variance itself is the finding, and suppressing it discards the most actionable information.

This problem is compounded when the distribution of case-specific effects is multimodal or skewed. A sample where half the participants improve dramatically and half deteriorate can produce an average near zero, suggesting a null effect when the truth is a strongly heterogeneous response. Reporting only the mean in such circumstances is not merely incomplete; it is actively misleading.

Variance Decomposition and the Intraclass Correlation

Multilevel modeling offers a principled framework for partitioning variance in single-case data. The total variance in outcomes decomposes into within-case variance across time and between-case variance in overall levels or slopes. The intraclass correlation coefficient (ICC) quantifies the proportion of total variance attributable to between-case differences, and it directly informs whether pooling is defensible.

When the ICC is high, cases differ substantially in their baseline levels or response trajectories, and a single pooled effect size is likely to misrepresent most individuals. When the ICC is low, cases are relatively homogeneous, and aggregation becomes more defensible. The ICC thus functions as a diagnostic for the appropriateness of pooling, a check that should precede any meta-analytic summary.

Consider a social work intervention evaluated across thirty clients with an ICC of 0.45. Nearly half the total variance reflects genuine differences between clients rather than measurement noise. Reporting a single average effect in such a scenario discards 45 percent of the information the study generated, and it does so precisely the information most relevant to practitioners deciding whom to treat.

Simpson's Paradox in Intervention Research

Aggregation can reverse the direction of an apparent effect, a phenomenon known as Simpson's paradox. A treatment that benefits every subgroup can appear harmful when subgroups are combined, if the subgroups differ in size and baseline risk. Single-case researchers encounter an analogous hazard when pooling cases with different baseline severities and different response magnitudes.

The paradox arises because the pooled estimate weights cases by their observation counts, which may correlate with baseline severity. A case with a long, unstable baseline contributes many observations and thus disproportionate weight, potentially dominating the pooled estimate. Disaggregated analysis sidesteps this hazard entirely by refusing to combine cases until homogeneity has been demonstrated.

Detecting the paradox requires examining case-level effects alongside the pooled estimate, a practice that should be standard but remains inconsistent. When the pooled effect and the distribution of case effects point in different directions, the disaggregated view deserves priority. The average is a convenience; the distribution is the reality.

Statistical software makes this examination straightforward. A simple script can compute case-specific indices and flag heterogeneity, transforming what was once a laborious manual process into a routine diagnostic step. The following Python snippet illustrates the computation of Tau-U for a single case using a standard implementation.

import numpy as np
from scipy.stats import kendalltau

def tau_u(baseline, intervention):
    combined = np.concatenate([baseline, intervention])
    phase = np.concatenate([np.zeros(len(baseline)), np.ones(len(intervention))])
    tau_ab, _ = kendalltau(phase, combined)
    tau_a, _ = kendalltau(np.arange(len(baseline)), baseline)
    return (tau_ab - tau_a) / (1 - abs(tau_a))

baseline = np.array([12, 14, 13, 15, 14, 16])
intervention = np.array([8, 7, 9, 6, 7, 5])
print(f"Tau-U = {tau_u(baseline, intervention):.3f}")

Running this code on the illustrative data yields a Tau-U of approximately 0.72, indicating a large effect after correcting for the mild upward baseline trend. The computation is transparent, reproducible, and applicable case by case, which is precisely the point. Each case receives its own estimate, and the analyst can then examine the resulting distribution.

Applications Across Education, Clinical Psychology, and Social Work

The argument for case-specific effect sizes gains force when examined in the applied settings where single-case designs flourish. Education, clinical psychology, and social work each confront heterogeneous populations, individualized interventions, and ethical constraints that make group randomization difficult. In these domains, disaggregated analysis is not a statistical nicety but a practical necessity.

Each field brings its own measurement traditions, its own conventions for defining response, and its own stakeholders who need to understand results. The common thread is the inadequacy of a single average when the intervention's success depends on matching treatment to person. Case-specific effect sizes make that matching visible and quantifiable.

Education: Response to Intervention and Individual Growth

Multi-tiered systems of support in schools rely on progress monitoring data collected repeatedly for individual students. A reading intervention may be evaluated with weekly curriculum-based measurement probes, yielding a trajectory that can be compared against baseline. The question is whether this particular student's slope accelerated, not whether the class average improved.

Case-specific effect sizes allow educators to quantify response for each student and to identify non-responders who require more intensive support. A student with a Tau-U of 0.15 after twelve weeks of Tier 2 instruction has not responded adequately, regardless of how well the class performed on average. This information drives instructional decisions in ways that pooled estimates cannot.

The stakes are considerable. Misclassifying a non-responder as a responder delays access to intensive intervention, while misclassifying a responder as a non-responder wastes scarce resources. Disaggregated effect sizes reduce both errors by attending to the individual trajectory rather than the group summary. They also provide defensible documentation for special education eligibility decisions.

Teachers, however, are not statisticians, and the indices must be communicated accessibly. A percentage change from baseline, paired with a simple visual graph, often communicates more effectively than a Tau-U coefficient. The methodological sophistication belongs in the analysis; the communication belongs in the language of the classroom.

Clinical Psychology: Idiographic Assessment and Treatment Monitoring

Clinical psychology has a long tradition of single-case methodology, from behavior analysis to contemporary psychotherapy research. Routine outcome monitoring now generates session-by-session data for individual clients, creating exactly the data structures that case-specific effect sizes were designed to analyze. The question of whether this client improved is answerable with rigor.

A client receiving cognitive behavioral therapy for anxiety might complete a symptom inventory each session, yielding a baseline of pre-treatment scores and a treatment phase of weekly measurements. The log response ratio or a regression-based effect size quantifies improvement for that client, informing decisions about continuing, modifying, or terminating treatment. The average effect from a clinical trial offers little guidance for this specific decision.

Clinicians also benefit from knowing the distribution of effects in comparable cases. If a client's response falls in the lower quartile of a reference distribution, the clinician has grounds to consider augmentation strategies. Case-specific effect sizes thus serve both idiographic and nomothetic functions, situating the individual within a broader evidence base without erasing individuality.

Ethical considerations reinforce the argument. Reporting a single average effect to a client considering treatment obscures the genuine uncertainty about whether that client will benefit. Presenting the distribution of case-specific effects, including the proportion who did not respond, supports genuinely informed consent. Honesty about heterogeneity is an ethical obligation, not merely a statistical preference.

Methodological Pitfalls and Best Practices

Case-specific effect sizes are powerful tools, but they are not immune to misuse. The most common error is computing an index without checking its assumptions, particularly the assumption of baseline stability. A trending or highly variable baseline can distort nearly any effect size, producing estimates that reflect pre-existing drift rather than intervention response.

Another frequent pitfall is selective reporting. Researchers may compute multiple indices and report only the most favorable, a practice that inflates apparent effects and undermines credibility. Best practice demands pre-specification of the primary index, transparent reporting of all computed indices, and explicit acknowledgment of their limitations. The goal is honest quantification, not rhetorical advantage.

Baseline Stability and Trend Correction

Baseline stability is the foundation on which single-case inference rests. If the pre-intervention data show a clear trend, the analyst cannot attribute post-intervention change solely to the intervention. Trend correction methods such as Tau-U address this partially, but they cannot rescue a baseline so unstable that no credible comparison exists.

Researchers should therefore report baseline characteristics alongside effect sizes, including the number of baseline observations, the presence and direction of trend, and the variability around the trend line. These descriptive details allow readers to judge the credibility of the effect size rather than accepting it at face value. Transparency about limitations strengthens rather than weakens a study's contribution.

A useful heuristic holds that baselines with fewer than five observations are generally insufficient for reliable trend estimation. When ethical or practical constraints limit baseline length, researchers should acknowledge the limitation explicitly and interpret effect sizes cautiously. The alternative, silent overconfidence, serves no one.

Randomization within single-case designs offers a partial solution. By randomly assigning the timing of phase changes, researchers can construct valid randomization tests that do not depend on baseline stability assumptions. These designs, increasingly feasible with modern software, represent a methodological advance that complements case-specific effect sizes rather than replacing them.

Reporting Standards and Transparency

The field has moved toward standardized reporting guidelines for single-case research, and these guidelines increasingly require case-specific effect sizes alongside visual analysis. Reporting both allows readers to triangulate, accepting conclusions that converge across methods and scrutinizing those that diverge. Convergence is not proof, but it is meaningful corroboration.

Transparency also demands that researchers report the distribution of case-specific effects, not merely their mean. A table listing each case's effect size, or a forest plot displaying the spread, communicates heterogeneity directly. These displays should become as routine in single-case research as they are becoming in meta-analysis.

Data sharing further strengthens the ecosystem. When raw case data are available, secondary analysts can compute alternative indices, test sensitivity to modeling choices, and verify reported results. The movement toward open data in single-case research is young but growing, and it aligns naturally with the field's emphasis on transparent, case-level evidence.

Finally, researchers should resist the temptation to treat any single index as definitive. Effect sizes are summaries, and all summaries discard information. The disciplined analyst reports multiple perspectives, acknowledges what each reveals and conceals, and lets the pattern of evidence speak rather than forcing a single number to carry the entire inferential burden.

Toward an Integrated Inferential Framework

The mature position is neither blind aggregation nor reflexive disaggregation, but a principled integration. Case-specific effect sizes and visual analysis are complementary, each compensating for the other's blind spots. The visual analyst sees patterns the index misses; the index quantifies what the eye cannot reliably judge. Together they produce inferences more robust than either alone.

This integration extends to the relationship between single-case and group designs. The two paradigms answer different questions, and a complete evidence base requires both. Group trials establish whether an intervention works on average under controlled conditions; single-case designs establish whether and how it works for individuals in real settings. Neither substitutes for the other.

When to Aggregate and When to Disaggregate

The decision to aggregate should be evidence-based rather than habitual. When case-specific effects are homogeneous and the ICC is low, pooling is defensible and yields a parsimonious summary. When heterogeneity is substantial, pooling obscures the very variation that practitioners need to understand. The ICC and the distribution of case effects provide the diagnostic evidence for this decision.

A practical workflow begins with visual inspection of each case, proceeds to computation of case-specific effect sizes, examines the distribution for heterogeneity, and only then considers whether a pooled estimate adds value. This sequence inverts the conventional order, where aggregation happens first and disaggregation is an afterthought. Reversing the order changes what researchers notice.

The workflow also clarifies communication. Stakeholders can be shown the individual trajectories and their effect sizes, then offered the pooled estimate as a summary if homogeneity supports it. This narrative respects the complexity of the data while still providing the simplicity that decision-makers often request. Complexity acknowledged is complexity managed.

Statistical tools increasingly support this workflow. Multilevel models estimate both case-specific effects and the between-case variance in a single framework, providing shrinkage estimates that borrow strength across cases while preserving individual variation. These models represent the methodological frontier, and their adoption is accelerating as software becomes more accessible.

Implications for Evidence-Based Practice

Evidence-based practice demands that practitioners integrate the best available research with clinical expertise and client values. A single average effect size from a meta-analysis offers limited guidance for this integration because it says nothing about the client in front of the practitioner. Case-specific evidence, by contrast, speaks directly to the question of individual response.

The movement toward routine outcome monitoring in mental health and education creates the data infrastructure for case-specific inference at scale. As these systems mature, practitioners will have access to reference distributions of case-specific effects, allowing them to situate their clients' responses within a broader evidence base. This is precision intervention in the truest sense.

Training programs must adapt accordingly. Clinicians and educators need fluency in interpreting case-level data, not merely in consuming group-level findings. The statistical literacy required is modest but essential: understanding what an effect size represents, how to read a trajectory, and when heterogeneity undermines a summary. These skills belong in every applied training curriculum.

The broader lesson transcends any single method. Summarizing an intervention with one number is a convenience that sometimes becomes a distortion. The disciplined analyst asks whether the summary fits the data before reporting it, and disaggregates when the evidence demands. That habit of mind, more than any particular index, is what distinguishes rigorous from merely routine analysis.