Observational evidence is worth less than randomized evidence
Standard EVSI quantifies the expected economic benefit of collecting more data to reduce parameter uncertainty in a health economic model, and most existing applications assume that future data arrive through an idealized mechanism such as a randomized controlled trial. In practice, much additional evidence comes from observational studies, which are cheaper and more feasible but are affected by confounding, selection bias, and population imbalance. The paper's answer is that you cannot simply plug observational data into the usual EVSI machinery: the naive summary statistic of observational data does not generally target the same estimand as a randomized design, so causal adjustment must come first.
The paper's headline finding is that EVSI computed from observational data, even after adjustment, is lower than EVSI based on randomized data, and that the loss in EVSI increases with the level of induced confounding. This is a statement about the value of information, not about whether the treatment works: the same nominal sample size can carry less decision-relevant information once covariate overlap between treated and control groups deteriorates.
Two mechanisms drain value from observational data
The paper identifies two key drivers of information loss when EVSI is computed from observational data. First, stochastic confounding-induced imbalance reduces covariate overlap when the true level of confounding is unknown a priori, which the authors describe as entirely realistic in practice. Second, adjustment procedures such as inverse probability weighting inflate variability and reduce the effective sample size relative to the nominal sample size.
These two drivers matter separately. Even when the adjustment procedure consistently targets the randomized estimand, the informational content of the observational study remains lower than that of a comparable randomized design. The paper stresses that the reduction in EVSI occurs even when the nominal sample size stays constant, so the value of additional information depends critically on covariate overlap and not only on how many individuals are enrolled.
Confounded data are built from a synthetic randomized trial
The method starts by generating individual-level data from a hypothetical randomized trial, or a similar idealized study, to preserve the relationship between the data-generating mechanism and the true parameter values. The joint distribution of covariates, exposure, and outcome is factorized into sequential conditional distributions, and the parameter set must include at least one of the underlying health economic model's parameters, which is the parameter informed by future data collection.
Confounding is then induced through stochastic population imbalance. The investigator specifies a probability that an individual with a given covariate value is discarded from the dataset, with the direction and functional form chosen to reflect the expected treatment allocation mechanism in the clinical setting of interest. Individuals are removed until a pre-specified number has been discarded from each treatment group. Outcomes are not altered, but because they are related to the covariate, confounding arises as the treatment groups come to differ in baseline characteristics. The strength of confounding is controlled by modifying the functional form of the discard probability and by varying how many people are removed.
IPW adjustment is followed by regression-based EVSI
After observational data are generated, the paper applies inverse probability weighting to obtain an adjusted summary statistic of the data targeting the corresponding randomized estimand. EVSI is then computed using a regression-based approach, in which nonparametric regression equations, one for each possible treatment, are fitted to approximate the preposterior mean net monetary benefit. The summary statistic fed into the regression should ideally be sufficient for the parameters informed by the data; when sufficient statistics are not obtainable, sample estimators or maximum likelihood estimators are reliable alternatives.
The degree of induced confounding is quantified using the population Mahalanobis distance between the treatment and control groups with respect to the confounding covariates. Because the selection mechanism induces confounding through systematic imbalance in those covariates, the Mahalanobis distance serves as a convenient summary measure of the resulting multivariate imbalance.
Two health economic models show the same direction of loss
The methodology is applied to two health economic models: a Normal-Normal conjugate model and a chemotherapy treatment model combining a decision tree and a Markov structure. In both, EVSI computed from observational data, even after adjustment, is lower than EVSI based on randomized data, and the loss increases with the level of induced confounding.
The paper also proposes a computationally efficient method to determine the sample size required to recover the EVSI achievable under an idealized randomized design. Standard sample size adjustment based on effective sample size calculation is insufficient to recover the value of information attainable under randomization. To address this, the authors integrate a variance-matching approach that identifies the observational sample size needed to replicate the variance of the randomized estimator, thereby restoring the original EVSI without repeatedly recomputing the full regression-based procedure.
The framework assumes the confounding structure is specified
The method requires the investigator to specify which covariate drives the imbalance, the direction of its relationship with treatment assignment, and the functional form of the discard probability. These choices may be informed by clinical expertise, previous observational studies, or expert elicitation, and when substantial uncertainty exists about the expected degree of confounding, the framework can be applied across a range of plausible imbalance scenarios. The paper notes that the approach can be extended to multiple covariates using dimensionality reduction techniques such as principal component analysis or factor analysis, or by discarding individuals in extreme regions of the covariate space.
The paper's own discussion points to open questions rather than settled answers. It calls for a more formal theoretical relationship between covariate overlap, effective sample size, and EVSI, and specifically for analytical links between weighting-induced variance inflation and the loss in EVSI. It also proposes investigating alternative adjustment methods, such as matching or doubly robust estimators, and comparing their impact on EVSI. These are stated as future directions, not as results established here.
Plan observational sample size around variance matching, not nominal size
The most decision-relevant takeaway is a comparison the paper makes explicit: nominal sample size is the wrong planning target when future evidence is observational and confounded. Because EVSI falls as population imbalance rises even at constant nominal sample size, and because effective-sample-size adjustment alone does not restore the randomized EVSI, the paper's variance-matching approach is the threshold that matters for sample size planning.
For a reader deciding whether to fund an observational study, the paper's framework implies that the relevant question is not how many individuals to enroll, but how much covariate overlap the design can preserve and how much variance inflation the adjustment will introduce. The paper's own framing is that EVSI reflects not only the quantity of data collected but also the structural properties of the data-generating mechanism, which makes the degree of induced confounding the variable to plan around.
Sources used in this answer
Calculating the Expected Value of Sample Information for Observational Studies affected by Confounding
EVSI computed from observational data, even after inverse probability weighting adjustment, is lower than EVSI based on randomized data, and the loss increases with the level of induced confounding.