Can calibrated AI predict adverse event seriousness from FAERS reports?

Predicting a reporter's seriousness label, not clinical severity

The paper asks whether machine learning can flag which adverse event reports for GLP-1 receptor agonists are serious. Seriousness here is defined by FAERS outcome codes: a report counts as serious if it carries at least one code such as death, hospitalization, or a life-threatening event. Each observation is a case–molecule pair, so one case can contribute more than one observation if several GLP-1 molecules are recorded.

That definition matters for reading every result. The models learn to reproduce how reporters and coders classified seriousness in the database; they do not estimate how often an event actually occurs, and the paper states that FAERS cannot provide incidence estimates. A high score therefore means a report resembles serious reports in the data, not that a patient faces a measured risk.

A 242,312-observation cohort built from free-text drug names

The analysis used FAERS quarterly data from January 2015 to March 2025. Because FAERS drug names are free text, the authors mapped names to standardized RxNorm ingredients and supplemented this with substring matching, then kept reports where the drug appeared as a primary or secondary suspect. An audit of 400 sampled drug-name mappings found 96.5% precision, and the authors report that mapping errors were rejected by the later GLP-1 filter.

Of 250,093 initially identified cases, 10,224 (4.1%) were dropped by the molecule and exposure-mapping criteria. Those excluded cases were more often serious (52.8% versus 25.1% among the 239,869 retained cases), so the exclusion was not random. A sensitivity analysis that included these cases with common features found negligible effects on held-out logistic-regression and random-forest performance and on random-forest feature rankings, but the primary models remain conditional on successful drug mapping.

Strong ranking and weaker raw probabilities

Discrimination and calibration answer different questions, and the paper separates them. Discrimination, summarized by ROC-AUC, measures whether the model ranks serious reports above non-serious ones; it ranged from 0.877 to 0.930 across models. Calibration measures whether a predicted probability matches the observed share of serious reports at that probability level, summarized by the Brier score.

The models ranked well, but their raw probabilities were not well aligned with observed outcomes. Applying post hoc calibration, Platt scaling or isotonic regression, moved the calibration curves closer to the diagonal of perfect calibration. The random forest with isotonic calibration reached a Brier score of 0.096, and the Platt-scaled version was nearly identical at 0.097. For anyone who wants to read a model output as a probability rather than a ranking, that calibration step is the part that makes the number interpretable.

Frequent gastrointestinal reports were weak predictors

The feature-importance results carry the paper's most distinctive message. Frequently reported gastrointestinal events had low predictive importance for seriousness, even though they are common in the data. In contrast, dosing and administration issues, injection-site events, and pancreatic events were among the strongest predictors. The authors describe this as a disconnect between how often an event is reported and how much it tells a model about reporter-classified seriousness.

The direction of the leading signals differed. In the primary KernelSHAP analysis for the BERT-based model, dosing and administration issues ranked first with a mean absolute SHAP value of 0.063, followed by pancreatic effects at 0.061 and the other adverse-event category at 0.056. The beeswarm plot shows pancreatic events shifting predictions toward serious outcomes, while high values for dosing and administration issues shifted predictions toward non-serious outcomes. These values are in probability units and describe contributions within this model, not clinical risk.

Top predictors shifted across model families

Feature rankings were not identical across models. Logistic regression prioritized reporter type, random forest identified dosing and administration issues as most influential, and the primary language-model SHAP analysis also placed that feature among the leaders, while transformer models highlighted specific molecules such as semaglutide and tirzepatide. The paper cautions that comparing SHAP magnitudes across architectures is difficult because models distribute importance differently.

Even within the language model, the exact order was unstable. Repeated KernelSHAP runs with different background and perturbation seeds produced high overall rank concordance, with pairwise Spearman correlations of 0.965 to 0.979, but the ordering of leading features varied. The authors therefore advise against reading small differences in mean absolute SHAP values or exact ranks as substantively meaningful. The robust claim is the broad grouping of influential features, not a precise leaderboard.

Internal validation leaves transportability untested

Several limits bound how far the results travel. Age was missing in 47.6% of analytic observations and median-imputed for the logistic-regression and random-forest models; an age-exclusion sensitivity analysis produced only minor changes in discrimination and calibration, but the missingness remains inherent to FAERS. Indications were coarse, with 63.5% coded as other, which limits disease-specific subgroup analysis and means molecule-specific SHAP values should not be read as comparative safety effects between molecules.

The binary seriousness outcome is itself an imperfect proxy that may misclassify clinically meaningful events, and the consumer reporter category mixes routine patient reports with litigation-motivated filings. Most importantly, no external validation was performed. The observed discrimination, calibration, and feature-importance patterns are internally validated within FAERS and may not generalize to other reporting systems, populations, coding practices, or regulatory settings; the authors name an independent source such as VigiBase as a necessary next step.

Calibration is the prerequisite for operational use

The paper's practical priority follows from the comparison between ranking and probability. All models discriminated strongly, so a high ROC-AUC alone would not distinguish them for a use case that needs an interpretable probability of reporter-classified seriousness. The calibration step is what changed raw outputs into probabilities close to observed outcome rates, with the isotonic-calibrated random forest achieving the best Brier score of 0.096.

The second priority is to treat feature importance as a guide to context rather than to reporting frequency. Dosing and administration, injection-site, and pancreatic categories carried more signal about seriousness than the frequently reported gastrointestinal events, and the direction of those contributions differed. Before any regulatory or operational use, the paper's own threshold is external validation and recalibration in an independent pharmacovigilance database, because the current evidence is confined to FAERS reports on GLP-1 receptor agonists from 2015 to March 2025.

Sources used in this answer