[Security Audit] Real Money, Fake Models: Why Shadow APIs Break LLM Reproducibility, Safety, and Trust

Executive Summary

TL;DR

Shadow APIs sell a seductive promise: unofficial access to frontier models at lower cost and without regional restrictions, while claiming equivalence to official APIs. This paper shows that the promise is frequently false. Across utility benchmarks, safety stress tests, and model verification probes, shadow APIs often diverge sharply from official endpoints, including near-50% collapses on high-stakes medical and legal tasks and widespread identity mismatches under fingerprinting.

Background Positioning

This is not a “new model” paper. It is an infrastructure integrity paper for the LLM era. Its real contribution is to move the conversation from anecdotal suspicion to systematic evidence: shadow APIs are not merely gray-market distribution channels, but an under-audited supply-chain layer that can silently corrupt experiments, safety claims, and product behavior.

Problem & Motivation

The paper starts from a practical reality: many researchers and developers cannot easily access official frontier APIs because of:

• high pricing,
• payment frictions,
• geographic restrictions,
• and resale ecosystems that offer seemingly compatible alternatives.

That market gap creates demand for shadow APIs: third-party endpoints that claim to serve official models indirectly. The problem is that the user sees only the interface, not the backend. If the endpoint says gpt-5 or gemini-2.5-flash, most users naturally assume model equivalence.

But API compatibility is not model equivalence.

That distinction is the entire paper.

Why this is a hard problem

There are three intertwined difficulties:

• Black-box opacity: users cannot inspect the deployed model or upstream routing.
• Behavioral non-identifiability: a substituted model may look plausible on casual prompting.
• Dynamic infrastructure: providers can silently swap upstream backends, prompt wrappers, or safety settings over time.

This means the failure mode is subtle. A shadow API may pass superficial checks, answer many easy prompts correctly, and still be scientifically unusable because it diverges exactly where robustness, reasoning, or safety matter most.

The core research intuition

The authors’ insight is that shadow APIs should be audited like untrusted model supply chains, not judged only by anecdotal usefulness. If a service really serves the same model, three things should align with the official endpoint:

• task performance,
• safety behavior,
• model identity signals.

If any of these drift systematically, the “same model” claim becomes doubtful.

Methodology - The Core

The paper answers three questions:

1. What shadow APIs exist and how widely are they used?
2. Do they behave consistently with official APIs?
3. Can model verification provide direct evidence of deception?

1. Mapping the shadow API ecosystem

The authors first identify 17 shadow APIs by mining papers and code repositories, beginning with ACL 2024 and ICLR 2024 projects and then expanding via GitHub endpoint reuse.

The scale is already nontrivial:

• 17 shadow APIs identified,
• used in 187 research papers,
• with the most popular one linked to 5,966 citations and 58,639 GitHub stars.

This matters because the paper is not auditing a niche curiosity. It is auditing infrastructure that has already entered the academic pipeline.

A second important ecosystem finding is governance weakness:

• 15 of 17 providers lack transparent identity or verifiable provenance.
• Only 1 provider has valid ICP registration.
• Many are built on OneAPI or NewAPI-style aggregation stacks, which simplify routing, resale, and secondary redistribution.

The technical takeaway is simple: the market structure itself incentivizes indirection and opacity.

2. Utility evaluation: can shadow APIs actually replace official ones?

The benchmarking design is reasonably strong for a black-box audit.

Model families

They evaluate across three major families:

• OpenAI: GPT-4o-mini, GPT-5, GPT-5-mini
• Google: Gemini-2.0-flash, Gemini-2.5-flash, Gemini-2.5-pro
• DeepSeek: DeepSeek-Chat, DeepSeek-Reasoner

Shadow providers

They choose three representative providers:

• Shadow API A
• Shadow API E
• Shadow API H

These are selected for popularity, accessibility, and model-family coverage.

Benchmarks

They split evaluation into:

• Science / reasoning: AIME 2025, GPQA
• Sensitive domains: MedQA, LegalBench
• Safety: JailbreakBench, AdvBench under multiple jailbreak attacks

This benchmark choice is well-motivated. If substitution exists, it should be especially visible on:

• reasoning-heavy tasks,
• specialized high-stakes knowledge,
• and safety-sensitive adversarial settings.

3. Verification: from indirect evidence to identity checks

Utility gaps alone do not prove substitution; they only show mismatch. So the paper adds direct model verification.

LLMmap fingerprinting

LLMmap is an active fingerprinting method. It queries the endpoint with curated probes and compares outputs against a model reference database using cosine distance.

Intuition:

• if the claimed backend is authentic, the response pattern should cluster near the official model;
• if not, it may align more with another model family or cheaper alternative.

Model Equality Testing (MET)

MET asks a distributional question:

• are outputs from the shadow endpoint statistically indistinguishable from outputs of the official endpoint?

This is a complementary test. LLMmap tries to identify which model it resembles; MET tests whether the output distribution matches the claimed one.

Metadata analysis

The authors also inspect:

• inference latency,
• token counts,
• and run-to-run variance.

This is a weaker but useful signal: unstable routing or backend switching often leaks through timing and output-length irregularities.

Why the method is convincing

What makes the methodology compelling is not any single tool, but the triangulation:

• performance mismatch says “the endpoint is not behaviorally faithful,”
• fingerprinting says “the endpoint may not even be the claimed model,”
• MET says “its output distribution is statistically different,”
• metadata irregularity says “the service infrastructure is unstable.”

That is a strong black-box audit pattern. In security terms, the paper does not rely on one indicator that could be dismissed as noise.

Experiments & Results

1. Utility gaps are not small noise; they are structural

The official APIs generally define the upper bound, while shadow APIs often show larger variance and degraded reasoning performance. Shadow API E is relatively stable in some science benchmarks, but APIs A and H frequently show larger divergences, especially on reasoning models.

A notable pattern is reasoning collapse: when the claimed model is supposed to have stronger deliberative or “thinking” capability, shadow versions often lose exactly that edge.

The most alarming result: high-stakes task collapse

The most damaging evidence comes from medicine and law.

For Gemini-2.5-flash:

• official MedQA accuracy: 83.82%
• shadow API average: 36.95%
• deficit: 46.51%–47.21%

On LegalBench:

• all shadow APIs trail official endpoints by 40.10%–42.73%

These are not marginal regressions. They indicate a service that cannot be treated as a faithful proxy in any serious workflow.

Why this matters more than leaderboard noise

A 2–3% benchmark fluctuation might be tolerable in ordinary evaluation. A 40%+ drop in medicine or law is qualitatively different:

• it changes conclusions,
• invalidates comparisons,
• and can invert which system appears safer or more capable.

This is the key reproducibility argument of the paper: if one lab uses the official API and another uses a shadow API under the same model name, they may not be evaluating the same system at all.

Concrete failure examples

The failure cases are especially revealing because they are semantically meaningful, not just numerically wrong.

• In MedQA, shadow APIs confuse HIV confirmatory testing with viral genotyping.
• In LegalBench, they mis-handle Rule 606(b), confusing admissibility with unrelated legal standards.

These are precisely the kinds of mistakes that surface when a model loses domain-specific reasoning fidelity or when the backend is not the claimed one.

2. Safety behavior is unpredictably inconsistent

The safety results are conceptually important. A user might tolerate small utility drift but still hope safety behavior remains aligned with official models. The paper shows that assumption is unsafe.

Examples:

• For GPT-5-mini under Base64 on JailbreakBench, shadow API A reaches a harmfulness score of 0.04 versus 0.02 for the official API — roughly 2× worse.
• For Gemini-2.5-flash under FlipAttack, all shadow APIs are around 0.67–0.68 while the official reaches 0.90, a gap of about 0.23.

This asymmetry is interesting. Shadow APIs are not uniformly less safe or more safe; they are unpredictably different. That is worse for evaluation.

Why? Because safety benchmarking depends on behavioral fidelity. If the proxy endpoint makes the model appear either safer or more vulnerable than the official one, then:

• the benchmark result is not portable,
• the defense evaluation is not valid,
• and comparative safety claims become suspect.

3. Fingerprinting gives direct evidence of substitution

The strongest evidence arrives in Section 5.

Across 24 evaluated endpoints:

• 45.83% fail fingerprint verification outright,
• another 12.50% show substantial cosine-distance deviation.

That means a majority are either directly inconsistent or suspiciously distant from the official baseline.

The substitutions are not random. They follow understandable economic incentives:

• premium proprietary models resembling cheaper open-source or domestic alternatives,
• reasoning models replaced by non-reasoning variants,
• version upgrades or downgrades masquerading under older or different names.

Examples from the table include:

• GPT-5 in shadow APIs A and E fingerprinting like glm-4-9b-chat
• GPT-4o-mini in shadow API H drifting toward Qwen2.5-7B
• DeepSeek-Reasoner in A and H behaving like DeepSeek-Chat

This is the heart of the paper’s title: “fake models” does not mean hallucinated accusations; it means measured response patterns inconsistent with the model being sold.

4. MET supports, but does not duplicate, fingerprinting

MET agrees with LLMmap in 74.1% of cases, with Cohen’s κ = 0.512. That level of agreement is meaningful because the methods are testing different aspects:

• LLMmap: identity similarity
• MET: distributional equivalence

Their partial disagreement is actually informative. It reveals a deeper point:

Identity-consistent serving does not guarantee behavioral fidelity.

The paper’s most interesting anomaly is Gemini-2.5-flash. It often passes identity checks at the family level, yet still collapses badly on sensitive-domain accuracy and diverges in safety behavior.

This suggests another failure mode beyond substitution:

• altered inference parameters,
• prompt preprocessing,
• context truncation,
• hidden middleware,
• or routing policies that preserve the “family signature” but distort task behavior.

That is a subtle and important insight. Verification of model identity is necessary, but not sufficient.

Critical Analysis & Conclusion

What the paper really contributes

The surface contribution is an audit of shadow APIs. The deeper contribution is conceptual:

1. It turns API choice into a reproducibility variable

In LLM papers, authors often report the model name and maybe the date, but not always the exact endpoint provenance. This paper shows that endpoint provenance is a first-order experimental variable.

A paper claiming “we used GPT-5” is incomplete if the actual endpoint was a shadow reseller whose backend may have been:

• substituted,
• misconfigured,
• or dynamically routed.

2. It exposes a new kind of benchmark contamination

There is a quiet failure mode in contemporary LLM research: two papers can use the same nominal model string and still evaluate different systems. This contaminates:

• benchmark comparisons,
• ablation validity,
• safety claims,
• and replication studies.

3. It shows that price is not a fidelity signal

The regression analysis finds no reliable predictive power for price ratio. More expensive shadow APIs are not reliably better; cheaper ones are not predictably worse. That is exactly what one would expect in a market dominated by information asymmetry rather than quality guarantees.

Why the method works, intuitively

The paper’s success comes from probing invariants that should hold if a service is genuine.

A faithful proxy should preserve:

• capability profile across benchmarks,
• safety profile under adversarial prompting,
• identity signature under active fingerprinting,
• operational regularity in latency and tokenization patterns.

Shadow APIs often fail multiple invariants at once. That multi-axis inconsistency is much harder to explain away as randomness.

Limitations

The paper is careful about several limitations, and they are real:

• Snapshot nature: the market changes rapidly, so results are time-bounded.
• No ground-truth backend access: substitution is inferred from black-box evidence, not internal logs.
• Partial provider coverage: 17 providers and 3 deeply audited ones do not exhaust the ecosystem.
• Family-specific blind spots: some behaviors may be caused by middleware or configuration drift rather than literal model replacement.

Still, these limitations weaken exact attribution more than they weaken the main conclusion. Even if some failures come from misconfiguration rather than explicit substitution, the endpoint is still not a faithful replacement for the official API.

Future Work

This paper opens several concrete directions:

• Longitudinal monitoring of shadow APIs to measure backend drift over time.
• Cryptographic or hardware-backed attestation for commercial model serving.
• Verification endpoints from official providers for identity checking.
• Conference reproducibility policy requiring endpoint disclosure and provenance reporting.
• Extension to multimodal APIs and agent systems, where hidden middleware may introduce even larger behavioral drift.

Practical Takeaway

For practitioners and researchers, the message is straightforward:

• Do not assume an OpenAI-compatible endpoint is model-equivalent.
• If you cannot use official APIs, treat third-party endpoints as untrusted infrastructure.
• Report endpoint URL, access date, model version, and verification evidence.
• If the endpoint fails fingerprinting, MET, or stability checks, do not present results as representative of the claimed official model.

Final Verdict

This is a timely and important paper because it identifies a failure mode that the LLM community has mostly ignored: silent model provenance corruption through third-party API layers. Its strongest value is not only empirical but methodological. It shows how to audit opaque model-serving ecosystems with enough rigor to make reproducibility claims falsifiable.

The broader lesson is bigger than shadow APIs: in the LLM era, model identity is part of the experimental setup. If we do not verify the serving layer, we are no longer doing controlled science—we are benchmarking a rumor.