Why multi-agent systems create a new kind of 'black box' problem
When AI agents work together, the overall system can become far harder to understand than any single agent. A 2025 study on multi-agent radiology systems calls this 'compound opacity'—layers of inscrutability that arise from interactions between specialized agents coordinating across tasks like image analysis, report generation, and treatment planning [4]. Traditional explainable AI methods, which work for isolated predictions, fail when applied to these multi-step reasoning chains [4].
This is not just a theoretical concern. In telecommunications simulations, researchers found that even when individual agents were well-designed, combinations of different 'personas' (e.g., agents with varying coding styles or planning orientations) led to persistent vulnerabilities like policy drift and variability in safety metrics [1]. The study tested 32 persona sets across multiple runs and found that while some metrics improved with iteration, others remained unstable [1].
Who is responsible when a multi-agent system fails?
A central risk highlighted across multiple studies is the diffusion of accountability. When decisions emerge from the collaboration of several autonomous agents, it becomes unclear who—or what—is responsible for errors. A multi-expert analysis published in 2025 explicitly identifies 'issues of attribution and shared accountability in decision-making' as a critical challenge for agentic systems [2]. The paper calls for 'robust governance frameworks' and 'transparent accountability mechanisms' to address this [2].
This concern is echoed in research on AI agents in scientific work, where authors warn of 'responsibility gaps' that could lead to poor policy decisions based on erroneous or biased AI outputs [5]. They propose designating an 'AI validator expert' or 'AI guarantor' to oversee integrity, but note that such roles are not yet standard practice [5]. A 2025 chapter on agentic AI safety further emphasizes that governance challenges require global coordination and public engagement, suggesting current oversight is insufficient for the pace of development [3].
The autonomy-transparency paradox: more capable agents are harder to trust
There is a direct tension between making AI agents more autonomous and keeping them transparent enough for safe use. The radiology study describes this as the 'autonomy-transparency paradox'—increasing AI capability directly conflicts with the interpretability required for clinical trust and regulatory oversight [4]. This means that as multi-agent systems become more powerful, they may also become riskier in high-stakes settings like healthcare.
However, the evidence also points to potential mitigations. The telecom study found that careful design of agent personas and planning orientation could improve safety metrics, with analyzer penalties and allocator-coder consistency showing progressive improvements across iterative runs [1]. This suggests that some risks can be managed through better system design, but the same study also found that vulnerabilities persisted under specific persona combinations, indicating that no single fix eliminates all risks [1].
About These Sources
This answer is built on 5 peer-reviewed studies — published from 2025 to 2026, 5 from 2024 or later, collectively cited 63 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 58 papers retrieved from a database of over 500 million.
Sources used in this answer
Safety and Risk Pathways in Cooperative Generative Multi-Agent Systems: A Telecom Perspective
In controlled simulations of telecom multi-agent systems with 32 persona sets, the study found that while some safety metrics improved over iterative runs, persistent vulnerabilities like policy drift and variability remained under specific persona combinations [1].
AI Agents and Agentic Systems: A Multi-Expert Analysis
A multi-expert analysis identifies critical challenges for agentic systems including attribution and shared accountability in decision-making, and calls for robust governance frameworks and transparent accountability mechanisms [2].
The Challenges of Agentic AI Safety
This chapter on agentic AI safety highlights risks including unintended optimization, deceptive alignment, and value drift, and argues that governance requires global coordination and public engagement [3].
Beyond Single Systems: How Multi-Agent AI Is Reshaping Ethics in Radiology.
In radiology, multi-agent AI creates 'compound opacity'—layers of inscrutability from agent interactions—and an 'autonomy-transparency paradox' where increased capability conflicts with interpretability needs [4].
Benefits and Risks of Using AI Agents in Research.
Risks of using AI agents in research include responsibility gaps, deskilling of researchers, and unverifiable AI-generated knowledge; the paper proposes designating an AI validator expert to oversee integrity [5].