How well does the agent collaborate with humans?
Task completion alone doesn't tell you if an AI agent is actually useful in practice. A study of 45 clinicians across nine institutions found that when radiologists used an AI-assisted system for breast cancer screening, false positives dropped by 27% and false negatives by 4%, and diagnosis time fell by 3 minutes per patient [1]. But the key insight is that 91% of clinicians reported higher satisfaction and acceptance of the AI only because the evaluation measured how they interacted with the system, not just whether it classified images correctly [1]. This shows that human-AI interaction quality—like whether the AI explains its reasoning and fits into existing workflows—is a separate, essential evaluation dimension.
Similarly, research on customer service chatbots found that AI agents expressing positive emotions can actually backfire: they evoke positive feelings in customers but also violate expectations, making the interaction feel less effective than when a human expresses the same emotion [2]. This means evaluations must measure user expectations and emotional responses, not just whether the issue was resolved. In scientific research, a bioinformatics copilot system that combines multiple AI agents with human researchers achieved state-of-the-art performance across diverse tasks, but its success depended on a well-designed human-agent interaction mechanism and continuous learning strategies, not just raw task accuracy [8].
Can you see how and why the agent makes decisions?
Without visibility into an AI agent's reasoning and actions, you can't trust its results or hold it accountable. A comprehensive framework for responsible AI metrics distinguishes three types: process metrics (how decisions are made), resource metrics (tools and frameworks used), and product metrics (outputs) [7]. This tripartite approach is especially important for generative AI, where concerns about data privacy and fabricated content are high [7]. The European AI Act proposal sets minimum requirements for explainability of high-risk AI systems, and researchers argue that compliance metrics must be risk-focused, model-agnostic, goal-aware, intelligible, and accessible [9].
For AI research agents, this means evaluations should include measures of explainability—can the agent justify its conclusions in a way a human expert can understand? A study evaluating ChatGPT-generated research articles found that while the AI could produce high-quality text, reviewers expressed serious concerns about ownership and integrity of the research, and the AI struggled most with developing the literature review and methodology sections [3]. This suggests that evaluations need to assess not just whether the output looks correct, but whether the reasoning process is transparent and reproducible. For IT operations, agentic AI systems that proactively predict and resolve issues require visibility into their decision-making to ensure accountability and avoid unintended consequences [4].
Does the agent actually improve outcomes in the real world?
Task completion in a controlled test environment doesn't guarantee real-world value. The medical AI study showed that the real benefit came from reducing clinician errors and saving time, not just from high classification accuracy [1]. Similarly, in bioinformatics, a multi-agent copilot system not only reproduced a complex data integration process from a seminal study but also uncovered rare cell types and introduced a recursive annotation strategy that captured continuous cellular states—discoveries that went beyond the original task [8]. This demonstrates that evaluations should measure whether the agent enables new insights or improves human decision-making, not just whether it completes assigned tasks.
Ethical considerations are equally important. Research on visibility into AI agents proposes three categories of measures—agent identifiers, real-time monitoring, and activity logging—to ensure accountability across different deployment contexts, from centralized to decentralized systems [6]. These measures help mitigate risks like privacy violations and concentration of power. In virtual reality, an LLM-based AI agent that simulates human behavior with appropriate facial expressions and gestures was evaluated not just on response plausibility but on how well it integrated into the social VR experience [5]. This broader evaluation lens—covering ethical, social, and practical impacts—is essential for AI research agents that will be deployed in sensitive domains like healthcare, education, or scientific discovery.
About These Sources
This answer is built on 9 peer-reviewed studies — published from 2022 to 2025, 5 from 2024 or later, 3 in Q1 journals, collectively cited 650 times — selected as the most relevant from 9 studies that passed quality screening, drawn from 59 papers retrieved from a database of over 500 million.
Sources used in this answer
BreastScreening-AI: Evaluating medical intelligent agents for human-AI interactions
In a study of 45 clinicians from nine institutions, an AI-assisted breast cancer screening system reduced false positives by 27% and false negatives by 4%, cut diagnosis time by 3 minutes per patient, and achieved 91% clinician satisfaction—showing that human-AI interaction quality is a critical evaluation dimension beyond task accuracy.
Bots with Feelings: Should AI Agents Express Positive Emotion in Customer Service?
Controlled experiments showed that AI agents expressing positive emotion in customer service can both evoke positive customer feelings and violate expectations, making them less effective than human agents expressing the same emotion—highlighting the need to evaluate emotional and expectation-based effects.
The Potential and Concerns of Using AI in Scientific Research: ChatGPT Performance Evaluation
Reviewers evaluating ChatGPT-generated research articles expressed concerns about ownership and integrity, and found the AI weakest in developing literature reviews and methodology—indicating that evaluations must assess reasoning transparency and research quality, not just output plausibility.
Agentic AI in Predictive AIOps: Enhancing IT Autonomy and Performance
Agentic AI in IT operations enhances autonomy and performance by proactively predicting and resolving system issues, but requires evaluation of decision-making transparency and accountability to avoid unintended consequences in complex environments.
Building LLM-based AI Agents in Social Virtual Reality
An LLM-based AI agent in social VR was evaluated on response plausibility and integration of facial expressions and gestures, showing that social and contextual appropriateness are important metrics beyond task completion for embodied agents.
Visibility into AI Agents
Proposes three categories of visibility measures—agent identifiers, real-time monitoring, and activity logging—to ensure accountability across centralized and decentralized AI agent deployments, addressing risks like privacy violations and power concentration.
Towards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI Accountability
Introduces a comprehensive metrics catalogue for AI accountability with three types: process metrics (procedural integrity), resource metrics (tools and frameworks), and product metrics (outputs), emphasizing the need for granular evaluation especially for generative AI.
A data-intelligence-intensive bioinformatics copilot system for large-scale omics research and scientific insights.
A multi-agent bioinformatics copilot achieved state-of-the-art performance across diverse tasks and uncovered rare cell types in a lung cell atlas, demonstrating that evaluations should measure whether agents enable new scientific insights, not just complete assigned tasks.
Metrics, Explainability and the European AI Act Proposal
Argues that metrics for AI explainability under the European AI Act must be risk-focused, model-agnostic, goal-aware, intelligible, and accessible—providing a framework for evaluating whether AI systems meet legal transparency requirements.