Do tool-using language models have enough real-world evaluation evidence?

Where do tool-using LLMs have the strongest real-world evidence?

The strongest real-world evidence for tool-using LLMs comes from software engineering and scientific chemistry. In a large-scale evaluation of 1,684 JavaScript API functions, the LLM-based tool TestPilot generated unit tests that achieved 70.2% median statement coverage and 52.8% branch coverage, significantly outperforming the state-of-the-art traditional tool Nessie (51.3% and 25.6%, respectively) [1]. This study also found that 92.8% of generated tests had ≤50% similarity to existing tests, meaning they were genuinely novel, not copies. In chemistry, the ChemCrow agent integrated 18 expert-designed tools with GPT-4 and autonomously planned and executed the syntheses of an insect repellent, three organocatalysts, and guided the discovery of a novel chromophore, demonstrating new capabilities beyond the base LLM [4]. These results show that when LLMs are augmented with domain-specific tools, they can deliver measurable, practical improvements in real-world tasks.

How well do tool-using LLMs perform in medical settings?

Medical evaluations show a more mixed picture, with performance highly dependent on the specific task and model. In emergency internal medicine, a prospective study of 73 real patient cases found that OpenAI's o1 model achieved a mean final rating of 3.63 (on a 4-point scale), statistically indistinguishable from human physicians (3.67, p=0.62), and correctly classified all abnormal lab values (100% accuracy) [3]. However, two other models (Claude-3.5-Sonnet and Llama-3.2-70B) scored significantly lower, mainly due to errors in therapy planning. In cervical cytology, a study of 200 images found that while LLMs could distinguish normal from abnormal with high sensitivity (85.4-100%), their performance on specific abnormal categories was poor: ChatGPT o3 detected only 6% of high-grade squamous intraepithelial lesions and 28% of adenocarcinomas [2]. The authors concluded that general-purpose LLMs are not recommended as diagnostic support tools in this area. For writing medical exam questions, only about half (50-57%) of 100 generated multiple-choice questions were deemed usable without modification by expert evaluators [5]. These results highlight that medical applications require careful task selection and model choice.

What factors make tool-using LLMs more or less effective?

Several factors consistently influence real-world performance. First, model size and training matter: in the test generation study, the larger GPT-3.5-turbo achieved 70.2% coverage, while the smaller StarCoder reached only 54.0% [1]. Second, prompting technique is critical: a study of open-source LLMs on medical questions found that advanced prompting strategies (like chain-of-thought) improved accuracy from 74% (basic prompting) to 79% for the best model, outperforming GPT-3.5 [7]. Third, the specific tool integration matters: in chemistry, the 18 expert tools enabled ChemCrow to perform tasks the base LLM could not [4]. Fourth, user experience studies reveal a gap between expectation and reality: in a study of 24 programmers using Copilot, most preferred it for daily tasks because it provided a useful starting point, but they struggled with understanding, editing, and debugging the generated code, which hindered task completion [6]. This suggests that even when tool-using LLMs produce correct outputs, usability challenges remain.

About These Sources

This answer is built on 7 peer-reviewed studies — published from 2022 to 2025, 5 from 2024 or later, 2 in Q1 journals, collectively cited 1,170 times — selected as the most relevant from 8 studies that passed quality screening, drawn from 58 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions