The "I Don't Know" Filter: Enhancing Agentic Reliability in Function Calling

2026-07-04

AI Summary

This work addresses the susceptibility of current language models to hallucination in function calling, which stems from biases in evaluation metrics and can lead to high-risk erroneous decisions. The authors propose a lightweight, trainable uncertainty-aware filter that actively identifies and blocks high-risk uncertain calls without modifying the underlying model. By introducing a novel evaluation metric that accounts for the negative consequences of erroneous calls and integrating it into a reliability validation framework, the method endows agents with the ability to abstain—effectively saying “I don’t know”—when faced with uncertain situations. This approach significantly suppresses harmful hallucinations while preserving task performance, thereby enhancing reliability in real-world deployment scenarios.

Abstract

The language models that underpin agents have seen a rapid rise in performance on function calling benchmarks. However, the metrics used in the training and evaluation of these models often encourage models to make positive claims even when the answer is uncertain, leading to hallucinations. Such hallucinations can be disastrous when language models are trusted to use function calls to make decisions in high stakes applications. To that end, we propose an agent evaluation metric that takes into account the negative outcomes associated with incorrect function calls. Further, to catch hallucinations before they can cause harm, we propose a lightweight trainable filter that can quantify a language model's uncertainty and remove potentially harmful function calls. By training that filter to detect and suppress uncertain function calls without modifying the underlying model, we demonstrate a practical path toward agents that know when to say "I don't know," a property we argue is essential to production reliability.

Problem

• function calling
• hallucination
• uncertainty
• language models
• agent reliability

Innovation

• uncertainty quantification
• function calling
• hallucination mitigation
• agent reliability
• trainable filter