Context-Adaptive Hearing Aid Fitting Advisor through Multi-turn Multimodal LLM Conversation

2025-09-08

AI Summary

Traditional hearing aids rely on static configurations, limiting their adaptability to dynamic acoustic environments and inter-individual hearing variability. To address this, we propose a multi-agent large language model (LLM)-driven framework for real-time, personalized hearing assistance. The framework integrates three modalities: environmental audio features (extracted via a lightweight YAMNet variant achieving 91.2% sound classification accuracy), individual audiograms, and multi-turn conversational feedback—enabling context-aware perception, subproblem decomposition, adaptive strategy generation, and ethical oversight. An LLM-based “Judge” agent ensures safety and regulatory compliance in parameter adjustment decisions. This work represents the first application of a multimodal, multi-turn LLM agent system to hearing aid personalization. Experimental validation demonstrates substantial improvements in contextual understanding and interactive efficiency, confirming the feasibility and practical potential of AI agents for safe, precise, and real-time adaptive hearing assistance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Modeling other Agents
• Humans and AI: AI for Accessibility

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Traditional hearing aids often rely on static fittings that fail to adapt to their dynamic acoustic environments. We propose CAFA, a Context-Adaptive Fitting Advisor that provides personalized, real-time hearing aid adjustments through a multi-agent Large Language Model (LLM) workflow. CAFA combines live ambient audio, audiograms, and user feedback in a multi-turn conversational system. Ambient sound is classified into conversation, noise, or quiet with 91.2% accuracy using a lightweight neural network based on YAMNet embeddings. This system utilizes a modular LLM workflow, comprising context acquisition, subproblem classification, strategy provision, and ethical regulation, and is overseen by an LLM Judge. The workflow translates context and feedback into precise, safe tuning commands. Evaluation confirms that real-time sound classification enhances conversational efficiency. CAFA exemplifies how agentic, multimodal AI can enable intelligent, user-centric assistive technologies.

Problem

• Adapting hearing aids to dynamic acoustic environments in real-time
• Providing personalized hearing adjustments through multimodal user inputs
• Overcoming limitations of static fittings with contextual sound classification

Innovation

• Multi-agent LLM workflow for hearing aid adjustments
• Real-time sound classification using lightweight neural network
• Multi-turn conversational system integrating audio and feedback