Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

2026-06-23

AI Summary

This study addresses the lack of joint evaluation methods for assessing both accuracy and user satisfaction concerning non-functional requirements (NFRs) in multi-turn interactions with large language model (LLM)-based dialogue systems. Focusing on the iTrust codebase, the authors organized 49 developers to engage in multi-round dialogues with GitHub Copilot regarding 148 HIPAA-related NFRs. Through empirical analysis along three dimensions—requirement fulfillment, reasoning quality, and code localization—they propose the first integrated model combining system accuracy and user satisfaction. Leveraging expert annotations, dialogue log analysis, and regression modeling, the study reveals that while developers tend to trust LLM judgments, these are often inaccurate; overly verbose or information-dense responses reduce satisfaction, whereas proactive interaction significantly enhances user experience. The findings highlight the critical roles of response length, information density, and proactiveness in collaborative NFR assessment.

Abstract

LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional correctness. This leaves a critical gap in assessing the quality and accuracy of these conversations when handling Non-Functional Requirements (NFRs), which are inherently vague, context-dependent, and involve many parts of a program. Evaluating how well these systems support collaborative reasoning about NFRs requires methods that go beyond single-turn accuracy to capture both the correctness of the system's outputs and the quality of the multi-turn interaction. In this paper, we investigate the accuracy and quality of multi-turn conversations between developers and an LLM-based agent in the domain of Health Insurance Portability and Accountability Act (HIPAA) regulatory compliance. We hired 49 programmers to interact with GitHub Copilot to assess 148 HIPAA-derived NFRs against the iTrust codebase, a system designed to comply with HIPAA regulations, across three dimensions: requirement satisfaction level, reasoning, and code localization. We find that developers tend to agree with LLM assessments, but accuracy against expert ground truth is low. We model user satisfaction and find that longer system responses and more information-providing turns negatively affect user satisfaction, whereas proactive interactions positively affect it. Our findings provide insights for designing LLM-based dialogue systems that support NFR assessment.

Problem

• Non-Functional Requirements
• Multi-Turn Dialogue
• LLM Evaluation
• User Satisfaction
• Regulatory Compliance

Innovation

• multi-turn dialogue
• non-functional requirements
• LLM evaluation
• user satisfaction
• regulatory compliance