ConvScale: Conversational Interviews for Scale-Aligned Measurement

2026-03-12

AI Summary

This study addresses the challenge of integrating qualitative, interview-based assessments into quantitative psychometric evaluation while preserving scale structural validity. The authors propose ConvScale, a novel method that seamlessly embeds structured psychological scales into naturalistic conversational interviews. Leveraging an AI-driven dialogue system, ConvScale guides participants through contextually rich interactions and employs natural language processing to map utterances to specific scale items, which are then aggregated into construct-level quantitative scores. This approach enables quantitative analysis without sacrificing contextual depth, thereby expanding the applicability of interviews in psychometrics. In an experiment with 18 participants, item- and construct-level scores generated by ConvScale showed strong agreement with self-reports and demonstrated moderate internal consistency. Although structural validity requires further refinement, the findings substantiate ConvScale’s feasibility as an innovative tool for quantitative psychological assessment.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Search and Optimization: Non-convex Optimization
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Conversational interviews are commonly used to complement structured surveys by eliciting rich and contextualized responses, which are typically analyzed qualitatively. However, their potential contribution to quantitative measurement remains underexplored. In this paper, we introduce ConvScale, an AI-supported approach that transforms psychometric scales into natural conversational interviews while preserving the original measurement structure. Based on interview data, ConvScale predicts item-level scores and aggregates them to derive scale-based assessments. In a within-subjects study with 18 participants, our results show that ConvScale-derived scores align closely with participants' self-report scores at both the item and construct levels, while maintaining moderate internal reliability; however, the structural validity was inadequate. In light of this, we discussed the potential of supporting quantitative measurement through interviews and proposed implications for future designs.

Problem

• conversational interviews
• quantitative measurement
• psychometric scales
• scale-aligned measurement
• item-level scoring

Innovation

• ConvScale
• conversational interviews
• psychometric scales
• AI-supported measurement
• quantitative assessment