Institution profile

CVS Health

Industry researchnorthamerica · us
Official website
Research library7linked papers
Opportunities0open roles
Selected work

Representative Papers

Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification

May 27, 2026

This work addresses the prevalence of functional errors in code generated by large language models (LLMs) and the limited efficacy of existing uncertainty quantification (UQ) methods for code generation. The authors systematically evaluate the transferability of various UQ approaches and introduce a novel paradigm grounded in functional equivalence: leveraging LLMs to assess whether generated code is functionally equivalent to a reference implementation. Building on this, they define code-specific UQ metrics such as “functional entropy.” By integrating token-level probabilities, sampling strategies, and LLM-driven functional equivalence judgments, their method achieves state-of-the-art performance, attaining the highest AUROC in 11 out of 15 model–benchmark combinations and demonstrating substantially improved calibration over existing techniques.

0 citationsRead paper

Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study

Feb 19, 2026

Existing uncertainty quantification methods struggle to effectively detect hallucinations in long-form text generation. This work proposes the first fine-grained uncertainty quantification framework tailored for long texts, which systematically evaluates generated content through a three-stage pipeline: response decomposition, unit-level scoring, and response aggregation. The framework introduces a unified taxonomy that integrates and extends existing black-box consistency-based methods, enabling fair comparison and modular component selection. It employs black-box consistency scorers—including claim–response entailment, claim-level, and sentence-level scoring—combined with uncertainty-aware decoding. Experimental results demonstrate that claim–response entailment scoring achieves the best performance, claim-level scoring outperforms sentence-level scoring, and uncertainty-aware decoding significantly enhances factual consistency in long-form generation.

0 citationsRead paper

UQLM: A Python Package for Uncertainty Quantification in Large Language Models

Jul 08, 2025

Hallucinations in large language models (LLMs) pose serious threats to the safety and reliability of downstream applications. To address this, we introduce the first open-source Python toolkit for LLM hallucination detection. Our core method is a response-level confidence scoring system grounded in uncertainty quantification: it integrates multiple state-of-the-art uncertainty estimation techniques—including logit entropy, sampling variance, and calibration-aware confidence—to produce interpretable, normalized confidence scores in the [0,1] range. The toolkit is designed for plug-and-play deployment, modular extensibility, and seamless integration with mainstream LLM frameworks (e.g., Hugging Face Transformers, vLLM). Extensive experiments across multiple benchmark datasets demonstrate that our approach significantly improves hallucination detection accuracy, achieving an average +12.3% F1-score gain over baseline methods. This advancement enhances both the trustworthiness of generated content and the operational safety of LLM deployments.

0 citationsRead paper
Recent publications

Latest Papers

Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification

May 27, 2026

This work addresses the prevalence of functional errors in code generated by large language models (LLMs) and the limited efficacy of existing uncertainty quantification (UQ) methods for code generation. The authors systematically evaluate the transferability of various UQ approaches and introduce a novel paradigm grounded in functional equivalence: leveraging LLMs to assess whether generated code is functionally equivalent to a reference implementation. Building on this, they define code-specific UQ metrics such as “functional entropy.” By integrating token-level probabilities, sampling strategies, and LLM-driven functional equivalence judgments, their method achieves state-of-the-art performance, attaining the highest AUROC in 11 out of 15 model–benchmark combinations and demonstrating substantially improved calibration over existing techniques.

0 citationsRead paper

Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study

Feb 19, 2026

Existing uncertainty quantification methods struggle to effectively detect hallucinations in long-form text generation. This work proposes the first fine-grained uncertainty quantification framework tailored for long texts, which systematically evaluates generated content through a three-stage pipeline: response decomposition, unit-level scoring, and response aggregation. The framework introduces a unified taxonomy that integrates and extends existing black-box consistency-based methods, enabling fair comparison and modular component selection. It employs black-box consistency scorers—including claim–response entailment, claim-level, and sentence-level scoring—combined with uncertainty-aware decoding. Experimental results demonstrate that claim–response entailment scoring achieves the best performance, claim-level scoring outperforms sentence-level scoring, and uncertainty-aware decoding significantly enhances factual consistency in long-form generation.

0 citationsRead paper

UQLM: A Python Package for Uncertainty Quantification in Large Language Models

Jul 08, 2025

Hallucinations in large language models (LLMs) pose serious threats to the safety and reliability of downstream applications. To address this, we introduce the first open-source Python toolkit for LLM hallucination detection. Our core method is a response-level confidence scoring system grounded in uncertainty quantification: it integrates multiple state-of-the-art uncertainty estimation techniques—including logit entropy, sampling variance, and calibration-aware confidence—to produce interpretable, normalized confidence scores in the [0,1] range. The toolkit is designed for plug-and-play deployment, modular extensibility, and seamless integration with mainstream LLM frameworks (e.g., Hugging Face Transformers, vLLM). Extensive experiments across multiple benchmark datasets demonstrate that our approach significantly improves hallucination detection accuracy, achieving an average +12.3% F1-score gain over baseline methods. This advancement enhances both the trustworthiness of generated content and the operational safety of LLM deployments.

0 citationsRead paper