SCOPE: Sequential Conformal Probing for Reliable OOD Rejection in LLM Services

2026-06-19

AI Summary

This work addresses the challenge of reliably rejecting out-of-domain (OOD) requests prior to generation in large language model (LLM) services. The authors propose a conformal gating mechanism grounded in hidden-layer signals, which leverages the first systematic identification of geometric disparities induced by OOD inputs within internal model representations. By integrating inductive calibration with supermartingale-based e-processes, the method statistically certifies the accumulating evidence at decision boundaries, thereby providing theoretical guarantees for retained in-domain inputs. Evaluated across multiple mainstream LLMs and six diverse boundary conditions, the approach significantly outperforms conventional OOD detection methods that rely solely on final-layer representations.

Abstract

Rejecting inputs outside the defined in-distribution (IND) service scope is critical for large language model (LLM) services, where unsupported requests should be filtered before full generation. Existing out-of-distribution (OOD) detectors often rely on final outputs or final-layer representations, leaving unclear where service-boundary signals are most clearly encoded inside the model; they also lack a theoretical guarantee for held-out inputs. In this paper, we introduce SCOPE (Sequential Conformal OOD Probing and Evaluation), a framework that selects a readable hidden layer, constructs a conformal gate with IND calibration, and uses a supermartingale e-process to certify persistent service-boundary evidence. Experiments across multiple LLM backbones and six carefully designed boundary conditions show that SCOPE improves gate-level rejection over standard final-layer detectors, while revealing how different OOD boundaries take different geometric forms in hidden space.

Problem

• out-of-distribution detection
• large language models
• service scope
• reliable rejection
• conformal prediction

Innovation

• conformal prediction
• out-of-distribution detection
• hidden layer probing
• supermartingale e-process
• LLM service boundary