RPRA: Predicting an LLM-Judge for Efficient but Performant Inference

2026-04-14

AI Summary

Deploying large language models on resource-constrained devices entails an inherent trade-off between computational efficiency and output quality. This work proposes the Predict-Answer (PA) and Reason-Predict-Reason-Answer (RPRA) inference paradigms, wherein a small model predicts—prior to generating a response—the score its output would receive from a larger model evaluator, and autonomously decides whether to answer directly or defer to the larger model. The small model is trained to forecast these evaluations via three approaches: zero-shot prediction, context-based report cards, and supervised fine-tuning. The report card and fine-tuning strategies improve prediction accuracy by 55% and 52% on average, respectively, substantially enhancing the small model’s awareness and judgment of its own capability boundaries, thereby improving both inference efficiency and overall performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) face a fundamental trade-off between computational efficiency (e.g., number of parameters) and output quality, especially when deployed on computationally limited devices such as phones or laptops. One way to address this challenge is by following the example of humans and have models ask for help when they believe they are incapable of solving a problem on their own; we can overcome this trade-off by allowing smaller models to respond to queries when they believe they can provide good responses, and deferring to larger models when they do not believe they can. To this end, in this paper, we investigate the viability of Predict-Answer/Act (PA) and Reason-Predict-Reason-Answer/Act (RPRA) paradigms where models predict -- prior to responding -- how an LLM judge would score their output. We evaluate three approaches: zero-shot prediction, prediction using an in-context report card, and supervised fine-tuning. Our results show that larger models (particularly reasoning models) perform well when predicting generic LLM judges zero-shot, while smaller models can reliably predict such judges well after being fine-tuned or provided with an in-context report card. Altogether, both approaches can substantially improve the prediction accuracy of smaller models, with report cards and fine-tuning achieving mean improvements of up to 55% and 52% across datasets, respectively. These findings suggest that models can learn to predict their own performance limitations, paving the way for more efficient and self-aware AI systems.

Problem

• computational efficiency
• output quality
• large language models
• performance trade-off
• resource-constrained devices

Innovation

• RPRA
• LLM-Judge
• self-aware inference
• model deferral
• efficient LLMs