Efficient Prompt Optimisation for Legal Text Classification with Proxy Prompt Evaluator

2025-10-09

AI Summary

Prompt optimization for legal text classification incurs high computational costs and suffers from low search efficiency. Method: This paper proposes an efficient prompt optimization framework integrating Monte Carlo Tree Search (MCTS) with a lightweight surrogate evaluator. A surrogate model rapidly approximates candidate prompts’ performance on Terms-of-Service (ToS) fairness detection, drastically reducing per-evaluation overhead; meanwhile, MCTS guides structured, goal-directed exploration of the prompt space under strict computational budgets. Contribution/Results: Experiments demonstrate that our method significantly outperforms baselines in both classification accuracy and search efficiency under identical resource constraints. It establishes a scalable, low-overhead paradigm for prompt optimization in high-cost legal NLP tasks, enabling practical deployment without sacrificing performance.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Prompt Engineering / Prompting
• Constraint Satisfaction and Optimization: Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Prompt optimization aims to systematically refine prompts to enhance a language model's performance on specific tasks. Fairness detection in Terms of Service (ToS) clauses is a challenging legal NLP task that demands carefully crafted prompts to ensure reliable results. However, existing prompt optimization methods are often computationally expensive due to inefficient search strategies and costly prompt candidate scoring. In this paper, we propose a framework that combines Monte Carlo Tree Search (MCTS) with a proxy prompt evaluator to more effectively explore the prompt space while reducing evaluation costs. Experiments demonstrate that our approach achieves higher classification accuracy and efficiency than baseline methods under a constrained computation budget.

Problem

• Optimizing prompts for legal text classification tasks
• Reducing computational costs in prompt evaluation methods
• Improving fairness detection in Terms of Service clauses

Innovation

• Monte Carlo Tree Search explores prompt space effectively
• Proxy evaluator reduces prompt scoring computational costs
• Framework achieves higher accuracy with constrained budget