Vibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting Agents

2026-09-11

AI Summary

研究通过Vibe Patenting平台评估LLM法官在专业专利撰写中的可靠性，使用迭代反馈方法提高AI生成的专利草稿质量。

Abstract

LLM judges are increasingly used to evaluate and improve AI-generated outputs, yet their reliability for complex professional work remains unclear. We study this problem through Vibe Patenting, an end-to-end patent-drafting testbed for AI-agent evaluation. A separately-invoked LLM judge evaluates generated patent drafts and provides structured feedback for iterative revision. Across multiple inventions and drafting-agent configurations, judge-guided revision consistently improves judge-assessed quality, while unguided revision tends to saturate. Notably, iterative judge feedback enables a low-reasoning agent to approach the performance of a substantially more expensive high-reasoning agent. Stronger models and increased reasoning generally improve judge-assessed drafting quality, while domain-specific agentic workflows provide further gains. We validate the judge against independent evaluation by a professional patent attorney and find meaningful but strongly metric-dependent agreement and systematic calibration differences. These results highlight both the utility and limitations of LLM judges as evaluators and optimization signals for complex professional workflows.

Problem

• LLM judges
• patent-drafting
• AI-generated outputs
• iterative revision
• evaluation

Innovation

• LLM Judges
• Iterative Feedback
• Patent Drafting
• AI Agent Evaluation
• Vibe Patenting