🤖 AI Summary
Existing evaluation methods struggle to effectively assess the quality of patent specifications with respect to long-text coherence and legal compliance, particularly regarding enablement and written description requirements. This work proposes Pat-DEVAL, the first multidimensional evaluation framework tailored for patent specifications, which introduces a novel Chain-of-Legal-Thought (CoLT) mechanism that explicitly incorporates patent-law-specific constraints into the assessment process. By integrating legal-knowledge-guided reasoning and multidimensional scoring within an LLM-as-a-judge architecture, Pat-DEVAL establishes a new paradigm that balances technical rigor with legal adherence. Evaluated on the expert-annotated Pap2Pat-EvalGold dataset, the framework achieves a Pearson correlation coefficient of 0.69 overall and 0.73 on the legal compliance dimension, significantly outperforming existing baselines and general-purpose LLM evaluators.
📝 Abstract
Patent descriptions must deliver comprehensive technical disclosure while meeting strict legal standards such as enablement and written description requirements. Although large language models have enabled end-to-end automated patent drafting, existing evaluation approaches fail to assess long-form structural coherence and statutory compliance specific to descriptions. We propose Pat-DEVAL, the first multi-dimensional evaluation framework dedicated to patent description bodies. Leveraging the LLM-as-a-judge paradigm, Pat-DEVAL introduces Chain-of-Legal-Thought (CoLT), a legally-constrained reasoning mechanism that enforces sequential patent-law-specific analysis. Experiments validated by patent expert on our Pap2Pat-EvalGold dataset demonstrate that Pat-DEVAL achieves a Pearson correlation of 0.69, significantly outperforming baseline metrics and existing LLM evaluators. Notably, the framework exhibits a superior correlation of 0.73 in Legal-Professional Compliance, proving that the explicit injection of statutory constraints is essential for capturing nuanced legal validity. By establishing a new standard for ensuring both technical soundness and legal compliance, Pat-DEVAL provides a robust methodological foundation for the practical deployment of automated patent drafting systems.