🤖 AI Summary
Patent specification quality assessment lacks systematic methodologies, particularly amid the surge in AI-generated content, where multidimensional issues—including regulatory compliance, technical coherence, and figure-text consistency—demand automated solutions. This study proposes the first large language model (LLM)-based, multi-dimensional evaluation framework tailored for patent specifications, integrating rule-based reasoning with LLM capabilities. It comprises four core modules: compliance checking, technical coherence analysis, figure-text consistency verification, and improvement suggestion generation. Evaluated on 160 real-world patents, the three detection modules achieve balanced accuracies of 99.74%, 82.12%, and 91.2%, respectively. The framework uncovers, for the first time, systemic structural misalignment in AI-generated patents and identifies domain- and author-type–specific quality variations.
📝 Abstract
Despite the surge in patent applications and emergence of AI drafting tools, systematic evaluation of patent content quality has received limited research attention. To address this gap, We propose to evaluate patents using regulatory compliance, technical coherence, and figure-reference consistency detection modules, and then generate improvement suggestions via an integration module. The framework is validated on a comprehensive dataset comprising 80 human-authored and 80 AI-generated patents from two patent drafting tools. Experimental results show balanced accuracies of 99.74%, 82.12%, and 91.2% respectively across the three detection modules when validated against expert annotations. Additional analysis was conducted to examine defect distributions across patent sections, technical domains, and authoring sources. Section-based analysis indicates that figure-text consistency and technical detail precision require particular attention. Mechanical Engineering and Construction show more claim-specification inconsistencies due to complex technical documentation requirements. AI-generated patents show a significant gap compared to human-authored ones. While human-authored patents primarily contain surface-level errors like typos, AI-generated patents exhibit more structural defects in figure-text alignment and cross-references.