Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports

2026-09-01

AI Summary

研究解决了知识图谱提取工具评分不一致的问题，通过重新评估和开发CTIForge系统来提高匹配准确性和验证层的稳定性。

Abstract

Security teams and researchers choose knowledge-graph extraction tooling for threat reports on the strength of published triple-F1 scores, yet those scores depend on how predicted triples are matched to gold annotations. We could reimplement the stated matching rule for only five of twelve inspected systems. Re-scoring ten system outputs on shared documents under eight protocols reverses eleven of forty-five pairwise orderings; one fixed prediction set spans 0.16-0.70 F1. On GRID's external 378-item calibration set, no mechanical matcher (lexical, embedding, or entailment) agrees with multi-reviewer adjudication above 71%, whereas an LLM judge reaches 86%. To separate component effects from matcher rewards, we build CTIForge, whose deterministic validation layer can vary while extraction is held byte-identical. Across seven tested deployment configurations, validation raises precision for all four hosted backbones and lowers it for all three offline backbones. Because backbone, decoding, and backend-specific prompting covary, this is a descriptive split rather than an isolated serving effect. It coincides with a roughly 2.8-fold increase in actions explicitly disputing entity type, consistent with hand-written rules encoding the conventions of the extractor against which they were developed. We release the pipeline, protocol suite, and per-triple audit records.

Problem

• knowledge-graph extraction
• threat reports
• reproducibility
• triple-F1 scores
• matching rule

Innovation

• CTIForge
• deterministic validation layer
• backbone precision
• LLM judge
• multi-reviewer adjudication