Secu-Table: a Comprehensive security table dataset for evaluating semantic table interpretation systems

2025-11-09

AI Summary

The cybersecurity domain lacks publicly available, semantically annotated tabular benchmarks—especially for evaluating large language models (LLMs) on semantic table interpretation. Method: We introduce Secu-Table, the first large-scale, manually curated tabular benchmark for cybersecurity, comprising over 1,500 real-world security tables and 15,000+ entities. It is constructed from CVE/CWE vulnerability data and enriched via fine-grained semantic alignment and annotation using Wikidata and the SEPSES Cybersecurity Knowledge Graph (CSKG). Contribution/Results: Secu-Table supports core tasks such as table-to-knowledge-graph mapping and serves as the official benchmark for the SemTab challenge. It enables unified evaluation of both open- and closed-weight LLMs—including Falcon-3B, Mistral-7B, and GPT-4o mini. The dataset, annotation guidelines, and preprocessing code are fully open-sourced, thereby filling a critical gap in the evaluation of semantic table understanding for cybersecurity applications.

Technology Category

• Application Domains: Security
• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Evaluating semantic tables interpretation (STI) systems, (particularly, those based on Large Language Models- LLMs) especially in domain-specific contexts such as the security domain, depends heavily on the dataset. However, in the security domain, tabular datasets for state-of-the-art are not publicly available. In this paper, we introduce Secu-Table dataset, composed of more than 1500 tables with more than 15k entities constructed using security data extracted from Common Vulnerabilities and Exposures (CVE) and Common Weakness Enumeration (CWE) data sources and annotated using Wikidata and the SEmantic Processing of Security Event Streams CyberSecurity Knowledge Graph (SEPSES CSKG). Along with the dataset, all the code is publicly released. This dataset is made available to the research community in the context of the SemTab challenge on Tabular to Knowledge Graph Matching. This challenge aims to evaluate the performance of several STI based on open source LLMs. Preliminary evaluation, serving as baseline, was conducted using Falcon3-7b-instruct and Mistral-7B-Instruct, two open source LLMs and GPT-4o mini one closed source LLM.

Problem

• Lack of public tabular datasets for security domain evaluation
• Need to assess semantic table interpretation systems using LLMs
• Addressing domain-specific challenges in cybersecurity knowledge extraction

Innovation

• Constructed security dataset from CVE and CWE sources
• Annotated tables using Wikidata and SEPSES knowledge graphs
• Evaluated semantic interpretation with open-source LLMs baseline