Exploring Semantic Labeling Strategies for Third-Party Cybersecurity Risk Assessment Questionnaires

2026-02-09

AI Summary

This work addresses the limitations of existing third-party cybersecurity risk assessment questionnaires, which rely on keyword matching and fail to capture the deep semantic relationships between security control domains and assessment scopes. To overcome this, the authors propose a semantic tag–based retrieval framework that explores two labeling strategies: direct annotation by large language models (LLMs) and semi-supervised semantic labeling (SSSL). The SSSL approach constructs a tag taxonomy through embedding clustering, LLM-assisted annotation, and k-nearest neighbor label propagation, enabling precise retrieval in the semantic tag space. Notably, SSSL achieves strong generalization across large-scale question banks with only minimal manual annotation, significantly reducing LLM invocation costs while maintaining high labeling quality and semantic consistency, thereby substantially improving retrieval alignment.

Technology Category

• Data Mining & Knowledge Management: Semantic Web
• Natural Language Processing: Question Answering
• Machine Learning: Semi-Supervised Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

Third-Party Risk Assessment (TPRA) is a core cybersecurity practice for evaluating suppliers against standards such as ISO/IEC 27001 and NIST. TPRA questionnaires are typically drawn from large repositories of security and compliance questions, yet tailoring assessments to organizational needs remains a largely manual process. Existing retrieval approaches rely on keyword or surface-level similarity, which often fails to capture implicit assessment scope and control semantics. This paper explores strategies for organizing and retrieving TPRA cybersecurity questions using semantic labels that describe both control domains and assessment scope. We compare direct question-level labeling with a Large Language Model (LLM) against a hybrid semi-supervised semantic labeling (SSSL) pipeline that clusters questions in embedding space, labels a small representative subset using an LLM, and propagates labels to remaining questions using k-Nearest Neighbors; we also compare downstream retrieval based on direct question similarity versus retrieval in the label space. We find that semantic labels can improve retrieval alignment when labels are discriminative and consistent, and that SSSL can generalize labels from a small labeled subset to large repositories while substantially reducing LLM usage and cost.

Problem

• Third-Party Risk Assessment
• Semantic Labeling
• Cybersecurity Questionnaires
• Control Semantics
• Assessment Scope

Innovation

• semantic labeling
• third-party risk assessment
• semi-supervised learning
• large language models
• cybersecurity questionnaires