ToMMeR -- Efficient Entity Mention Detection from Large Language Models

2025-10-22

AI Summary

To address the inefficiency and fine-tuning dependency of entity mention detection in large language models (LLMs), this paper proposes ToMMeR—a lightweight probe model (<300K parameters) leveraging early-layer hidden representations of LLMs. Methodologically, ToMMeR exploits the emergent, structured entity representations naturally present in early Transformer layers, enabling zero-shot, high-recall mention boundary identification. It integrates a span classification head with the LLM’s intrinsic discriminative capability to filter and calibrate candidate mentions. Evaluated across 13 NER benchmarks, ToMMeR achieves 93% zero-shot recall; after LLM-based discrimination, precision exceeds 90%. With an extended classification head, F1 scores reach 80–87%, approaching state-of-the-art performance. This work is the first systematic demonstration that early LLM layers encode transferable, trainable-free mention detection capacity—establishing a new paradigm for efficient, training-free entity recognition.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Identifying which text spans refer to entities -- mention detection -- is both foundational for information extraction and a known performance bottleneck. We introduce ToMMeR, a lightweight model (<300K parameters) probing mention detection capabilities from early LLM layers. Across 13 NER benchmarks, ToMMeR achieves 93% recall zero-shot, with over 90% precision using an LLM as a judge showing that ToMMeR rarely produces spurious predictions despite high recall. Cross-model analysis reveals that diverse architectures (14M-15B parameters) converge on similar mention boundaries (DICE >75%), confirming that mention detection emerges naturally from language modeling. When extended with span classification heads, ToMMeR achieves near SOTA NER performance (80-87% F1 on standard benchmarks). Our work provides evidence that structured entity representations exist in early transformer layers and can be efficiently recovered with minimal parameters.

Problem

• Efficiently detecting entity mentions from early LLM layers
• Achieving high recall and precision in zero-shot NER benchmarks
• Proving structured entity representations exist in early transformers

Innovation

• Lightweight model probing early LLM layers
• Achieves high recall zero-shot mention detection
• Efficiently recovers structured entity representations