Sense Representations Are Inducible Interfaces

2026-05-27

AI Summary

This work addresses the limitation of existing pretrained language models, which require explicit incorporation of lexical semantic structures during training to obtain interpretable word sense representations, thereby compromising their generalizability. The authors propose ACROS, a novel method that leverages a gated residual addition mechanism to dynamically induce a unified, explicit word sense pathway within a frozen pretrained decoder—eliminating the need for fine-tuning while supporting diverse semantic tasks. ACROS is the first approach to demonstrate effective word sense disambiguation, lexically guided generation, and cross-lingual alignment without modifying model weights. Evaluated on SmolLM2-360M, it achieves a zero-shot word sense disambiguation F1 score of 64.95, approximately 90% lexical guidance recovery rate, and strong cross-lingual adaptation performance on the four-language SENSIA benchmark with R@1 = 0.988 and PPL = 7.94.

Abstract

Sense representations (explicit, per-token meaning decompositions) are useful for disambiguation, steering, and cross-lingual alignment, but existing approaches require models to be pretrained with sense structure baked in. We introduce ACROS, which induces an explicit sense pathway into a frozen pretrained decoder LM through a gated residual addition. On SmolLM2-360M, ACROS preserves base LM quality while supporting three uses of the same induced variables: zero-shot word-sense disambiguation (64.95 F1 on Raganato ALL, competitive with the WordNet first-sense heuristic), low-KL lexical steering across 5,161 CoInCo cases where a simple non-oracle proxy recovers about 90% of positive shifts, and SENSIA cross-lingual adaptation to four languages (mean R@1 0.988, target FLORES PPL 7.94). ACROS makes sense representations an inducible interface for ordinary pretrained LMs.

Problem

• sense representations
• word-sense disambiguation
• lexical steering
• cross-lingual alignment
• pretrained language models

Innovation

• sense representations
• inducible interface
• ACROS
• word-sense disambiguation
• cross-lingual adaptation