SURGELLM: Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization

2026-06-23

AI Summary

This work addresses key challenges in fine-tuning encoders for heterogeneous NLP tasks—namely, mismatched inductive biases, feature distortion due to class imbalance, and insufficient integration of lexical knowledge—by proposing SURGELLM, a unified multitask framework. SURGELLM incorporates three lightweight innovations: surgically precise feature gating (theoretically shown to mitigate representation degeneration), task-conditional prefix tokens, and instance-weighted normalization (IWN), which jointly enable lexical-driven feature alignment and debiasing. Evaluated across four diverse tasks, the model achieves an average macro F1 score of 0.940, outperforming the strongest baseline by 0.036, with a notable gain of 0.130 on authorship attribution. Ablation studies confirm that performance improvements stem from effective lexical information integration rather than increased parameter count.

Abstract

Fine-tuned encoders deployed across heterogeneous NLP tasks face three compounding problems: mismatched inductive biases, class-imbalance corruption of feature statistics, and no mechanism to condition attention on external lexical knowledge. We introduce \textbf{\surgellm}, a unified transformer framework that addresses each with a dedicated lightweight module: a \emph{surgical feature gate} (learned per-dimension sigmoid over curated lexical indicators and \texttt{[CLS]}; provably degenerates to identity when features are uninformative), \emph{task-conditioned prefix tokens} (quantized feature values and task identity prepended to every input), and \emph{Instance-Weighted Normalization} (IWN; removes class-prior bias from gate statistics). We prove an excess-risk bound linking gate benefit to \emph{surgical feature alignment}. Across four tasks, SST-2, multi-hop retrieval, LLM-prompt attribution, and authorship detection, covering 17,830 examples and eleven model variants over three seeds, the IWN variant achieves macro-F1 \textbf{0.940} ($+0.036$ over the strongest non-IWN baseline; $+0.130$ on authorship detection). A random-vocabulary control ($-0.028$ avg.\ F1) confirms gains are lexical, not parametric. Code, vocabularies, and a $99.5\%$-recovery auto-extraction recipe are released.

Problem

• inductive bias
• class imbalance
• feature statistics
• lexical knowledge
• multi-task evaluation

Innovation

• surgical feature gating
• task-aware prefix tokens
• instance-weighted normalization
• class-balanced representation
• multi-task NLP evaluation