Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models

2026-07-09

AI Summary

This work proposes Sensitivity-Aware Threshold Calibration (SATS) combined with token-wise dynamic routing to reduce the computational overhead of large language model inference while preserving performance. SATS replaces conventional percentile-based thresholds with a local sensitivity proxy to enable more precise sparsification of MLP activations. Coupled with a lightweight conditional path selection mechanism, the approach dynamically determines the computational path for each token. Experiments across multiple open-source large language models demonstrate that the proposed method significantly outperforms existing baselines at the same sparsity level, achieving a superior trade-off between output quality and inference throughput.

Abstract

Efficient inference in Large Language Models (LLMs) requires deciding where computation can be reduced while preserving model quality. We study this problem through multilayer perceptron (MLP) activation sparsification and token-level conditional routing. We first propose Sensitivity-Aware Thresholding for Sparsity (SATS), a threshold calibration method to choose layerwise gate thresholds using a local MLP output sensitivity proxy rather than calibrating thresholds directly from activation percentiles. While SATS retains the existing mechanism of sparsifying MLP activations by thresholding gate activations, it replaces percentile-based calibration with a sensitivity-aware selection rule. We then introduce a lightweight token routing framework that dynamically selects between a base path and a modified path on a per-token basis, rather than applying the modified computation uniformly to all tokens. We evaluate both methods on multiple recent open-weight LLMs. Our results show that SATS improves over the threshold-based sparsification baseline at matched actual sparsity and that token routing yields a more favorable quality-throughput trade-off than static activation modification baselines. Overall, our results suggest that improved threshold calibration and token routing can improve the quality-throughput trade-off in LLMs.

Problem

• activation sparsification
• token routing
• inference efficiency
• quality-throughput trade-off
• large language models

Innovation

• activation sparsification
• sensitivity-aware thresholding
• token routing
• large language models
• efficient inference