In-Context Source and Channel Coding

2026-01-15

AI Summary

This work addresses the vulnerability of conventional separated source-channel coding to residual bit errors at low signal-to-noise ratios, which often leads to arithmetic decoding failure—particularly in large language model (LLM)-driven applications. The authors propose a receiver-side in-context decoding framework that, without modifying the transmitter, leverages an error-correcting code Transformer to estimate bit reliability. Guided by these reliability metrics, the method generates candidate sequences through reliability-aware bit flipping and integrates an LLM to perform arithmetic decoding with confidence-weighted fusion. This approach is the first to incorporate contextual information and LLMs into the receiver-side decoding process, offering both empirical stability in sampling and theoretical guarantees of convergence. Experimental results demonstrate significant performance gains over traditional separated and representative joint source-channel coding schemes under both AWGN and Rayleigh fading channels.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Separate Source-Channel Coding (SSCC) remains attractive for text transmission due to its modularity and compatibility with mature entropy coders and powerful channel codes. However, SSCC often suffers from a pronounced cliff effect in low Signal-to-Noise Ratio (SNR) regimes, where residual bit errors after channel decoding can catastrophically break lossless source decoding, especially for Arithmetic Coding (AC) driven by Large Language Models (LLMs). This paper proposes a receiver-side In-Context Decoding (ICD) framework that enhances SSCC robustness without modifying the transmitter. ICD leverages an Error Correction Code Transformer (ECCT) to obtain bit-wise reliability for the decoded information bits. Based on the context-consistent bitstream, ICD constructs a confidence-ranked candidate pool via reliability-guided bit flipping, samples a compact yet diverse subset of candidates, and applies an LLM-based arithmetic decoder to obtain both reconstructions and sequence-level log-likelihoods. A reliability-likelihood fusion rule then selects the final output. We further provide theoretical guarantees on the stability and convergence of the proposed sampling procedure. Extensive experiments over Additive White Gaussian Noise (AWGN) and Rayleigh fading channels demonstrate consistent gains compared with conventional SSCC baselines and representative Joint Source-Channel Coding (JSCC) schemes.

Problem

• cliff effect
• Separate Source-Channel Coding
• low SNR
• Arithmetic Coding
• bit errors

Innovation

• In-Context Decoding
• Error Correction Code Transformer
• Reliability-Guided Sampling
• LLM-based Arithmetic Decoding
• Separate Source-Channel Coding