Compressed Computation under $L^4$ Loss is likely Computation in Superposition

2026-07-06

AI Summary

This study investigates whether a single-hidden-layer ReLU neural network can simultaneously compute multiple sparse input functions using fewer neurons than the number of target functions, via superposition. Training a network with only 50 neurons under L⁴ loss successfully achieves an overcomplete joint representation of 100 sparse binary feature functions. The work presents the first end-to-end trained model exhibiting an interpretable superposition-based computation mechanism, which relies on sparse codewords and a pseudoinverse decoding structure. Reverse engineering reveals that most of the network’s performance can be replicated using just three scalar parameters. Furthermore, manually constructed equivalent codewords validate the efficacy of this mechanism, offering novel insights into the internal representations learned by neural networks.

Abstract

Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions. It is natural to ask whether they can also compute more functions than they have neurons, i.e., perform computation in superposition. In this regime many functions of sparse inputs are evaluated by a layer with fewer neurons than there are functions to compute. Representation in superposition is by now fairly well understood, but computation in superposition is not, and there are few toy models of it arising through training rather than being hand designed. As a toy model of computation in superposition we study the compressed-computation setup: a single-hidden-layer ReLU network with 50 neurons that must compute the ReLU of each of 100 sparse input features. We show that training it under an $L^4$ loss (the mean fourth power of the error), rather than the usual $L^2$, elicits a solution that appears to compute all features in superposition. We then reverse-engineer this solution. We find that the network assigns each feature a sparse binary codeword over neurons and decodes it with a pseudoinverse of the encoder. Given these codewords, a description with only three scalars recovers most of the network's performance, and we validate it by building equivalent networks from hand-designed codes.

Problem

• computation in superposition
• compressed computation
• sparse inputs
• neural representation
• superposition

Innovation

• computation in superposition
• L4 loss
• compressed computation
• sparse binary codewords
• neural network reverse engineering