Problem
Research questions and friction points this paper is trying to address.
temporal correlations
memory
linear recurrent neural networks
learning dynamics
How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks
🤖 AI Summary
研究通过解析方法探讨了输入序列相关性如何影响线性递归神经网络的记忆形成过程，揭示了相关性对记忆成本及学习动态的影响。
📝 Abstract
The linear recurrent neural network (LRNN) is a simple model for studying how much memory a network builds up as it trains. For uncorrelated inputs, earlier work found that training itself settles the network between keeping the past and reacting only to the present. Real sequences are correlated, and we solve the learning dynamics exactly for correlated inputs. In the solution, keeping the past carries a cost. The whole effect of correlation lands on that cost. This cost reduces to the earlier one when inputs are uncorrelated and grows once they are positively correlated. Three findings follow. (1) Correlation reshapes the course of learning, not only its end. Memory builds, overshoots, and is partly removed, and the settled network keeps less of the past. (2) Memory switches off at a threshold set by one number, how much each input resembles the one just before it. Neither sequence length nor longer-range correlation moves this threshold. Memory is worth keeping only when the task needs the previous input more than the current input already supplies it through correlation with the past. (3) The best network changes too. Zero error demands a feedthrough, a path that passes the current input straight to the network's output and remembers nothing, and training builds it unprompted when given one spare hidden dimension. Our work turns one property of the input into a prediction of whether a network learns memory and explains why correlated data turns recurrent networks into change detectors.
Innovation
Methods, ideas, or system contributions that make the work stand out.
correlated inputs
learning dynamics
memory formation
change detectors
🔎 Similar Papers
No similar papers found.
