Four-dimensional transitions can preserve operation order
Symmetry in Time: A Representation Theory of Linear Recurrent Memory · arXiv:2611.17162
In plain words
Some sequences describe actions, such as turning or swapping things, where doing the same actions in a different order changes the answer.
The paper explains what kind of recurrent memory can keep that order, then introduces HyperSSM, which lets different memory channels use different kinds of numbers.
On a test that asks for the result of sequences of operations on five objects, HyperSSM got every tested answer right when sequences were drawn from a set of 32 available operations, called generators.
Earlier recurrent models were near chance on a task with the same number of generators, though the exact operations differed.
The research
A sequence model that loses operation order cannot solve tasks whose answers depend on composing actions in sequence.
Knowing what transition structure those tasks require can change which recurrent memory is a plausible fit, and the framework offers a way to make that choice from the operations a task uses.
A representation-theory framework identifies which transition number systems let a recurrent channel preserve the order of composed operations, then HyperSSM implements that choice.
HyperSSM learned the S₅ word task from final answers, without being told the group, training on sequences of 3–40 operations and testing on longer ones. With 32 generators, it was exact; the prior-work PD-SSM scored 1.07% accuracy, against 0.83% by chance.
The task asks for the result of composing permutations of five objects. The contrast fits the theory’s prediction that order-sensitive compositions need transitions that preserve order. On held-out gyroscope recordings, HyperSSM also had lower rotation error than direct integration at two minutes.
The theory assumes bounded reachable transitions and a continuous readout. The experiments use transitions without an input term and cover selected tasks; the paper leaves the effects of decay and noise, and the behavior of large models mixing these channels with attention, unresolved.
What the evidence shows
- Why order gets lost
The paper’s representation-theory framework is a way to determine which operation groups a linear recurrent memory can track. The motivating problem is that many recurrent models update each channel by multiplying a real or complex number.
Those multiplications commute, so sequences containing the same operations in different orders can leave the same state, even when the order changes the correct answer.
- A lower bound for order
The framework connects a channel’s reachable transition states to the group of operations it must track, while allowing its readout to ignore some state distinctions. It shows that transitions with fewer than four real dimensions can track only groups whose operations commute.
This makes four dimensions a minimum for preserving order in non-commuting operations, and the bound concerns the transition algebra rather than necessarily the stored state.
- Putting the theory to work
HyperSSM is a recurrent model that applies this choice of number system channel by channel. A quaternion channel can represent transitions whose order matters, while the channels’ states are combined in the readout.
The design makes the theoretical distinction usable in a sequence model while retaining the same parallel scan used by diagonal state space models.
- A sharp permutation result
On the S₅ task with 32 generators, HyperSSM predicted every composed permutation correctly, while the prior-work PD-SSM reached 1.07% accuracy, near chance.
This supports the order-preserving design on this task, though the baseline used different random extra generators, so the task instances were not identical.
- A physical sequence test
The paper also tested whether the order-preserving memory helps with measured motion. On held-out gyroscope recordings, HyperSSM’s rotation error at two minutes was 26.5°, compared with 30.9° for directly integrating the same readings.
This supports transfer to this physical sequence task, where orientation is built from ordered rotations, while the test remains limited to recordings from one sensor.
Deep
- Original paper authors
- Anif N. Shikder, Ramit Dey, Jan Minac, Lyle E. Muller
- Submitted to arXiv
- License
- CC BY 4.0