Appendix: The Intuition Pack

Why can't you just train an RNN in parallel?

Training an RNN is a relay race. Runner 47 cannot start until runner 46 hands over the baton — no matter how many runners you hire, the race takes T handoffs. That handoff is $h_{t-1}$; the race is backprop through time.

A parallel scan is a group photo. Everyone shows up at once, the photographer needs a few group arrangements, done. More people = a wider photo, not a longer race.

The entire minGRU project is: what do you have to give up so the race can become a photo?

Main deck: "The problem, and the trick."

What exactly did minGRU give up?

The GRU's gates read the memory before deciding what to do with it. The minGRU deal removes exactly that privilege:

You're a librarian who must write tomorrow's filing rule tonight — before seeing what's on the shelves. You may write a very clever rule ("shrink fiction by 20%, flip the sign of every ledger") based on tomorrow's mail. But you cannot peek at the shelves.

Every mixer in the talk is a richer rulebook for that blind librarian — never permission to peek. Peeking is the thing that costs T handoffs.

Main deck: "The algebra: delete the state from the gates."

What does the scan actually combine?

Each token hands the memory a slip of paper:

"Multiply what you have by $a$, then add $b$."

The magic is that two slips staple into one slip of the same form. "Do $(a_1, b_1)$, then $(a_2, b_2)$" collapses to a single instruction: multiply by $a_2 a_1$, add $a_2 b_1 + b_2$. Same shape in, same shape out — so slips can be merged in pairs, pairs of pairs, and so on: the group photo.

The entire mixer ladder is one question: what is $a$ allowed to be? A fraction? A signed number? A rotation matrix? Each answer is a rung.

Main deck: "Our combine op: composing affine maps."

What does "associative" buy us — and what doesn't it?

Add a list of numbers: $(3+1)+4$ or $3+(1+4)$ — same answer. You can hand chunks to friends and combine their results. That's associativity, and it's ALL a scan needs.

Careful — this is not "order doesn't matter":

• Associativity: re-bracket freely. $(a \circ b) \circ c = a \circ (b \circ c)$. ✔ required
• Commutativity: re-order freely. $a \circ b = b \circ a$. ✘ never required

Matrix multiplication keeps the first and drops the second — which is exactly why a scan can track things where order matters.

Main deck: "What is an associative scan?"

What is a "word problem," really?

A choreography is a list of moves. The word problem asks: after moves 1..t, what single move equals the whole routine so far?

Give the dancer's body a facing direction. Every move turns it, some moves mirror it. Then the answer is always just: which way are you facing, and are you mirrored? No memorization — the body is the running answer.

Main deck: "Interlude: groups, S₃" + "How a rotating state tracks the group."

What does "non-abelian" mean?

Socks then shoes ≠ shoes then socks. That's the whole of it. Any process where A-then-B differs from B-then-A cannot be summarized by a bag of counts — you must track the order-sensitive net effect.

Main deck: "Why diagonal can never track S3."

What can a constant-depth circuit do — and provably not do?

• Constant depth (TC⁰): put everyone in one room, ask "raise your hand if…", count hands. However many people arrive, it's still one round of looking. Wider room, never a longer meeting.
• Recurrence: the telephone game. Each person whispers to the next. Depth = number of people, by construction.

Deep theorem, kitchen version: some games are provably telephone games — shuffling 5 cups (S5) is one. No hand-raising protocol with a fixed number of rounds plays it. Shuffling 3 cups is not — it fits in the room. (The algebra word: S3 is solvable — buildable from commutative layers; S5 isn't.) That's why the whole talk lives on 3 cups.

Main deck: "What 'constant-depth circuit' means" + "The price."

The scan runs in log T rounds — doesn't that escape the ceiling?

No — and the reason is a distinction worth hammering: the depth of the computation you run is not the depth of the function you compute. A vending machine can be gorgeously complicated inside; its power is its input→output table, nothing more.

The theorem says: whatever table a fixed stack of these scans implements, some constant-depth counting circuit implements the same table. The cleverness of your algorithm never upgrades the function it computes.

Main deck: "What 'constant-depth circuit' means" (the theorem line).

What is the ladder actually measuring?

One question per rung: what are you allowed to multiply the memory by? Fractions that fade $(0,1)$ → signed numbers that also flip $(-1,1)$ → a rotation that turns → many rotations at once. Each family strictly contains the last.

The reframe this card exists for: tasks are locked doors; multiplier families are key rings. Rung 1's failure on parity is not a training failure, and no amount of data fixes it — the key isn't on the ring. A rung is a permission, not a skill level. (These rungs trace the rotation lineage, topping out at Givens' small-state group-composition niche; the delta rule is a second key-ring — the one that opens the most public-benchmark doors, and the promoted default when state can grow.)

Main deck: "Rung 1 — base minGRU" through "The ladder so far."

Why does parity need −1?

Parity is a light switch. A 1 flips it, a 0 leaves it alone. After a million bits, the answer is just: is the light on?

The base minGRU cannot be a light switch: its memory can only fade and blend — multiply by numbers between 0 and 1. A switch needs −1. That's the entire signed-mixer story: let the multiplier reach −1, and the switch exists.

Main deck: "Rung 2 — SignedMinGRU."

Why do rotation matrices help get to a higher rung?

Because the thing being tracked is itself made of rotations:

Once the group's own moves live inside the multiplier family, the state stops describing the sequence and starts acting it out — the final pose is the answer.

Main deck: "How a rotating state tracks the group" + "Rung 3."

Why can't diagonal do it — at any width, with any data?

A diagonal transition gives each memory channel its own private volume knob: channel $i$ ends the sequence at its start value times $a_1 a_2 \cdots a_T$ — a product of plain numbers.

Plain numbers commute. $0.5 \times (-1) \times 0.8$ is the same in any order. So a diagonal memory is order-blind by arithmetic — not hard to train, provably impossible.

A rotation is a steering wheel, not a volume knob — it mixes channels. Turn 90° then flip ≠ flip then turn 90° (try it with your hand). Once multipliers stop commuting, the final state can depend on order — exactly what a word problem asks for.

Main deck: "Why diagonal can never track S3."

Why do rotations survive 10,000 steps when nothing else does?

Multiply memory by 0.9, a thousand times: $0.9^{1000} \approx 10^{-46}$ — a photocopy of a photocopy of a photocopy, blank page. Multiply by 1.01 instead: overflow. Any knob not at exactly 1 destroys the signal over a long product, one way or the other.

A rotation moves the arrow without shrinking or stretching it — after 10,000 steps the memory is exactly as loud as at step 1. Orthogonal maps are the only multipliers you can compose forever and lose nothing.

The group-friendly multipliers and the long-sequence-friendly multipliers are the same family. Not a coincidence — the same property.

Main deck: "How a rotating state tracks the group" + the Givens rungs.

Why do some fits hold at length while others rot?

• A volume knob with click-stops (detents): bump it, it settles back into the notch. Every use re-centers it. That's an attractor — the snap grid manufactures clicks at exactly the group's angles; tanh's flat ends are clicks at ±1.
• A smooth knob: set it to 120°… you'll get 119.7°. Nothing pulls it back. Every turn adds its tiny error.

Length generalization is not about fitting the data. It's about whether the knob has clicks.

Main deck: "Rung 3 — RotationMinGRU" (snap), "Rung 2" (tanh).

How do you train an angle that's been snapped to a grid?

Rounding has no gradient — the snap is a staircase, and staircases are flat almost everywhere. The straight-through estimator is karaoke autotune:

• The audience (the loss, the forward pass) only ever hears the snapped note — the network genuinely performs exact group angles.
• The singer's earpiece (the gradient, the backward pass) plays the raw voice — so learning still knows which direction to drift.

Perform quantized, learn smooth. That's how you manufacture a click on a knob that never had one.

Main deck: "Rung 3 — RotationMinGRU," protocol piece 1.

Why does the continuous composer decay at length?

A compass that reads 1° wrong is fine for a walk to the corner store and catastrophic across a desert: the error is $\varepsilon$ per mile, and miles multiply it into $\varepsilon T$.

A continuous rotation composer is that hiker. It learned 119.7°, not 120°. At T=64 nobody notices; by T=1024 it's in the wrong county. Clicks (snapping) zero the error each step — but the smooth hiker is the one that reliably finishes training. Pick your desert.

Main deck: "The trade Givens does not escape."

Expressivity vs. trainability — what's the actual difference?

The deepest distinction in the talk — and the reason the repo exists:

• Expressivity asks: can the piano play this piece? (Do weights exist? Theorems answer this — Merrill says some pieces need a different instrument; Grazzi says this piano has more octaves than you thought.)
• Trainability asks: will the pianist actually learn it? (Does SGD find those weights? No theorem answers this. You have to run seeds.)

The 2D and 8D rotation composers are the same piano — identical expressivity class, both can represent the S3 tracker. One is learned 1 time in 12, the other 8 times in 12. The piece didn't change. The learnability did.

Main deck: "Rung 3" (representable ≠ trainable) + "Map richness is the trainability lever."

Why is S3-hier hard — on purpose?

It hides each dance move in a pair of symbols, coded so that one symbol alone is useless — like a safe needing two keys where each key alone tells you nothing about the combination.

The "Latin square" is just the mathematically perfect version of that uselessness: lay the code out as a grid and every move appears exactly once in each row and column (Sudoku's rule). Know the row? Still clueless. Know the column? Still clueless. Know both? Certain.

So the model is forced to learn two skills: read pairs (extraction), then dance (composition). No shortcut exists — by construction.

Main deck: "Task three: S3-hier."

Why does the stack need two different layers?

S3-hier needs two different jobs, in order — an assembly line:

• Station 1 unwraps: pair of symbols → which move is inside.
• Station 2 dances: stream of moves → running net effect.

Every one-station factory fails: a rotation-only stack has no unwrapper; a signed-only stack has no dancer. The heterogeneous stack (["signed", "givens"]) is the right factory layout.

But layout isn't enough. With the 2D composer at station 2, the job is so finicky that 1 hire in 12 works out. Givens rebuilds the station so 8 in 12 do. Same factory, same jobs — an easier job description.

Main deck: "The harder problem: extract then compose."

What is a Givens rotation actually doing?

Grabbing two sliders on a mixing desk and cross-fading them by an angle — every other slider untouched. Perfectly reversible, never distorts the signal (orthogonal), one knob's worth of freedom.

One cross-fade is timid. So: do four disjoint pairs at once, then the staggered pairs, then again — three rounds and all eight channels have talked to each other. Enough rounds could reach any clean re-mix (all of SO(8)); three is the budget that measured best per dollar.

Main deck: "What is a Givens rotation?" + "Rung 3½."

What is a Householder map actually doing?

It's one mirror. A mirror is drastic — it flips the whole world across a plane — but it is perfectly faithful: nothing stretches (orthogonal, at β = 2 exactly).

Barbershop fact: two mirrors facing each other make a rotation. Stack k mirrors per token and you can build any clean turn — that's DeltaProduct's move. Same game as Givens, opposite atom: Givens composes tiny turns, DeltaProduct composes whole mirrors. And mirror-count is a real dial: a $k$-cycle needs $k-1$ reflections, so bumping delta to nh=4 is what first gets traction on the S5 word problem (L4 benchmark) where too few mirrors get none.

At β < 2 the mirror goes dim — the reflected copy comes back smaller. Useful (next card), but no longer distortion-free.

Main deck: "The other lineage: the delta rule."

What is the delta rule actually doing?

Not turning anything. Its state is a whiteboard of sticky notes: each note = a key → value pair. A new token walks up, peels off whatever note is under its key, sticks a new one.

$$S_t = S_{t-1}(I - \beta k k^\top) + \beta v k^\top \quad = \quad \text{peel, then stick}$$

Obvious consequence, empirically confirmed: this strategy lives and dies by whiteboard size — and how many notes it rewrites per token (its nh Householder corrections). Small board → it trails Givens at a matched state; a big board → it fits nearly every seed. Rotation families barely care about board size; they care about the richness of their turns. Two lineages, two dials — and across the public benchmark suite (recall, long accumulation, group word-problems) the delta family is the broad workhorse, while Givens stays the small-state group-composition specialist.

Main deck: "Four ways to move a state" + the reliability bars.

If 2D and 8D are equally expressive, why does 8D train better?

Landing the 2D composer is parallel-parking into one exact spot: exactly 120°, reflections laid out just right. One narrow way to be right — a needle.

The 8D Givens composer has 12 angle knobs and a continuum of equally-correct configurations — the same automaton embedded a thousand ways. A whole parking lot of valid spots. SGD is a sleepy driver: it rarely finds the needle, it usually finds a spot in the lot.

What's measured is the jump: 1/12 → 8/12. The parking lot is the best available explanation — a landscape hypothesis, not a theorem.

Main deck: "Map richness is the trainability lever."

What does time decay do to a rotating memory?

Token index is not time: a 3-second gap and a 3-day gap should not move memory equally. Decay multiplies the memory by $\gamma = e^{-\lambda \Delta t}$ — coffee cooling: the longer the gap, the cooler the memory; λ is how well-insulated the mug is.

Two things worth hammering:

• For rotating memories, cooling only shrinks — it never turns: a scalar commutes with the rotation; the group structure survives.
• Start with a great thermos (λ ≈ 0, persist by default) and let training discover where forgetting pays — the heavy-decay init only tied its baseline; the thermos init wins consistently. Measured.

Main deck: the two time-decay slides.