54  Roads Not Taken

TipTL;DR

Three architectures that targeted real limitations of their era’s mainline but did not become dominant: Capsule Networks (Sabour et al., 2017), Neural Turing Machines (Graves et al., 2014), and Neural ODEs (Chen et al., 2018).

Depends on: The Transformer

54.1 Why these matter

Each idea below solved a specific, real limitation of its era’s mainline architecture and was not displaced because the problem was fake; it was displaced by a simpler alternative that scaled better.

54.2 The ideas

mainline (survivors) Capsule Networks Neural Turing Machines Neural ODEs
Architecture Problem targeted Core mechanism
Capsule Networks (Sabour et al., 2017) Max-pooling discards spatial pose (position, orientation, scale) Vectors (“capsules”) encode presence and pose; routing-by-agreement replaces pooling
Neural Turing Machines (Graves et al., 2014) A fixed-size RNN hidden state bottlenecks algorithmic generalization (copy, sort) A differentiable external memory matrix with content- and location-based read/write heads
Neural ODEs (Chen et al., 2018) Depth is a fixed, discrete architectural choice A ResNet block \(y = x + F(x)\) reinterpreted as one Euler step of \(\frac{dz}{dt} = f(z(t), t)\), solved by an ODE solver

54.2.1 Capsule Networks: Pose-Aware Routing

Capsule length encodes the probability a feature is present; orientation encodes its pose. Routing-by-agreement replaces pooling: a lower-level capsule sends its output to whichever higher-level capsule’s prediction it agrees with most, so a face capsule activates only when its eye, nose, and mouth capsules agree on a consistent pose, not merely on presence.

54.2.2 Neural Turing Machines: Differentiable External Memory

A controller network reads and writes an external memory matrix through differentiable heads, addressing memory by content (find the slot most similar to a query) or by location (move to a nearby slot). Controller and memory access train end to end with backpropagation, the same mechanism every other architecture in this book uses.

54.2.3 Neural ODEs: Continuous-Depth Networks

Stacking \(N\) discrete ResNet blocks approximates a continuous transformation via the Euler method. Replacing the discrete stack with an ODE solver makes depth an emergent property of how many solver steps a given input needs, rather than a fixed architectural choice, an adaptive, per-input compute budget.

54.2.4 What to observe

  • Each targets a specific, real weakness of its era’s mainline: pooling’s spatial information loss, RNN hidden-state capacity, fixed discrete depth.
  • Each introduces a genuinely new computational primitive (routing-by- agreement, differentiable memory addressing, an ODE solver) rather than recomposing existing primitives, which is precisely what made each expensive to scale.
  • All three remain trainable end to end with backpropagation.

54.3 Why they did not take over

Architecture Why it lost
Capsule Networks Routing-by-agreement is iterative and expensive per capsule; did not scale to the depth and data volume that made CNNs, then attention, effective through scale alone
Neural Turing Machines Differentiable memory addressing was unstable to train at scale; a Transformer’s long context window is itself an attended-to memory, delivering most of the benefit without a separate module
Neural ODEs Numerical ODE solvers are slower in practice than a fixed discrete stack for comparable accuracy; few tasks need adaptive per-input depth badly enough to justify the cost
NoteKey takeaway

Each architecture here lost to a simpler alternative that scaled better, not to a better argument against it. The pattern, an elegant special- purpose mechanism displaced by a simpler primitive trained at greater scale, recurs throughout this book: attention over recurrence, next-token prediction over explicit world models.