Depends on: Multimodal · BERT & GPT
Every architecture in this book so far consumes and produces symbols: text tokens, image patches, denoised pixels. None of them act in a physical environment. A robot arm following the instruction “pick up the red block” must go from pixels and language to a sequence of motor torques or end-effector positions, a fundamentally different output space from anything covered so far.
The VLA insight is that this does not require a new architecture family. If actions can be represented as tokens, the same Transformer that predicts the next word can predict the next action, reusing every mechanism this book has built (self-attention, autoregressive decoding, large-scale pretraining) for physical control.
An embodied agent runs a repeating cycle:
This loop is the physical-world analogue of an LLM’s autoregressive generation loop (Decoding): each action, like each token, conditions on everything that came before.
The key architectural move is representing continuous robot actions as discrete tokens drawn from the same vocabulary space as language tokens. A typical action is 7-dimensional: a 3D position delta, a 3D rotation delta, and a gripper open/close state. Each dimension is uniformly binned (e.g. into 256 bins) and mapped to a token ID, often reusing the least-used tokens in an existing language tokenizer’s vocabulary so no architecture change is needed at all, only a vocabulary reinterpretation:
\[ a \in \mathbb{R}^{7} \;\longrightarrow\; \text{discretize} \;\longrightarrow\; (a_1, \ldots, a_7) \in \{0, \ldots, 255\}^7 \;\longrightarrow\; \text{token IDs}. \]
Once actions are token IDs, the model trains with the exact next-token-prediction objective from BERT & GPT. The loss function does not know or care whether a token represents a word piece or a robot joint delta.
import numpy as np
def discretize_action(a, n_bins=256, lo=-1.0, hi=1.0):
bins = np.linspace(lo, hi, n_bins)
return np.digitize(a, bins)
action = np.array([0.42, -0.87, 0.05, 0.0, 0.0, 0.33, 1.0]) # 7-DoF continuous action
tokens = discretize_action(action)
print("continuous action:", action)
print("discretized tokens:", tokens)continuous action: [ 0.42 -0.87 0.05 0. 0. 0.33 1. ]
discretized tokens: [182 17 134 128 128 170 256]The loop shown above is the perceive-reason-act cycle made concrete: vision and language enter through the same kind of encoders as any multimodal LLM (Multimodal), the Transformer backbone processes them in one shared context, and the only genuinely new component is the last mile: detokenizing predicted action tokens into a continuous command the actuator executes, after which the changed environment produces a new camera frame and the cycle repeats.
RT-2 (Brohan et al., 2023) fine-tunes a pretrained vision-language model (the same ViT + LLM combination from Multimodal) by adding action tokens to its output vocabulary and training on robot-demonstration data interleaved with the web-scale vision-language data the base model already saw. This co-training is the central finding: mixing in the original web data, not just robot data, lets the model transfer general visual and semantic knowledge (recognizing an object it never saw a robot manipulate, for example) into the control task, an emergent-generalization result analogous to emergent abilities in language models.
OpenVLA (Kim et al., 2024) is an open-source replication of this recipe: a Llama-2 backbone with a vision encoder, fine-tuned on a large open cross-embodiment robot dataset. It demonstrates that the VLA recipe is reproducible outside closed labs and generalizes across different robot hardware (“embodiments”) without redesigning the architecture per robot.
Robot data is expensive to collect; there is no equivalent of “scrape the web” for real-world manipulation trajectories. Two mitigations dominate practice:
| Component | Role | Lives on as |
|---|---|---|
| Vision-language backbone | Perception + instruction understanding | Reused unchanged from Multimodal VLMs |
| Discretized action tokens | Bridge continuous control into the token vocabulary | RT-2, OpenVLA, and most current VLA systems |
| Autoregressive decoding | Action sequence generation | Same mechanism as Decoding, applied to motor commands |
| Web-scale co-training | Transfer general knowledge into control | The generalization story behind RT-2’s headline results |