53  Reasoning and Test-Time Compute

TipTL;DR

Test-time compute is a second scaling axis, independent of model and data size (Snell et al., 2024): chain-of-thought (Wei et al., 2022), self-consistency (Wang et al., 2022), and outcome-reward RL training (Jaech et al., 2024) all trade inference compute for accuracy on hard problems.

Depends on: Scaling Laws & Emergence

53.1 Why this matters

Scaling Laws introduced chain-of-thought (Wei et al., 2022) as a side effect of scale: large models spontaneously produce better answers when allowed to write intermediate reasoning steps first. Reframed as a resource rather than a side effect, reasoning tokens are compute spent at inference time, a distinct lever from the parameter- and data-scaling this book has covered so far.

53.2 The mechanism

53.2.1 Chain-of-thought as spent compute

Each reasoning token generated before an answer is additional computation on that specific problem. A longer chain-of-thought trades latency and inference cost for a higher chance of a correct answer.

Snell et al. (Snell et al., 2024) show that, for a fixed total compute budget, spending it at test time (more reasoning) can outperform spending it at train time (a bigger model) on some problems, establishing test-time compute as an independent lever rather than a byproduct of scale.

53.2.2 Spending test-time compute: two strategies

Strategy	Mechanism	Compute spent on
Self-consistency (Wang et al., 2022)	Sample several independent reasoning chains; take the majority-vote answer	Redundancy
Tree search	Branch at intermediate steps, evaluate partial chains, prune the weaker branches	Exploration

Tree search over reasoning traces mirrors the explore-then-commit structure in Agent Architectures’ planning mechanisms, applied to token sequences rather than external actions.

import numpy as np
rng = np.random.default_rng(0)

def self_consistency_vote(answers):
    values, counts = np.unique(answers, return_counts=True)
    return values[np.argmax(counts)]

chains = np.array([42, 42, 17, 42, 39])   # 5 independent reasoning chains
print("chain answers:", chains)
print("self-consistency vote:", self_consistency_vote(chains))

chain answers: [42 42 17 42 39]
self-consistency vote: 42

53.2.3 Training for reasoning: outcome-reward RL

OpenAI o1 (Jaech et al., 2024) trains the reasoning process with reinforcement learning (Deep RL) rather than prompting for it as a fixed instruction.

Reward type	Supervises	Cost
Process reward	Each intermediate reasoning step	Requires step-by-step human-labeled traces
Outcome reward (o1)	Only the final answer’s correctness	Cheap to collect; sparser training signal

Outcome-reward training sidesteps collecting step-level labels, at the cost of a sparser signal: a wrong reasoning path can still land on a correct final answer, and the reward alone cannot distinguish sound reasoning from a lucky guess.

53.2.4 What to observe

• Test-time and train-time compute are independent levers (Snell et al., 2024).
• Self-consistency spends compute on redundancy; tree search spends it on exploration, paralleling the debate-versus-decomposition distinction in Multi-Agent Systems.
• Outcome-reward RL (o1) reuses the PPO-style machinery from Deep RL, with “was the final answer correct” as the reward instead of a human-preference score (RLHF).

53.3 Outlook

How far outcome-reward RL training generalizes beyond verifiable domains (math, code) to open-ended tasks without a clean correctness signal is an open question as of this writing.

NoteKey takeaway

Chain-of-thought, reframed as spent compute, opens a second scaling axis alongside model and data size. Search and self-consistency spend that compute at inference; outcome-reward RL spends it during training, without ever labeling what correct reasoning looks like step by step.

Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv Preprint arXiv:2203.11171.

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPS).

Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. (2024). OpenAI o1 system card. arXiv Preprint arXiv:2412.16720.

Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv Preprint arXiv:2408.03314.