AI Agents, In Plain English

Course Overview: How AI Models Learned To Think

This opening lecture tells how AI language models went from clever autocomplete to tools that think through a problem and then carry out a job on their own. For anyone building a business assistant, it is the map of what is under the hood and why each piece exists.

Watch

Stanford CS329A Part 1: Course Overviewabout 70 minutes

Plays on YouTube, in a new tab

Listen for

• At 20 min, the "Large Language Monkeys" result, where a small model asked many times beats a top model asked once. At 28 min, how "thinking" models were built by feeding a model's own best answers back into its training. At 42 min, what turns a chatbot into an agent. At 50 min, the "generator-verifier gap", why checking answers is the real bottleneck.

What To Know

• The course is taught by Azalia Mirhoseini (Stanford professor, formerly Google Brain, DeepMind, Anthropic) and Aakanksha Chowdhery (Stanford adjunct, researcher at Reflection AI), for the second time. Website: cs329a.stanford.edu.
• Bigger has meant better. Three things improve a model predictably: more computing power spent training it, more text to train on, and more parameters (the model's adjustable internal settings). Each lowers error on a smooth curve. These curves are called scaling laws.
• Model size exploded from 2018 to 2024: BERT had 340 million parameters, GPT-2 1.5 billion, GPT-3 175 billion, PaLM 540 billion, and GPT-4 is estimated in the trillions. Around 2024 this route began to level off.
• Bigger models gained three things: higher test scores, few-shot learning (learning a task from a handful of examples placed in the question, no retraining), and emergent behavior (new skills that appear only past a certain size and could not be predicted).
• Chain of thought is the most important emergent skill. If the worked example you give shows the reasoning steps, not just the answer, the model copies that habit and solves new problems better. The lecture's example: five tennis balls plus two cans of three each equals 11.
• Chain of thought only helps large models. At around 7 to 8 billion parameters, older models got nothing from it; the benefit switched on at larger sizes. Other skills, such as unscrambling words, also appeared suddenly at a certain size.
• ChatGPT launched in November 2022 and reached 1 million users in 5 days. The two additions beyond raw size that made it work were instruction tuning and learning from human feedback (RLHF).
• A freshly pre-trained model knows statistics about the world but has no sense of right and wrong and does not follow instructions. Alignment, steering it toward human goals and values, is not a solved problem.
• In RLHF, people compare pairs of model answers and pick the better one. Those votes train a reward model, a stand-in judge that scores answers for correctness, helpfulness, specificity, or harmlessness, weighted by what the company cares about. The main model is then adjusted until the judge approves.
• Until about a year and a half ago, all the gains came from training. Then inference scaling arrived: spending more effort at answer time, with the model itself unchanged.
• The "Large Language Monkeys" study from Mirhoseini's lab had a model attempt each problem from 1 time up to 10,000, then used a checker to pick a correct one. Small open models (Llama 3 at 8 billion parameters and a 7 billion one) lost to GPT-4o on one attempt but beat it given many. Models know far more than a single ask reveals.
• For some problems only 3 or 4 of the 10,000 attempts were right. The share of problems solved rises in a straight line each time attempts multiply by ten (a log-linear curve).
• Temperature is the dial that makes answers vary between attempts. Past about 1.2 the output turns to nonsense.
• Many attempts can run at once, so waiting time is not the main cost; computing cost is. Where no automatic checker exists, AI judges and reward models stand in; a later lecture covers checkers.
• OpenAI's o1 (September 2024) showed the same straight-line gain on hard competition math as thinking time grew, but measured on the model's first answer, not its best of many. DeepSeek followed in December 2024.
• The breakthrough behind thinking models: use many attempts plus a checker to generate piles of correct solutions, then train the model on them. Attempt, check, keep the good ones, retrain. That loop is the "self-improving" idea the whole course is about.
• Thinking models work in stages: analyze the problem, break it into smaller tasks, try something and use feedback (run the code, use a calculator), correct themselves mid-stream, and back up to try another route. They beat GPT-4o on math, data analysis, and programming, but not on personal writing or editing.
• Models prefer their own reasoning over reasoning handed to them by a stronger model. Bigger models reason better, so if you split the work, the big one thinks and a smaller one summarizes.
• Chain of thought was discovered, not designed. It first showed up on a grade-school math test and then, in PaLM, in the model explaining jokes. Today's reasoning models are deliberately trained to think, including when a question needs lots of thought and when it does not.

How It Works, Step By Step

How a chat model is built, in order

1. Pre-training: the model reads enormous amounts of text and learns to guess the next word. Cheapest step, no sense of right or wrong yet.
2. Fine-tuning on high-quality text: same next-word training, but on carefully chosen books and essays that companies pay millions, sometimes hundreds of millions, of dollars for.
3. Instruction tuning: the model studies pairs of instructions and good answers (for example, a question about the boiling point of nitrogen with its answer), including examples that show the reasoning steps. A mix of human-written and machine-generated examples.
4. Learning from human feedback (RLHF): people compare answer pairs, a reward model learns their taste, and the main model is tuned until the reward model approves.
5. Inference scaling: at answer time, make many attempts and let a checker choose. The model is not changed.
6. Self-improvement loop: keep the attempts the checker approved, train on them, and repeat. This produced o1, DeepSeek, and Gemini's thinking modes.

How an agent carries out a job

1. Receive a goal and, where the request is vague, clarify what the person actually wants.
2. Plan the steps.
3. Act: call tools (search the web, open a file, run code) and read the result.
4. Check the result, often against a test the agent wrote itself.
5. Correct and repeat, remembering the goal across steps, until done or until it reports that it cannot finish.

Words To Know

Parameters

the millions or billions of adjustable settings inside a model; more means bigger.

Scaling laws

the predictable rule that more computing, data, or parameters lowers error.

Pre-training

the first stage, learning to predict the next word from huge amounts of text.

Instruction tuning

training on instruction-and-answer pairs so the model follows directions.

RLHF

reinforcement learning from human feedback; people's answer rankings train a judge that then coaches the model.

Reward model

the automated judge built from human rankings.

Chain of thought

writing out the reasoning steps before the answer.

Emergent behavior

a skill that appears only once a model passes a certain size.

Inference (test-time) scaling

spending more effort when answering, without retraining.

Verifier

anything that checks an answer automatically, such as a code test.

Temperature

the dial controlling how much answers vary between attempts.

Agentic workflow

a hand-built chain of model calls, tools, and checks: prompt chaining, routing, parallel calls, a manager model directing helpers, or a judge model reviewing output.

Thinking Longer: Test-Time Compute Scaling

This lecture is about making a model smarter at the moment you use it, without retraining it: ask it many times, let it revise, and pick the best answer. It explains why a cheaper model with more tries can beat an expensive one, and why the whole trick lives or dies on how well you can check answers.

Watch

Stanford CS329A Part 2: Test-Time Compute Scalingabout 63 minutes

Plays on YouTube, in a new tab

Listen for

• Listen for, at 2 min: a small open model, asked the same question hundreds of times, out-solves GPT-4o answering once.
• At 16 min: majority voting flattens out while a perfect checker keeps climbing: the generation-verification gap.
• At 38 min: when extra thinking at answer time beats buying a bigger model, and the one case where it does not.
• At 52 min: fusion (one model blending several drafts into one answer) beats even a perfect picker.

What To Know

• A model's life has three stages: pre-training (months, huge computer time, trillions of words), fine-tuning (orders of magnitude less data), and inference (using the model). This lecture works on the third stage only; the model's settings never change.
• The core idea of the "Large Language Monkeys" paper is repeated sampling: ask the same question 10, 100, or 1,000 times, then let a verifier (anything that can check an answer) pick the correct one.
• With enough tries plus a checker, weaker models (Llama 3 at 8 billion and 70 billion settings) beat GPT-4o answering once, on hard math, coding, and question-answering tests. The lecturer's read: small models often know the answer, they just do not give it first time.
• On SWE-bench (a test that mimics a software engineer fixing code), an open DeepSeek model given 1,000 tries solved more problems than Claude 3.5 or o1-preview. Since code can be checked with unit tests (small automatic tests of a program), the whole loop runs with no human.
• "Coverage" means the share of problems solved by at least one of the tries. It rises steadily as tries go from 1 to 1,000.
• Coverage versus number of tries follows a predictable curve (a power law), like the pre-training scaling laws from Lecture 1. Two fitted numbers predict how many tries a target coverage needs, so cost can be budgeted ahead. The curve held for every model tested, 70 million to 70 billion settings (Llama 3, Gemma, Pythia), across every subject.
• Why the curve exists: if one try is right with chance p, at least one of k tries is right with chance 1 minus (1 minus p) to the power k. That becomes a smooth power law across a whole test only when the test has a long tail of very hard problems, and real tests do: most fall on try one, then a thin tail gets harder and harder.
• The money picture has flipped. Companies used to spend hundreds of millions or billions on pre-training and almost nothing per question. Now a big share of spending can move to answer time, much of it offline while an agent keeps working.
• Checking is easy in some fields: formal proof tools for math, unit tests for code (writing a test is often easier than writing the program), and translation between computer languages, where you run both versions on the same inputs and compare outputs. The lab's "AI as a compiler" project (turning PyTorch code into CUDA, the language of graphics chips) has a perfect checker for that reason.
• Where no checker exists, the model produces far more right answers than you can catch. Majority voting (pick the answer that showed up most often) flattens after about 10 to 50 tries while a perfect checker keeps climbing. Trained scoring models (reward models, next lecture) narrow the gap but do not close it; it is wider on the harder MATH test than on GSM8K. This is the generation-verification gap.
• Majority voting fails on the hardest problems because the correct answer may appear once or twice in 1,000 or 10,000 tries, and rare answers never win a vote. The lab released all 10,000 tries per problem so others can attack the gap. Hand-checking found roughly 97 to 98 percent of "solved" math problems truly correct; weak unit tests are a known failure mode.
• A second paper, from Google DeepMind, compared two ways to spend answer-time effort: parallel (many independent tries) and sequential (one draft the model keeps revising). Reasoning models now revise internally; any instruction-tuned model revises if asked.
• Two kinds of scoring models: an outcome reward model scores only the final answer; a process reward model scores each step of the working. Step scores allow a beam search: 4 drafts per step, keep the top 2, extend, repeat. Step scorers work best on the task they were trained on but carry over somewhat.
• Results on the MATH test (about 12,000 training problems, 500 test problems, a PaLM model): majority voting worst, outcome scoring better, process scoring better still, a tuned mix of parallel plus sequential best of all. Problems were sorted into 5 difficulty bins by how often one try succeeded.
• On easy problems, leaning sequential (more revising, fewer parallel drafts) gave the best accuracy. On the hardest two bins the best ratio shifts and is still open.
• For easy and medium problems, spending more at answer time beat spending the same on a bigger pre-trained model. For the hardest problems, bigger frontier models still won, even with a very large answer-time budget. Pre-training is paid once and answer-time cost every time, but not everyone can afford to pre-train.
• The Archon paper treats all of this as a design problem: given target tests, a call budget, available models, and a menu of techniques, an optimizer called ITAS (Inference Time Architecture Search) assembles a layered recipe. The techniques are all prompts, no training: generator, fuser, critic, ranker, verifier, unit-test generator, and unit-test evaluator (the model judges a draft against the tests without running them).
• Ranking by the model beat random picking; a perfect picker beat ranking; fusion beat the perfect picker; ranking to the top 5 then fusing beat everything. The same order held with drafts from 1 to 10 different models.
• Deeper stacks helped: several models generating, then 3 rounds of critic plus fuser, then a final fuser, beat the best model once, or the best model 8 times with one fusion layer, by a wide margin. The lecturer compares it to adding layers in a neural network.
• Using only open-source models, Archon matched or beat the closed frontier models of the time by a large margin on many tests, while still handing back one answer.

How It Works, Step By Step

1. Repeated sampling: send the same question many times (10 to 1,000), collect every answer, run each through a checker, and return the one that passes.
2. Beam search with a process scorer: generate 4 partial solutions, score each step, keep the top 2, extend those, score again, and continue until a full answer is reached.
3. Archon's search: the first layer is always generators; a Bayesian optimizer (one that learns from each trial what to try next) picks one technique per later layer, always a critic before a ranker or fuser and an evaluator after a unit-test generator; designs are scored by accuracy per model call on a training set and the best one for the budget is kept.

Words To Know

Inference

using a trained model to answer; also called test time, so "test-time compute" is effort spent while answering.

Verifier

anything that checks whether an answer is right: a test, a tool, or another model.

Coverage

share of problems where at least one try was right.

Majority voting

pick the answer that appeared most often among the tries.

Outcome reward model

a trained scorer that grades only the final answer.

Process reward model

a trained scorer that grades each step of the working.

Beam search

keep only the top few partial answers at each step and extend those.

Fusion

show a model several drafts and ask it to write one combined answer.

Checking The Work: Robust Verification

Last lecture showed that if you ask an AI model a hard question many times, one of the answers is usually right. This lecture is about the other half: building a checker that picks the right answer out of the pile, or grades the work as it goes. For anyone running an AI assistant, the checker is what turns "right once in twenty tries" into "hands you the right one."

Watch

Stanford CS329A Part 3: Robust Verificationabout 73 minutes

Plays on YouTube, in a new tab

Listen for

• the surprise that a big answer-writer with a small checker beats the reverse; why accuracy falls once the checker has 800 answers to sort; the three-continuation example that shows how machines can grade steps without people; and the closing chart where a small open model plus a pool of checkers matches a model roughly nine times its size.

What To Know

• The problem is the generation-verification gap: the model can produce a correct answer somewhere in many tries, but cannot tell which one it is.
• Four papers in date order, each changing the method: train one checker on whole answers (OpenAI, 2021), grade every step with human labels (OpenAI, 2023), grade every step with no human labels (Math-Shepherd), then stop training checkers and combine many existing ones (Weaver, Stanford, 2025).
• Paper one created GSM8K: 8,500 grade-school math word problems that each need a few reasoning steps. It is still a standard test for smaller models and a good model for building your own tests.
• Its verifier recipe: fine-tune the answer-writer for 2 passes, have it write 100 answers per question, mark each right or wrong against the human answer key, train the checker for 1 pass on those labels. At test time, write 100 answers, score them, return the top scorer.
• The checker trained on two goals at once (predict right or wrong, and keep predicting the next word), and both together worked better. It scores every word, but only the last word's score decides the verdict.
• With enough training data the checker beat plain fine-tuning on both the 6B and 175B models; with under about 1,000 examples it added little. A checker also leaves the base model general, where fine-tuning bends it toward one task.
• Big answer-writer plus small checker beat small answer-writer plus big checker. Writing a solution is harder than judging one.
• Gains continued up to about 400 answers per question, then accuracy fell: with too many near-identical answers the checker could not separate right from wrong. They settled on 100. Majority voting from last lecture stopped helping around 50.
• Paper two, "Let's Verify Step by Step," compared an outcome reward model (ORM: one grade for the whole answer) with a process reward model (PRM: a grade per step, multiplied together). Humans labelled each step correct, incorrect, or neutral, producing PRM800K, about 800,000 open step labels on GPT-4 output.
• Sending labellers the "convincing wrong" answers first (right final number, broken steps) made labelling 2.6x more data-efficient than random picks.
• PRM beat ORM, which beat majority voting; majority voting stalled at about 100 samples. The PRM found right answers on problems where fewer than 5% of samples were correct, needed far fewer labelled problems, and generalised to new problem types better than both. Majority voting generalised better than the ORM.
• Step grading catches false positives, where the final number is right but the work is wrong. Newer systems combine PRM and ORM, and the PRM threshold becomes one more setting to tune.
• Math-Shepherd dropped the human labellers: a step is scored by how often continuing from it reaches the right final answer. Hard estimate: good if any of N continuations succeed. Soft estimate: the fraction that succeed. With N = 3 and two successes, hard gives 1, soft gives 2/3.
• Weak spots: hard problems give no signal because every continuation fails; unusual valid approaches score low; a wrong middle step can be marked good if the answer still lands. Best results came at N = 4; hard and soft scored about the same, so they used hard.
• Math-Shepherd's PRM beat the human-labelled PRM800K checker on the harder MATH test, across three open models. Training Mistral-7B with reinforcement learning against it beat the same training against an ORM, with verification over 256 answers, though gains plateaued.
• Weaver assumes every checker is imperfect ("weak") and combines a pool: trained reward models plus plain LLM-as-judge prompts. Averaging the top 5 or 10 helped unevenly; learning one weight per checker with simple statistics helped every time.
• Weaver's unsupervised version uses only 1% of training labels and beat majority voting and Multi-Agent Verification (rubric prompting, no better than majority voting). On hard tests accuracy rose from just over 40% to over 70%.
• Llama 3.1 8B with checkers of 8B or smaller averaged 70%, matching majority voting on a 70B model. At the 70B class Weaver hit 86.2%, in o3-mini's range.
• Running many checkers on 100 answers is costly, so Weaver was distilled into a 400 million parameter model keeping 97% of the accuracy at over 99% less compute. Both are open-sourced.
• Ideas raised for later: give the checker a rubric or calculator; for code, have the model write its own tests as the checker (CodeMonkeys); the long-term goal is right on the first try, at the risk of losing creativity.

How It Works, Step By Step

1. Collect questions with known answers (for math, a human answer key).
2. Have the answer-writer produce many answers per question (100 in paper one).
3. Label each answer right or wrong against the key. For step grading, label every step: by humans (paper two) or by sampling N continuations from each step and counting how many finish correctly (Math-Shepherd).
4. Train the checker on those labels. Score a whole answer by its last-word score (ORM) or by multiplying step scores (PRM).
5. At test time, write many answers, score each, return the top scorer. Optionally use the checker as a reward to retrain the answer-writer, then repeat.
6. Weaver's version: gather existing checkers, drop the ones that score badly on a tiny labelled set, learn a weight for each from how they agree and disagree, combine, select. Then distil the combination into one small model.

Words To Know

Verifier (checker)

a second model that reads an answer and says how likely it is to be correct.

Generator (answer-writer)

the model producing the candidate answers.

Outcome reward model (ORM)

a checker that gives one grade for the whole answer.

Process reward model (PRM)

a checker that grades each step; the answer's score is the product of the step scores.

Majority voting / self-consistency

ask many times, pick the most common final answer; no checker involved.

Repeated sampling

asking the same question many times in parallel to get many candidates.

Fine-tuning

extra training that bends a model toward one task or dataset.

Hard vs soft estimate

scoring a step by whether any continuation succeeds, or by what fraction do.

Weak supervision

learning how much to trust each imperfect labeller from how they agree and disagree, using very few true labels.

Ensemble

a pool of checkers combined into one verdict.

Distillation

training a small model to copy a large one so it runs cheaply.

LLM-as-judge

using a language model, by prompt alone, as a checker.

Learning From Feedback With Tools And Code

This lecture is about where an AI agent's feedback comes from: the outside world (a search, a running program, a test) or the model's own critique of its work. It covers three methods that each turn one of those feedback sources into a loop that makes the model better, which is the heart of any assistant that does real jobs rather than just chat.

Watch

Stanford CS329A Part 4: Learning from Feedback with Tools/Codeabout 71 minutes

Plays on YouTube, in a new tab

Listen for

• At 2 min: the think, act, observe loop humans use, and why models could not do it on their own. At 9 min: the Apple Remote question solved four ways, showing why thinking and searching must take turns. At 28 min: the coding loop, where the model runs tests, reads the failure, and tries again. At 47 min: an example "constitution" and how the model grades and rewrites its own answers.

What To Know

• Three methods: ReAct (think, then act, in turns), RLEF (learning to code from test results), and Constitutional AI (the model critiques itself against written rules). All are self-improvement loops; the difference is where the feedback comes from.
• A model that only reasons works from memory, so it makes facts up when it has none. Ask a plain model for today's temperature and it fails unless it can search.
• Before ReAct there were two camps: models that reason step by step, and models trained only to click around a web browser. ReAct joined them with nothing more than a prompt, using an older Google model that was never retrained, plus a few worked examples.
• ReAct alternates: thought 1, action 1, observation 1, thought 2, action 2. Each step feeds the next, the way a thirsty person walks to the kitchen, finds no water, then decides to go to the store (the lecturer's analogy). Two benefits: answers are grounded in real information, and the written trail can be read by a person, which builds trust.
• To keep actions valid, give the model a fixed list of allowed actions and have it pick one, like a multiple-choice question. Google robotics work used the same trick.
• Apple Remote example: plain prompting was wrong, step-by-step reasoning was wrong, action-only got lost. ReAct searched "Apple Remote", learned it was built for Front Row, searched that, failed, rephrased to "Front Row software", and got the answer. Today's "thinking mode" models do this on their own because they were trained on such trails.
• Tests: HotpotQA (questions needing several Wikipedia lookups), FEVER (checking whether a claim is true), WebShop (buying the right product on a website from a customer's instructions).
• Results: ReAct beat action-only every time. It beat reasoning-only on fact-checking but not always on the question task. Best was a combination: run ReAct, and if it fails after a set number of steps fall back to majority-vote reasoning, or the reverse.
• Reasoning-only failed mostly by making facts up; ReAct made fewer errors because it could look things up. Retraining on ReAct trails beats prompting, and adding reward-based training beats that again.
• On WebShop, ReAct scored 66.6 against 82.1 for human experts, ahead of models trained by copying humans, but with headroom left. Success rates on multi-step jobs stay low because one early wrong step cascades.
• ReAct's costs: a large menu of actions needs more examples than fit in the prompt, and every thinking step costs more to run.
• Class ideas for noisy feedback: a reflection step, backtracking, confidence scores, repeated runs. Other habits worth adding: breaking a task into pieces, trying several approaches in parallel, memory, and tuning how much to think before acting, since some models overthink simple requests.
• RLEF trains a coding model with execution feedback: the code is actually run against tests, and the reward is pass or fail with no partial credit.
• Two tiers of tests: public tests give quick feedback while drafting; private tests, hidden from the model, decide the reward, so it cannot memorise answers. The model writes code one word at a time, but one score covers the whole turn.
• Shown on Llama 3.1 trained on competitive programming problems. Solve rates rose clearly on a chart whose sample-count axis grows tenfold per step, and the gains carried to coding tests it was not trained on.
• The win is targeted repair: with RLEF the model has fewer wrong answers on turns 1, 2 and 3, and its edits aim at the error it was shown. An untrained model does not improve just from seeing its failed code and the test result. Reward-based training tends to help on new kinds of problems; ordinary fine-tuning (showing good examples) helps mainly on problems like the ones it saw.
• RLHF, the older safety method, has humans rank pairs of answers, trains a reward model to copy their taste, then trains the main model against it. It needs tens of thousands of human labels.
• Constitutional AI replaces that labour with 16 written principles. Humans only write the rules; it works because models follow instructions well enough to grade and rewrite their own answers.
• More revision rounds make answers safer; helpfulness slips a little; the combined score keeps rising. The best balance came from constitutional reward-based training with chain-of-thought reasoning; the critique-and-revise stage alone was worse than plain RLHF.
• Post-training is roughly 5% of a model's computing cost, so refreshing a constitution is affordable. Making a model truly forget an old rule is unsolved. Models grading themselves are often overconfident; a panel of other models sometimes grades better.

How It Works, Step By Step

1. ReAct loop: read the goal, write a thought, pick one action from the allowed list, receive the observation, add all of it to the prompt, and repeat until the model chooses "finish".
2. RLEF loop: receive the problem in plain words, write code, run the public tests, read the pass or fail, rewrite, repeat until it passes or the turn limit is hit, then run the hidden private tests, turn that into a reward, and update the model.
3. Constitutional AI: write the principles, generate answers to provocative prompts, critique each against a principle, revise, fine-tune on the revisions, then train a preference model from the AI's own judgements and train the main model against it.

Words To Know

ReAct

a prompting pattern where the model alternates a written thought with a real action.

Observation

what comes back after an action, supplied by the world, not the model.

Grounding

tying an answer to real, checked information instead of memory.

Chain of thought

the model writing out its reasoning steps before answering.

Self-consistency

asking the same question many times and taking the majority answer.

Action space

the menu of moves the agent is allowed to make.

Execution feedback

the result of actually running code or its tests.

Public and private tests

the tests the model sees while drafting, and the hidden ones that score it.

RLHF

training a model on human rankings of its answers.

Constitution

a short written list of principles the model grades itself against.

Preference model

a judge model trained to predict which answer is better.

Planning And Multi-Step Reasoning

This lecture covers three ways to make an AI agent better at jobs that take many steps: searching through possible plans like a game tree, teaching a model to work on independent steps at the same time, and training it step by step so it learns when to use a tool and when to stop. Multi-step work is where agents break, and this is what the research says fixes it.

Watch

Stanford CS329A Part 5: Planning and Multi-Step Reasoningabout 75 minutes

Plays on YouTube, in a new tab

Listen for

• At 8 min: the six stages of the tree-search method, walked through with a maze example. At 36 min: the training recipe for parallel thinking and the accuracy and speed numbers. At 60 min: how the third method trains a model to use tools without ever running the tools during training. At 68 min: the surprise result that training on math with a calculator made the model better at search questions too.

What To Know

• Three papers are covered: LATS (tree search over actions, 2024), SPRINT (parallel planning, 2025), and SWiRL (step-by-step reward training, 2025). All three attack the same weakness: models are poor at exploring different routes and at keeping accuracy across many steps.
• LATS borrows Monte Carlo Tree Search, a game-playing technique: try several actions, score where each leads, and spend more effort on the promising branches.
• LATS has six stages in order: selection, expansion, evaluation, simulation, back-propagation, reflection.
• In LATS the score for a state adds two numbers: a judge score, where the model is asked to rate the state from 0 to 1, and a self-consistency score, which is how often that action came up when sampled many times (for example 50 samples, with one action appearing 75% of the time).
• Choosing which branch to expand uses a formula called UCT. It takes the branch's value and adds a bonus for branches visited less often than their parent. That bonus is what keeps the search from locking onto an early favorite.
• When a trajectory ends in success or failure, that result is pushed back up the tree to update the value of every state along the way. After that, the model writes a short reflection on why it worked or failed, and that reflection measurably improved results.
• LATS was tested on HotpotQA, a question set designed so every answer needs facts from at least two different Wikipedia pages, and on WebShop, a simulated shopping task. On WebShop it came close to human expert performance with no training at all, purely by spending more effort at answer time.
• LATS weaknesses: cost was never analyzed (every expansion costs money), and it assumes actions can be undone. Paying for a service cannot be tried three ways. Unlike Math-Shepherd (an earlier lecture), which scored the reasoning itself, LATS scores the outcomes of actions.
• SPRINT starts from an observation: today's reasoning models (the o1 family, Gemini 2.5 Pro, DeepSeek R1) think longer on harder problems, and longer thinking tracks with higher accuracy. During DeepSeek R1's training, accuracy and average response length rose together.
• Many steps in a long reasoning trace do not depend on each other. SPRINT teaches the model to write out independent plans together, run them side by side, then plan again.
• SPRINT training data: one model (DeepSeek R1) solved problems, then another (GPT-4o) marked each step, labeled plan versus execution, and drew a dependency map showing which steps can run at once.
• SPRINT recipe: 6,000 thinking traces on math problems, keep the ones with the most parallel steps, then supervised fine-tuning on a 7B DeepSeek-R1 distilled model.
• SPRINT results: about 3.5% higher accuracy than the base 7B model, fewer sequential steps than a 32B model, and around 40% fewer sequential tokens on math at comparable accuracy. It also improved on Countdown and GPQA Diamond, tasks it was never trained on. Savings are biggest on hard problems; on easy ones it can be worse than the baseline.
• SWiRL (Step-Wise Reinforcement Learning) trains a model to know when to call a tool, write a good query for it, stay accurate across steps, recover from errors, and know when to stop and answer.
• SWiRL never runs a tool during training. Tool results are collected once, offline, while making the data. During training a judge model rates each proposed step on its own quality, since a good search query can be judged before seeing the result.
• SWiRL setup: a 27B Gemma 2 model generated about 50,000 multi-step trajectories from HotpotQA and GSM8K (grade-school math).
• SWiRL filtering result: keeping trajectories where every step was judged good, regardless of final answer, trained better than keeping only trajectories with correct final answers. Correct-only data teaches the model what it already knows.
• SWiRL transfer: trained on math with a calculator, the model's HotpotQA score rose from 65 to about 75. Trained on HotpotQA itself with a search tool, it rose from 65 to 73. The reverse transfer (search training helping math with Python) also held.
• SWiRL scaling: going from 100 to 10,000 training examples kept improving performance on a task the model was never trained on.
• SWiRL's multi-step RL beat supervised fine-tuning by a good margin. Supervised fine-tuning needed data filtered on both process and outcome, because it copies examples; RL can give a new action a fresh reward and break out of a bad example.

How It Works, Step By Step

LATS, the tree search method

1. Selection: pick the state to expand using UCT (value plus the bonus for less-visited branches).
2. Expansion: sample several actions from that state, for example open left door, open right door, inspect the room.
3. Evaluation: run each action, record what came back, and score the new state (judge score plus self-consistency score).
4. Simulation: follow the best branch forward until it succeeds, fails, or hits the budget.
5. Back-propagation: push the end result back through every state on the path, updating its value by its visit count.
6. Reflection: have the model write what caused the success or failure and keep it for future searches.

SWiRL, the step-by-step training method

1. Prompt the model one step at a time: it may reason, call a tool, or give a final answer. Feed the tool's result back and prompt again until it answers.
2. Have a judge model rate each step given everything before it.
3. Filter the trajectories (by step quality, by final answer, both, or random).
4. Train with reinforcement learning so each action is rewarded given the prior context, with no live tool calls.
5. At use time, prompt the same way: here are your tools, here is what happened so far, call a tool or answer.

Words To Know

Trajectory

the full chain of thoughts, actions and results from a question to an answer.

Tree search

trying several actions from each point, scoring them, and growing the promising branches.

Exploration vs exploitation

spending effort on untried options versus the option that looks best now.

UCT

the selection formula that balances those two by rewarding less-visited branches.

Back-propagation

sending a final result back to update the score of every earlier step.

LLM as judge

asking a model to grade a step or state rather than a person.

Self-consistency

how often the same action shows up when you ask the model many times.

Sequential tokens

words the model must produce one after another; fewer means a faster answer.

Supervised fine-tuning

training a model by showing it example answers to copy.

Reinforcement learning

training by rewarding good actions instead of copying examples.

Process reward

a score for each step, as opposed to a score only for the final answer.

Out of distribution

a task or tool the model was never trained on.

Training Smarter: Scaling Reinforcement Learning

This lecture closes the loop: instead of only squeezing better answers out of a finished model at question time, you take the model's own checked answers and train it on them, so it gets better for good. It matters because this is how small models started beating giant ones on hard reasoning, and it is the recipe behind every "thinking" model on the market.

Watch

Stanford CS329A Part 6: Train Time Scaling/Scaling RLabout 73 minutes

Plays on YouTube, in a new tab

Listen for

• At 4 min: the one loop to remember (generate, filter, train, repeat). At 17 min: the STaR trick of handing the model the answer and asking it to work backwards. At 44 min: why GRPO drops a whole copy of the model and scores answers against their own group. At 59 min: the five fixes in DAPO, each with the score it added.

What To Know

• The lecture covers three papers in the order they appeared: STaR (teach a model to reason from its own correct answers), DeepSeekMath (a lighter form of reinforcement learning called GRPO), and DAPO (fixes that make that learning stable on long, hard problems).
• The yardstick is AIME, a competition math test. A 175 billion parameter GPT-3.5 scores about 5%. A 7 billion parameter DeepSeekMath scores 51.7% (60% with extra tricks), and DAPO on a 32-billion Qwen model scores 50%. Small models can beat big ones if trained this way.
• Three ideas carry the lecture: a model improves by learning from its own filtered outputs; training on those outputs can stand in for making the model bigger; and reinforcement learning only works at scale when the small details are right.
• The whole subject in one loop: run the question-time tricks from earlier lectures (sample many answers, vote, check), keep the answers that pass, fine-tune the model on those, repeat.
• This only works well where answers can be checked. Math and code can be checked; personal writing cannot. Thinking models beat GPT-4o over 50% of the time on programming, data analysis and math, but barely at all on writing and editing.
• Thinking models spend tokens on recognisable habits: understanding the problem first, breaking it into steps, checking their own work, backing up after a mistake, trying a second approach.
• STaR starts with a handful of worked examples, has the model write out reasoning for about 10,000 problems, keeps only the ones whose final answer is right, trains on those, and loops.
• Rationalization is STaR's second trick: for problems the model got wrong, show it the correct answer and ask it to explain how to get there, then train on that explanation with the hint removed. This lets it learn from harder problems.
• STaR was tested on a 6 billion parameter open model (GPT-J) using grade-school math (GSM8K, about 9,000 problems), everyday multiple-choice questions (CommonsenseQA) and made-up multi-digit addition.
• STaR results: it threw away a lot of data (only 70 to 87% survived the filter), it plateaus after a few rounds because it is not true reinforcement learning, and human raters found its explanations reasonable on the everyday questions. With rationalization it reached 72.5% on those questions using 86% of the data a plain fine-tune would need.
• On the easy grade-school math, rationalization did not help; the problems were already within reach, and the number of calculation steps barely changed.
• STaR's weak spots: a right answer is only a stand-in for good reasoning (wrong steps can sneak through), there is no good way to grade the reasoning itself, and the style of the few starting examples shapes every explanation the model produces.
• DeepSeekMath's first win was data: it started from a code-trained model and mined math from ordinary web pages instead of academic papers. Code-to-math helped reasoning and tool use; academic papers did not help much.
• Standard reinforcement learning (PPO) keeps four copies of a model in memory: old policy, new policy, a critic, and a reward scorer. GRPO drops the critic. For each question it samples a group of answers, scores them, and rates each answer by how far it sits from the group's average, divided by the spread. That took the MATH test score from 46.8% to 51.7%, the first open 7-billion model past 50% without a critic.
• GRPO needs a spread of scores to learn from. If every answer in a group is right, or every one is wrong, there is nothing to learn, so questions must be neither too easy nor too hard. A "do not drift" penalty (the KL term) stops the model forgetting what it already did well.
• DeepSeekMath's honest finding: with 32 tries, the share of tries that agreed on the right answer went up, but whether at least one try was right did not. The model got more consistent, not smarter.
• DAPO: plain GRPO on Qwen-32B hits only 30% on AIME, with confidence collapsing, unstable training, and answer length running away. DeepSeek's R1 recipe reached 47%.
• DAPO's score ladder on Qwen-32B: filter overlong answers 30 to 36, allow bigger upward moves on unlikely tokens (asymmetric clipping) 38, soft penalty on cut-off answers 41, score per token instead of per answer 42, dynamic sampling 50. That beat R1's distilled 32B at 47.
• Three dials to watch during this training: answer length, entropy (how much the model is still exploring), and the share of groups scoring all-correct. If scores stop moving, the reward scorer may be maxed out.
• All three methods improve consistency, answer formatting and multi-step coherence. None yet teaches a model to solve genuinely new kinds of problems.

How It Works, Step By Step

1. Prime the model. Start from a base that already has some skill in the area (DeepSeekMath started from a coding model and fed it curated web math first).
2. Pick problems with checkable answers that are neither all easy nor all hard, so the model can score some wins and some losses.
3. For each problem, have the model produce many answers with its reasoning written out (STaR: one attempt per problem; GRPO: a group of, say, 64).
4. Score them. STaR keeps only correct ones. GRPO rates each answer against the group's average and spread. DAPO throws out groups that are all right or all wrong and samples more to refill the batch.
5. For problems the model failed, STaR hands it the answer, asks for the reasoning, then hides the hint.
6. Update the model on the kept answers, scoring per token (DAPO) so long rambling answers do not outweigh short good ones, with a gradual penalty for answers cut off mid-thought.
7. Watch answer length, entropy and the all-correct share, then repeat the loop.

Words To Know

Train time scaling

improving the model itself by training it on its own filtered outputs.

Test time scaling

spending more effort at question time (many tries, voting) without changing the model.

STaR

Self-Taught Reasoner: generate reasoning, keep what was right, train, repeat.

Rationalization

giving the model the answer so it can write the reasoning backwards for problems it missed.

Reinforcement learning (RL)

training by reward: the model tries, gets scored, and shifts toward what scored well.

Policy

the model being trained, in reinforcement-learning talk.

Critic

a second model that guesses how good a partial answer is; GRPO removes it.

GRPO

Group Relative Policy Optimization: score each answer against its own group's average and spread.

Advantage

how much better than average one answer was; the number the update is built on.

Entropy

how spread out the model's choices are; low means it has stopped exploring.

Dynamic sampling

oversample, then drop groups that give no learning signal.

Pass@K vs majority@K

pass@K: any of K tries right; majority@K: most tries agree on the right answer.

Self-Improvement And Deep Research Agents

Two ways of searching for a good answer when a model could produce many: generate a huge pile of candidates and pick the best (AlphaCode), or let a thinking model look things up mid-thought when it hits a gap in what it knows (deep research). The right answer is usually somewhere in what the model could say; the hard part is finding it.

Watch

Stanford CS329A Part 7: Self-Improvement and Deep Research Agentsabout 72 minutes

Plays on YouTube, in a new tab

Listen for

• At 6 min: one million candidate programs per problem, then filtering and grouping before anything is submitted. At 17 min: why "pick the best 10" is the bottleneck. At 28 min: AlphaCode 2 throws away 95 percent of its samples and a scoring model picks winners. At 50 min: one chemistry question answered three ways, showing why pasting documents into the prompt is not enough.

What To Know

• The theme is search: a model that can produce the right answer somewhere in a thousand tries is useless until you can find that one and hand it in.
• AlphaCode (DeepMind, 2022) tackled competitive programming: a written problem plus a few example inputs and outputs, and the job is to write a whole working program. That is far harder than an autocomplete finishing one line.
• AlphaCode placed in the top 54% of human entrants across 10 live contests of about 5,000 people each, with 10 submissions allowed per problem. It was the first proof that an AI could solve whole problems end to end.
• The recipe: train its own model on about 700 GB of public code, then fine-tune on a bank of contest problems with known solutions (CodeContests), with a training trick called GOLD that sharpens precision.
• Then it samples at a scale nobody had tried: 1 million candidate programs per problem, half Python and half C++, with the prompt's difficulty labels shuffled and a high randomness setting so the programs differ from each other.
• Only a handful can be submitted, so there is a selection stage. First, keep only programs that pass the example tests in the problem. Second, group the survivors by behaviour: a separate model invents fresh test inputs, and programs that give the same outputs count as the same idea written differently. Submit one program from each of the biggest groups.
• The score is "10 at k": generate k programs, submit 10. "Pass at k" means unlimited submissions, a pretend scenario. Comparing the two shows what the selection step costs.
• The 41 billion parameter model beat the 9 billion one at every budget, and 41 billion plus grouping beat both. Solve rate climbs by a steady step each time samples are multiplied by ten, the same pattern seen in earlier lectures, and better models have steeper lines.
• Selection is the bottleneck: with unlimited submissions the solve rate passed 40%, with 10 submissions it stopped near 30%. More samples only help if they keep being different, which is why grouping matters.
• AlphaCode weaknesses: enormous sampling, weak on some problem types (dynamic programming, constructive algorithms), and it worked in one shot rather than in steps.
• AlphaCode 2 (Google, 2023) changed three things. It started from an existing model (Gemini Pro) and fine-tuned it instead of training from scratch. It trained a family of variants, each on a different slice of the problem bank, so samples would be diverse. And it trained a separate scoring model, on a smaller hand-checked set of problems, that rates each program's chance of being correct from 0 to 1.
• AlphaCode 2 pipeline: sample across the family in C++ only with randomised settings, run everything on the test inputs, throw out anything wrong or that will not run (removes 95%, leaving about 50,000), keep the 10 largest behaviour groups, score every program, submit the best from each group.
• AlphaCode 2 matched AlphaCode's solve rate with 100 samples instead of 1 million. At 1 million it solved 43% against AlphaCode's 25%, almost 2x. Against humans it reached the 85th percentile, up from 46th; best of its top two tries beat 99.5% of entrants.
• Lesson: a stronger base model plus a learned scoring model beats piling on samples. The scoring model trains on data kept separate from the generator's, and learns "given two solutions to this kind of problem, which is better."
• Cheaper routes raised in class: refine a few answers using feedback, use fewer samples on easy problems, and break hard problems into parts with hints for each.
• Deep research agents sit on top of large reasoning models (models trained to think at length before answering). Those models have a knowledge cutoff, and when they hit a gap the gap shows up as hedging words like "perhaps" or "wait" in their thinking, and the guess poisons every later step.
• Plain retrieval (look up once at the start, paste the documents in, answer) helps on simple questions but fails when several rounds of look-up are needed. Agentic retrieval lets the model search mid-thought whenever unsure, but raw documents are long and noisy and 10 or 20 of them swamp the model.
• Search-o1 adds one step: after each search it reads the documents, pulls out only the part relevant to the current question and the thinking so far, and adds just that. In the chemistry example (count carbon atoms in a product after several reactions), plain thinking guessed wrong, pasted documents still got it wrong, Search-o1 got it right.
• Search-o1 results: more documents make it better, while plain thinking and plain retrieval flatline. On the GPQA expert exam it matched or beat human experts in physics, came close in biology, trailed in chemistry (compare each subject to its own experts). On multi-step question sets (HotpotQA, 2WikiMultihop, MuSiQue, Bamboogle) it posted the top numbers. Hedging words in its thinking dropped sharply.
• Search-R1 is the follow-on: Search-o1 is driven by prompting, Search-R1 uses reinforcement learning to teach the model when and how to search.
• Models are overconfident: one that is right half the time may act 80% sure and dig in when corrected. Knowing what they do not know is open research.

How It Works, Step By Step

1. AlphaCode 2 generation: a family of fine-tuned models writes candidate programs with randomised settings, up to 1 million per problem.
2. Run every candidate on the known test inputs; discard anything that fails or will not run (about 95 percent).
3. Group the survivors by behaviour on freshly invented inputs; keep the 10 largest groups.
4. The scoring model rates every program; the top-rated one in each group gets submitted.
5. Search-o1 loop: the model thinks, and when its thinking shows uncertainty about a fact it writes a search query inside special markers.
6. A search tool fetches documents; a reading step extracts only the chunk relevant to the query and the thinking so far.
7. That chunk is added to the thought process and the model keeps going, repeating steps 5 to 7 as needed before the final answer.

Words To Know

Sampling

asking the model for many different answers to the same question.

Solve rate

the share of problems where a submitted answer was correct.

10 at k

generate k answers, hand in only 10; measures generation and selection together.

Clustering

grouping answers that behave the same way so you submit one of each kind.

Scoring model (reward model)

a second model trained to rate how likely an answer is to be correct.

Fine-tuning

extra training of an existing model on one kind of problem.

Large reasoning model

a model trained to think at length, step by step, before answering.

Retrieval augmented generation (RAG)

looking up documents and pasting them into the prompt before answering.

Knowledge gap

a fact the model never learned or that changed after training; shows up as hedging.

Multi-hop question

a question that needs several separate look-ups chained together.

Grading Agents: Evaluations And Long-Horizon Tasks

This lecture is about how to measure whether an AI agent is really getting better at long, real-world jobs, and why the old chatbot tests no longer tell you much. An agent that looks impressive in a demo can still fail half the time on a one-hour task, and you cannot fix what you do not measure.

Watch

Stanford CS329A Part 8: Agentic Evaluations and Long Horizon Tasksabout 75 minutes

Plays on YouTube, in a new tab

Listen for

• At 14 min: the task length an AI can finish has doubled every seven months, from two seconds in 2019 to 59 minutes in 2025.
• At 20 min: the reliability catch. Demand an 80% pass rate instead of 50% and that hour drops to about 15 minutes.
• At 35 min: the OpenAI work benchmark, where the best model wins or ties against ten-year professionals on 47.6% of real tasks; at 42 to 44 min the occupations already near par.
• At 54 min: the Stanford research-writing benchmark that no AI system has scored above 19% on.

What To Know

• Old tests (answer a question, hold a short chat) are "saturating": the models have maxed them out, so they no longer separate a good agent from a great one.
• The lecture uses three newer yardsticks: how long a task the agent can finish (METR), whether its work beats a paid professional's (GDPval), and whether it can research a topic and write it up with sources that check out (DeepScholar Bench).
• METR's task set has three tiers: tiny actions of 1 to 30 seconds (about 66 tasks), mixed software and research jobs of 1 minute to 30 hours (about 97 tasks), and full machine-learning research projects of up to 8 hours (about 7 tasks). About 170 tasks in all.
• The yardstick for every task is how long a skilled human (around five years of experience) took to do it well. The catch: experts tend to underestimate how hard a task is, because they already know what "done" looks like.
• The headline trend: the length of task a model can finish half the time has doubled roughly every 7 months. In 2019 that was about 2 seconds; by 2023 (GPT-4) about 8 minutes; by 2025 (Claude 3.7 Sonnet) about 59 minutes. Extended forward, the forecast is a month-long task somewhere between 2028 and 2031.
• Raise the bar to 80% success and the same 2025 model manages only about 15 minutes of work, roughly a quarter of the 50% figure. That gap is the reliability headroom.
• What has driven the gains: better reasoning, better code writing, better tool use, less repeating the same move, recovering from mistakes, and keeping the end goal in mind. Students added planning-then-replanning and learning from user corrections.
• Common ways agents fail on long tasks: poor planning, wrong tool choice, wrong mental math, quitting early because they cannot tell when the task is done, and looping on a failed action. The looping was far worse in GPT-4 than in the later reasoning model o1.
• METR's caveats: models do worse on messy tasks with no single right answer, and most coding tests use public code the models have already seen. On private company code, outside contractors are 5 to 18 times slower than the people who maintain it, and the models perform like the contractor: a smart stranger, not an insider.
• GDPval, from OpenAI, grades model output against professionals with over ten years of experience. It covers 9 industries (the biggest slices of the economy), 44 occupations, and about 1,320 tasks, with 220 released publicly.
• GDPval tasks are real work: a 3D part for an assembly line, a nurse's consultation report, a reply to an angry customer, a property sales brochure, a pricing audit across purchase orders. They average about 7 hours of human time, about 70% need reference files, and are worth about $400 each on the public set.
• GDPval scores: win-or-tie rate against the expert went from 12.4% (GPT-4o, 2024) through the 20s and 30s to 47.6% for Claude Opus 4.1. That is a straight-line climb, not METR's doubling curve, because here the output must actually be good enough.
• Claude models were better at looks, formatting, PDFs, spreadsheets and slides; GPT-5 was better at following instructions, calculations and plain text work.
• The most common GDPval failure is not following instructions, including promising to read the reference file and then making things up instead. Formatting errors come next.
• For GPT-5, graders judged about half of outputs acceptable but below the expert's, about 20% genuinely better, and about 29% bad or catastrophic.
• Letting the model retry and fix its own work raised GPT-5's cost advantage to about 1.6x and speed to about 1.4x over an unaided expert, at under 10% of the expert's pay where it succeeds.
• Strip context out of the task and the win rate drops a few points, but the real loss is the model no longer knows what to work on. Humans decide the problem; the model executes.
• DeepScholar Bench, from Stanford, asks an agent to write the "related work" section of a research paper. It uses fresh papers across 22 fields, reruns monthly, and only uses papers published after the big models were trained, so nothing can be memorised.
• It grades three things: did it capture and organise the key facts, did it find the relevant and important sources, and do the citations back the claims. Human graders agreed with the scoring 70 to 80% of the time.
• No system scores above 19%. Sources judged important were found less than 12.5% of the time. Even handed the perfect reading list, systems captured only about 50% of the key facts. OpenAI's deep research wrote the smoothest prose but had weaker citation support; the Stanford baseline hit about 90% citation precision but read worse. Nobody does both well.

How It Works, Step By Step

1. Collect candidate tasks and have people vet them so each one is clearly specified and has a checkable outcome.
2. Have skilled humans do each task and record how long a successful attempt took. Average the times; get more people when they disagree a lot.
3. Run the agent on each task many times and record the share of attempts that succeeded.
4. Plot success rate against human time, fit a curve, and read off the task length where success falls to 50% (or 80%). That length is the model's "time horizon".
5. Repeat for models by release date to get the trend line, then study the failures by type to see where the next gains will come from.

Words To Know

Evaluation (eval)

a fixed test set used to score how well an agent does a job.

Benchmark

a published evaluation everyone can run, so models can be compared fairly.

Long-horizon task

a job with many steps that takes a person minutes to days, not seconds.

Time horizon

the longest task (in human time) a model finishes at a chosen success rate.

Success rate

share of attempts at one task that came back done right.

Win rate

how often graders preferred the model's work (or called it a tie) over the expert's.

Failure mode

a named, repeating way the agent goes wrong, like looping or quitting early.

Reference files

the documents, sheets or images the task says to use; ignoring them is a top failure.

Research synthesis

reading many sources and writing one coherent, cited summary.

Verifiability

whether each claim in the write-up is backed by the source it cites.

Context

everything the model has been told about the job; low context makes it act like an outside contractor.

Where This Is Going: Future Research Areas

The last lecture names the three walls self-improving agents keep hitting (practice answers that all look alike, weak checking, and running out of good training questions) and the papers trying to climb them. It closes with a money-and-power argument: most everyday AI requests could run on a laptop, and useful answers per unit of electricity will soon matter as much as raw accuracy.

Watch

Stanford CS329A Part 9: Future Research Areasabout 68 minutes

Plays on YouTube, in a new tab

Listen for

• At 6 min: why one model writing its own practice data stalls, and the fix of several models that debate. At 16 min: the "meta verifier", a checker that checks the checker. At 24 min: a model that writes its own training tasks, kept "not too easy, not impossible". At 46 min: intelligence per watt, and local models handling 88.7% of real chat requests.

What To Know

• Course in one line: an agent is a model with a goal that acts, collects feedback, and uses it to fix the next step. Most real agents today are still hand-built workflows; coding agents are the first place the agent runs the workflow itself.
• Three bottlenecks block the self-improvement loop: the practice answers are too similar, checking is weak, and the training questions are picked by hand.
• Sameness: when one model writes its own practice solutions they come out alike even at high randomness, so gains stop after a few iterations, at most tens.
• Multi-agent fine-tuning fix: train several generator models and several critic models from the same base, let generators see each other's answers, let critics pick holes, take the majority answer, and train on the matches. Across three open models on a hard maths set the multi-agent version kept improving while the single-model version flattened or collapsed; answers stayed diverse; Llama 3 responded best; gains carried to a grade-school maths set it never trained on. A cheap version is using models from different makers.
• Checking: most rewards ask only "does the final answer match the key?" That saturated contests like AIME, but a right answer can sit on wrong steps. Models asked to judge proofs call invalid proofs valid, while human experts spot the gap between steps without a reference answer.
• DeepSeek-Math V2 trains a verifier to find issues with no reference solution and score proofs at 0, 0.5 or 1, then adds a meta verifier that asks: do the flagged issues really exist, and does the score follow? Once seeded by humans the labelling runs on its own. A better verifier trains a better generator, whose harder proofs train the verifier. Built on DeepSeek V3 with the GRPO method.
• Numbers: over 8 rounds the single-try proof score kept climbing, and best of 32 attempts reached close to 42% on the 2024 Olympiad shortlist. Still limited to fields where checking is possible.
• Which questions to train on: today experts write the question-and-answer pairs, so maths needs maths experts and code needs strong coders. As models pass human level, experts run out.
• Absolute Zero: one model both proposes tasks and solves them, with no human-written prompts, in code because a computer grades the result for free. Three task shapes: deduction (program plus input, predict the output), abduction (program plus output, find the input), induction (examples plus a description, write the program).
• The proposer's reward is 1 minus the solver's success rate, zero if the solver never succeeds, so it is paid for middle-difficulty tasks and writes harder ones as the solver improves. Coach and player, mildly adversarial, improving each other.
• Results: state-of-the-art coding scores with zero human-curated prompts, beating models trained on tens of thousands of expert examples; strong maths scores though it only practised code; task complexity rose over time; bigger models gained more. The earlier SWiRL lecture saw the same: practice on look-up questions improved calculator and maths skills.
• Hard-to-check fields: where the check takes days (chip design simulations, wet-lab chemistry) or is a matter of taste (creative writing). Training needs hundreds or thousands of quick checks, so a slow one breaks the loop. The workaround is a trained stand-in that predicts the outcome from past data; it is only as good as that data and can be gamed.
• Intelligence per watt: average task accuracy divided by average power draw. "Local model" here means 20 billion or fewer active parameters.
• Demand is exploding: Google's cloud compute grew roughly 1,200x in under two years, Nvidia roughly 10x year on year, Google's monthly tokens rose from 160 trillion to 1.3 quadrillion in about 20 months, and serving it needs on the order of 250 gigawatts of data centres.
• Yet 77% of real chat requests are practical guidance, information look-ups and writing, which small models already handle. Laptop memory grew about 126x since 2012; a MacBook can hold around 100 GB, enough for very large models stored in compressed 8-bit form.
• Study scale: over 20 local models, data-centre and laptop chips, 1 million real chat queries plus three reasoning benchmarks, all released openly.
• Findings: local accuracy on real queries rose 3.1x since 2023; local models answer 88.7% of those queries; intelligence per watt improved 5.3x in two years (3.1x better models, 1.7x better chips); an Apple M4 Max still gets 1.5x less than an Nvidia B200 because laptop chips were never designed for this.
• Open directions: why repeated sampling finds correct answers at all; continual learning (models learn in offline batches, humans learn as they go); serving systems built for many-sample agent traffic; hybrid routing of easy requests to a local model and hard ones to the cloud; energy as the scarcest resource ahead.
• Memory versus retraining: if the new thing is facts, update the knowledge store; if it is a skill or reasoning in a new field, only changing the model's weights works. A robot does not learn a new body from a bigger memory.
• Simulated environments matter only as stand-ins for real tasks; games are easy to simulate because the space is finite.

How It Works, Step By Step

Multi-agent fine-tuning, one round of debate

1. Each generator writes its own first answer.
2. Every generator receives a summary (or the full text) of the others' answers.
3. Each critic reviews the updated answers and points out flaws.
4. The generators answer again with the critiques in hand.
5. The majority answer wins; question-and-answer pairs that match it are kept.
6. Generators are fine-tuned on the kept pairs; critics on records that started wrong and were corrected during the debate.
7. Repeat with the improved models.

Absolute Zero, one cycle

1. The proposer drafts a task in one of the three shapes, drawing on the buffer for variety.
2. The task is validated: it runs, passes safety checks, gives the same output every time.
3. The solver attempts it; the computer grades it.
4. Proposer rewarded for middle difficulty, solver for correctness; both update the one shared model, and the task joins the buffer.

Words To Know

Self-improvement loop

a model generating practice, getting graded, and training on what passed.

Rejection sampling

generate many attempts, throw away the wrong ones, train on the rest.

Diversity (of reasoning chains)

how different the practice answers are from each other; without it the loop stalls.

Outcome reward

a grade based only on whether the final answer matched the key.

Verifier

the model or program that grades an answer or finds its errors.

Meta verifier

a checker of the checker: are the flagged errors real, and does the score match them?

Proposer and solver

the two roles one model plays when it writes its own tasks and then attempts them.

Curriculum learning

feeding tasks that get harder as the learner improves.

Synthetic data

training material written by a model instead of a person.

Reward hacking

the learner finds a way to score well without doing the real job.

Intelligence per watt

average accuracy divided by average power used.

Continual learning

learning on the go from every task, instead of in occasional offline retraining batches.