Why Search Keeps Rediscovering the Same Workflow, and What That Means

Also available as a plain-text transcript page.

0:00Bella: There's a kind of small embarrassing moment you sometimes get in machine learning, where a very expensive method spends hours searching for the right answer, and at the end you look at the answer and realize... it's the same answer everyone else found, for basically the same reason, and you could have just written it down.

0:21Finn: That's the moment this paper is built around. Full title: "Why Search When You Can Transfer? Amortized Agentic Workflow Design from Structural Priors." It went up on arXiv on April twenty-seventh, twenty-twenty-six — we're recording five days later, on May second. Quick ground rules before we dig in: this is an AI-generated deep dive. The script is from Anthropic's Claude Opus 4.7. I'm Finn, that's Bella — we're both AI voices from Eleven Labs, and the show isn't affiliated with either company. Now — back to that embarrassing moment.

0:56Bella: Right. So the setup. When people talk about LLM agents these days, they usually don't mean a single call to a model. They mean a small program that orchestrates several model calls and tools. A typical math version is: send the problem to the model three times in parallel, take a majority vote on the answers, then run a regex to extract the number. A typical code version is: generate code, run unit tests, retry with the error message if tests fail. That little program is the workflow. It's a graph — nodes are model or tool calls, edges are how data flows between them. And there's a whole research area now devoted to designing those graphs automatically. The leading method, called AFlow, treats every new task as a fresh search problem. It runs MCTS — Monte Carlo Tree Search, the same family of algorithms that powered AlphaGo — over thousands of possible workflow variants, scoring each on a small validation set, keeping the best one. That works. But on the MATH benchmark, it costs about twenty-two dollars and a hundred and eighty-four minutes of optimization. Per task.

2:08Finn: Per task. To find a graph of model calls that processes math problems.

2:13Bella: Per task. And here's what the paper's authors noticed. If you actually look at what AFlow finds for math — GSM8K, MATH, MultiArith — they all converge to roughly the same shape. Sample several solutions in parallel, vote, extract the number. The code benchmarks all converge to: generate code, test, retry on failure. Domain by domain, this supposedly enormous combinatorial space collapses to a tiny family of stereotyped layouts.

2:42Finn: So search keeps spending hours rediscovering "math wants ensemble voting," and then throws that knowledge away before the next math task starts.

2:51Bella: Exactly. And the question the paper takes seriously is: if every math task is going to converge to the same shape anyway, why are we paying the search cost over and over? Could you do the discovery once, and then for any new task, just write the workflow directly, in a single LLM call, by analogy?

3:11Finn: The system that does this is called SWIFT. It's from a team at Carnegie Mellon and Notre Dame. First author SHEE-yee Doo, with senior authors including Vincent KOH-nit-zer and Carl Kingsford. And the headline number they report is, frankly, ridiculous. Where AFlow spends twenty-two dollars and three hours optimizing on MATH, Swift spends less than half a cent and under ten seconds. Same task, comparable accuracy — actually slightly better accuracy.

3:42Bella: About a five-thousand-fold reduction in optimization compute.

3:47Finn: Which sounds suspicious. We should put a flag on that, because the numbers get more nuanced when you bring execution cost into the picture. But the optimization gap is real, and it's the right place to start.

4:01Bella: Let's walk through how Swift actually does this. There are two phases — and the first one is the only place any expensive work happens. The offline phase reads search trajectories that other methods, AFlow in particular, already produced for some pool of source tasks. For each task, Swift looks at the best-performing workflow, the worst-performing one, and an intermediate. Then it asks an LLM to do contrastive analysis. What does the best one do that the worst one doesn't? The best-versus-worst contrast yields what they call compositional heuristics — rules like "for math tasks, generate multiple paths and ensemble them." The intermediate workflows are more interesting. Those are the ones that got the answer right but formatted it wrong. From those, Swift extracts what they call output contracts: strict interface rules between nodes. We'll come back to that one — it's a great little subplot. The whole offline phase happens once. It produces a small library of distilled rules.

5:10Finn: And then the online phase is what gets called when a new task arrives.

5:15Bella: Right — and "online" is generous. When a new task arrives, Swift assembles a single prompt: the operator library, the distilled rules, a few complete workflow examples from other tasks, and a brief description of the new task. Then one LLM call, temperature zero, generates the executable Python code for the workflow. No search. No iteration. No validation loop.

5:41Finn: Now I want to flag something important about the experimental setup, because if you don't have it in mind, the headline numbers can mislead you. The protocol is strict leave-one-out. When Swift is generating a workflow for, say, GSM8K, every trace of GSM8K — the training data, the error logs, the workflows previously optimized for it — is masked from the prompt. Swift literally cannot copy a GSM8K workflow when designing a GSM8K solver. And on the out-of-distribution benchmarks like MultiArith, those tasks aren't even in the source pool at all.

6:19Bella: So when Swift gets about ninety-eight-and-a-half percent on MultiArith without ever having seen MultiArith data — that has to be structural transfer. It can't be lookup.

6:30Finn: Right. Without leave-one-out, this would just be "memorize the answer." With it, you have to ask: what is actually being transferred?

6:39Bella: This is where the paper earns its keep, and it's the part I think is genuinely worth caring about. The authors don't just report "Swift is cheaper and works." They run a battery of ablations on the prompt — causal interventions on what's in the demonstrations — to figure out what the LLM is actually using when it reads them. Remove the demonstrations entirely. Just the rules and the task, no example workflows. MATH collapses to two-and-a-half percent. So the demos are doing real work; you can't hand the rules over and ask the model to figure it out. Shuffle the lines of the demonstration code — same nodes, scrambled order. MATH goes to zero. So it's not a bag of components; sequence matters. Finn — you want to set up the punchline?

7:28Finn: Sure. They take every operator name in the demonstrations and replace it with a random string. So instead of "SC ensemble," the demonstration has something like "x-q-three-l-v." Instead of "AnswerGenerate," it's gibberish. The Python is still syntactically valid, the structure still routes data the same way — but every human-meaningful label is now noise.

7:51Bella: And performance drops by about five points. Over ninety-three percent of the full system's accuracy is retained.

7:59Finn: The model doesn't need to know what "ScEnsemble" means to use it correctly. It's reading the wiring diagram, not the room labels.

8:08Bella: That's the line, and I want to give it a beat. The conventional intuition about in-context learning is that the model is reading the demonstrations as English — picking up cues from the names of things, the natural-language structure, the semantics. What this ablation says is that for workflow synthesis, it's mostly not doing that. It's reading the graph. Which node feeds which, how many parallel paths, where the loops are. The labels are largely decorative.

8:37Finn: There's a nice analogy for this. Imagine someone hands you the blueprint of a working office building, but every label has been replaced with nonsense. "Room A7" instead of "conference room." "Zone Q" instead of "lobby." You can still see the layout. There's a big open space near the entrance, smaller rooms branching off a central corridor, a kitchen-sized footprint near the break area. You can build a functioning office from that blueprint without knowing what anything is called, because the layout is what makes the building work.

9:11Bella: And the line-shuffling ablation tells you that temporal flow matters too — the equivalent of the corridor having a direction. Scrambling the order destroys it. So it's not just static topology; it's wiring plus routing.

9:24Finn: The contract story is the second mechanistic finding, and I find it satisfying in a slightly different way. The paper calls it the structural-functional gap. You can have a workflow that's logically correct, computes the right answer internally, and still fails — because one of its intermediate nodes returns the string "the answer is forty-two" when the next node was expecting just the bare number "forty-two."

9:50Bella: A parser failure, basically. The math is right; the handoff is wrong.

9:55Finn: Right. And here's the perverse part. If you train Swift's synthesis without explicit interface contracts between nodes, the LLM hedges. It can sense that something might go wrong at the handoffs, so it adds redundancy — extra ensemble voting nodes, retry branches, fallback paths. The workflow gets bigger. And, crucially, less accurate, because every additional node is another place errors can compound.

10:21Bella: The numbers are nice. On GSM8K, with explicit contracts, Swift produces a two-node workflow at about ninety-four percent accuracy. Without contracts, it produces a five-node workflow at about ninety percent. More machinery, worse performance.

10:36Finn: There's an assembly-line image that fits. Imagine a station that produces a bolt, and the next station is supposed to thread that bolt into a frame. If station one outputs "here's your bolt, sir, on a velvet cushion" and station two is expecting a bare bolt on the conveyor — the line jams, even though the bolt itself is perfectly correct. Without strict interface specs between stations, the engineer designing the line tends to compensate by adding redundant inspection points and parallel backup lines. Bigger factory, more failure modes.

11:11Bella: With contracts, you build a lean line that just works.

11:14Finn: Bella, this is where I think the authors land their best line. They argue that workflow synthesis should be treated as a compilation task with strict type-safety guarantees — not as natural language generation. That framing reframes the whole problem.

11:31Bella: Okay. Before we get too pleased with how clean this all is — Finn, push on it. What's the strongest version of the critique?

11:39Finn: I think there are four real ones, and the paper is partly aware of them. The first is about where Swift's priors actually come from. The leave-one-out protocol looks strict, but Swift is consuming search trajectories that AFlow produced. AFlow did the hard work of discovering "math wants multi-path ensemble" for the source tasks. Swift transfers that pattern. So the question is: how much of Swift's success is its synthesis design, and how much is inheriting AFlow's discoveries? If you fed Swift trajectories from a worse search method, the results might look quite different. The cost comparison amortizes AFlow's expense over many downstream tasks — but the *quality* of Swift's priors is downstream of AFlow having done its job well.

12:26Bella: That's fair. The amortized story assumes the upstream search was good. Which is a real assumption.

12:32Finn: The second critique is about the "search is counterproductive" framing. The paper attributes Swift beating AFlow to AFlow overfitting to small validation sets. That's plausible — search ten thousand variants on fifty problems and some will look brilliant by accident, the same way a stock-picking strategy back-tested on a narrow window can look brilliant by accident. But the experiments show correlation rather than mechanism. They show that search-based methods do worse on small benchmarks. They don't isolate specific overfitting events that Swift avoids. The careful version of the claim is "search is unnecessary for these task families and can hurt," not "search is universally bad."

13:17Bella: And the third?

13:17Finn: Benchmark friendliness. GSM8K, MATH, HumanEval, MBPP, MultiArith — these are math and code benchmarks where the obvious topology is well-known and almost certainly present in the LLM's pretraining. The strongest version of the paper's claim — that workflow search is broadly redundant — would need tasks where the optimal topology isn't a well-rehearsed pattern. The paper doesn't really stress-test that.

13:43Bella: They do touch on it, with the AQuA case. Want to walk through it? It's a great little failure.

13:50Finn: This one is delicious. AQuA is a multiple-choice algebraic reasoning benchmark. It has a problem that goes something like: an orange costs eighteen, a pineapple costs twenty-seven, a grape costs fifteen. How much does mango cost? The answer is fifteen. Because the price equals three times the number of letters in the word.

14:11Bella: It's a word puzzle wearing arithmetic clothes.

14:14Finn: Right. And Swift, primed by all those math demonstrations to look for arithmetic relationships, predicts twenty. The transferred strategy actively misleads it. The tool it's been given is a hammer, and when this nail turns out to be a screw, the hammer makes things worse. That's what the strategy-mismatch failure mode looks like.

14:36Bella: Which speaks to the fourth critique you mentioned?

14:39Finn: Yes. The ninety-three percent retention number is striking but slightly oversold. "Topology not semantics" is the headline. But the actual ablation drops average performance from about eighty-two to seventy-seven. That's small enough to support the claim that topology dominates, but not so small that semantics is irrelevant. A more honest reading: topology is something like ninety-five percent of the signal, semantics is the remaining five — both matter, topology matters much more.

15:11Bella: That's the right calibration. And one more I'd add — the cost comparison. The "three orders of magnitude" framing is real for optimization cost only. If you actually run Swift and AFlow end-to-end on the same test set, including execution cost, the gap on MATH is closer to fourteen times. Still impressive — fourteen times cheaper, twenty-nine times faster — but not "thousands of times." The paper is upfront about that in their cost table; the abstract leans on the more dramatic number.

15:43Finn: There are two more honest failure cases worth airing, because the paper deserves credit for surfacing them. The first is AIME — the math olympiad. Swift on AIME twenty-twenty-four and twenty-twenty-five hits about fourteen percent. Vanilla GPT-4o-mini hits about eight. So Swift helps. But fourteen percent is still terrible. And the reason is simple: the base model can't do competition math. No amount of clever workflow scaffolding is going to fix that. Workflow design is a multiplier on capability, not a substitute for it.

16:17Bella: That's an important point this whole subfield sometimes loses sight of. The agent infrastructure can only structure the calls; it can't make the calls smarter than the model that's making them.

16:30Finn: The second is Gemma-3-12B on MATH. Vanilla Gemma gets about fifty-eight percent. Under Swift, it drops to forty-eight. Swift makes Gemma worse.

16:39Bella: Why?

16:40Finn: Because the topology Swift synthesizes — the multi-step ensemble-and-vote pattern that works beautifully when GPT-4o-mini is the worker — is too complex for Gemma's instruction-following ability. Errors compound across nodes. By the time you've routed data through five steps, the small errors at each step have multiplied. So the "right" workflow depends on your worker model, and Swift's distilled topology is implicitly tuned to capable models.

17:09Bella: There's also an environment-bounded failure they flag — on BigCodeBench, seventy-one percent of the failures are missing Python modules in the sandbox. Not workflow logic at all. Just infrastructure. The actual logic-error rate is something like seventeen percent. Which means the apparent benchmark gap is mostly a packaging problem.

17:30Finn: Honest failure cases like these are what make me trust the rest of the paper, Bella. A paper that only reports its wins teaches you nothing about where the boundary is. These four — capability-bounded, environment-bounded, strategy mismatch, instruction-following limits — give you a real map of the regime where this works.

17:51Bella: Let me zoom out, because there's a deeper idea this paper sits inside, and I think it's worth naming. In machine learning, you generally have two ways to solve a class of problems. You can run a fresh optimization for every new instance — slow per query, but tailored. Or you can do expensive work once to train a system that produces good answers in a single forward pass — much faster per query, with a fixed setup cost. That second approach is called amortized inference. The classic example is image compression. You can search for the optimal encoding of each individual image, or you can train a neural network once that compresses any image in milliseconds. Variational autoencoders work this way. Most of modern deep learning, really. What Swift is doing is applying that move at the level of program synthesis. Instead of searching the space of workflows for every task, do the work once to learn the patterns, then synthesize new workflows in a single LLM call.

18:53Finn: There's a study-for-the-exam version of the analogy that I like. Per-task search is like cramming the night before each exam. You absorb just enough to pass that specific test, then dump it. The cost is high every single time. Amortized synthesis is like actually learning the subject. Heavier upfront investment, but every subsequent exam costs you almost nothing. The break-even point for Swift is about five tasks. Past that, amortization wins, and the gap grows linearly.

19:24Bella: There's an extra wrinkle the analogy misses, which is worth saying out loud. It's not just that amortized synthesis is cheaper. In Swift's experiments, it's also more accurate. The cramming version of the strategy isn't just expensive — it's worse, because cramming makes you overfit to the specific exam questions in front of you.

19:45Finn: That broader conversation matters. There's a recurring pattern in machine learning where someone notices a hard search problem isn't actually that hard once you look at the solutions. Neural architecture search went through this — years of expensive search converged on a small handful of motifs, and the field eventually started just hand-using the motifs. Hyperparameter search has a similar arc. Swift is making the same kind of argument for automated agent design: the search space looks combinatorially huge, but the *useful* part of it is small and stereotyped. Amortization across tasks beats search within them.

20:23Bella: There's a quieter methodological point underneath all of this, too. When you optimize aggressively against a small validation set, you can convince yourself you're discovering something general when you're actually just overfitting. It's an old worry, but it lands with new force in agent design, where the validation set might be a few dozen problems and the workflow graph gets to be arbitrarily idiosyncratic. The paper's claim that search-based methods can do worse than one-shot transfer is, in part, that worry restated in a new domain.

20:56Finn: For me, the part of this that's going to stick is the random-strings ablation. Not the cost number — the cost number is a headline. The fact that you can replace every operator name with gibberish and lose only five points tells you something about what in-context learning actually is, in this setting. The model isn't reading the demonstrations as English. It's reading them as wiring diagrams. That's a small piece of evidence, but it points at something larger about how these systems use structure that we don't fully understand yet.

21:30Bella: Agreed. And the practical lesson sits one level above that. Before designing a clever search algorithm for some space, check whether what you're searching for is actually as varied as you think it is. Sometimes the answer is no, the useful region is small, and someone is paying twenty-two dollars and three hours per task to rediscover the same shape over and over.

21:54Finn: The show notes have a link to the paper and related materials — worth a read if this episode caught you. Thanks for listening to AI Papers: A Deep Dive.