There are two high-level ways to make an AI agent work on something hard for a long time: give it enough context to hold more of the problem in mind, or build scaffolding around it so it doesn’t have to. Scaffolding can include retrieval, memory, planning, tools, tests, checkpoints, and sub-agents.
I recently sat down with Prateek Jain to talk through what the right combination of these two approaches is. Prateek is a Distinguished Scientist at Google DeepMind, where he works on new architectures for frontier models with a focus on efficiency and elasticity, and co-leads Gemini’s long-term innovation area that focuses on risky and ambitious research projects.
Recently Prateek has been focusing a lot on long-horizon agents, systems that stay on a task longer than most models are trained to do right now. Prateek explains the goal in terms of human effort: “Let’s say there is a task that requires ten experts to be at a problem for five days, or ten days, or maybe a month. We want to see if models can go and solve those tasks, which means that our models should know how to plan for a month. They should know how to delegate. They should know how to share information across all these tasks and subtasks and sub-agents.”
But before this can happen, a few things need to be solved first:
Cost: Attention cost grows quadratically with context length, so doubling what a model holds in mind roughly quadruples what it costs to serve.
Handoff: When a main agent delegates to a sub-agent, the sub-agent typically only sees what was written down for it. This means the reasoning that produced the subtask is thrown out.
Rationing: Anyone running agents at scale has to manage compute costs. But it’s still unclear how an agent should manage the budget it gives each sub-agent it spins up.
There are tradeoffs across each of these. More context gives the model more room to work, but it also makes the system more expensive to run. A text-only handoff is cheaper, but the sub-agent only gets the summary, instead of all the work that led up to it. A tight compute budget for the sub-agent might result in the sub-agent failing and thus the work needing to be redone.
In our conversation we explored these tradeoffs, and what has to change architecturally in order for long-horizon agents to work at scale.
What is a long-horizon agent, anyway?
“Long-horizon” is a term that gets thrown around a lot but what actually makes something “long-horizon”? I asked Prateek and he said he defines it along two axes: task difficulty and training distributions.
On difficulty: a long-horizon task is one that would take multiple human experts multiple days (think ten experts working for a week or a month).
On training distribution: standard RL post-training have multiple rollouts where the model has practiced full attempts at a task with trajectories of somewhat limited length. A long-horizon task runs past anything in that range, so the model is improvising instead of drawing on something it has done before. As Prateek put it: “The model was trained for a certain rollout length range, and now you are asking the model to do maybe 10x more work,” at which point the task is “out of distribution” with respect to training data.
Long context is necessary, but not sufficient
When I asked Prateek whether he thinks we should give models ever-longer context windows, or build agentic scaffolding around shorter-context models, he said he sees them as a complement. As tasks get harder, “the amount of context we will need to solve the problem is going to be extremely high. So having agentic scaffolding, as well as ideas like retrieval or maybe separate memory, will be critical.”
But squeezing everything through a short context window breaks down, because hard tasks are underspecified: “For very hard tasks you might not already know what all you need to put in the context.”
He gave an example: “If you are trying to ask Gemini about a fairly underspecified problem ‘oh, my wife’s birthday is tomorrow, can you please plan the whole day?’ Gemini would need to understand how your relationship is, what are the things you like to do for fun. It might have some rough idea: okay, let’s find pictures where both of these people are together, or emails around itinerary planning. But it might not be able to pinpoint the exact things apriori.”
Pure retrieval only works if you know what to look for, which can be hard when the task is underspecified. Meanwhile, long context is the opposite because instead of deciding upfront what to look for, you bring in a broad field of information and then let the model decide what’s relevant and how the pieces connect.
Long context simplifies orchestration. With a short context window, you need more retrieval, more sub-agent calls, and more tool calls. This is all extra orchestration burden, which, as Prateek put it, “is not ideal. You want to give as much context to the model as possible.”
The goal instead is “a really clean, simple solution which is also fairly general and can solve very challenging long-horizon tasks”: a capable long-context model that needs less machinery around it, not more.
The efficiency wall, elastic models, and routing
Every token of context has a price. “The serving cost, especially with super long context, is going to grow quadratically, which is challenging,” Prateek acknowledged. Inference economics is an architectural constraint for agents doing long-running work at scale.
He’s optimistic, however, that active research in frontier labs and academia “might be able to bring down the cost significantly.”
Some of Prateek’s work is focused here, on model efficiency, but from the angle of how much model each token uses. “Models are sort of monolithic. For every task and every phase, you are basically passing the data through the same number of layers and doing the same amount of work.” This means every step, no matter how hard or easy it is, costs the same.
MatFormer, one of Prateek’s best-known projects, asks whether a model can instead be elastic, dialing its power up or down continuously. In practice, Prateek explained, that could mean adjusting the model “based on the complexity of the task or maybe based on how loaded your servers are or what is the cost currently of your tokens.”
“If agents can say, hey, this is a simple task, I can delegate it to a small model within this large family of models versus, oh, this is a slightly harder task, let me go one notch higher. That can enable high quality solutions at relatively low cost,” Prateek said.
This decision-making requires routing, meaning a policy has to decide which model to call, whether to spawn a sub-agent, what budget it gets, and when to keep retrying versus stop.
“As agents scale to really large numbers of tasks per second,” Prateek said, “we will need to start rationing how much compute you are giving to these agents.” The risk, however, is that you pick a sub-agent too small for the task and it fails, forcing the main agent to evaluate and redo the work. In this case, you end up spending more than if you had just had a more generous upfront routing budget to begin with.
Getting routing right means jointly assessing “the complexity of the task and the capability of the sub-agent.” And budgets cascade, since the sub-agent then decides for itself how much thinking its allotment buys.
This kind of elastic routing hasn’t fully arrived. “There is still more research to be done” on the quality-versus-cost tradeoff. But the direction feels inevitable to him: “As agents percolate to pretty much everything we do on a day-to-day basis, then having to do this rationing of compute and of capabilities becomes unavoidable.”
The handoff problem
Delegating to a smaller model requires planning a precise sub-task for the smaller model along with enough context/hints to the smaller model so that it can solve it. In today’s main-agent/sub-agent pattern, it’s mostly whatever the main model serialized into text (e.g. a task description rather than the internal state that produced it). The main model does a bunch of work, writes up a subtask, and hands it off. “The sub-agent’s view is only the subtask that has been given. It doesn’t know what all work the primary model would have done,” Prateek explained. Everything the main model figured out in its KV cache is essentially thrown out at the handoff.
Tandem Transformers, another of Prateek’s projects, is focused on this tension. The work pairs a small model with a large one, trained together so the small model consumes the large model’s representations rather than a text description of the task. “Whatever work the main model has been doing in its full form, in the KV-cache format itself, can be transferred to the sub-agent,” Prateek said. “Which hopefully can provide it with a much richer context so that it can solve the task in a much better fashion.”
In short, the sub-agent gets the main agent’s working memory, making the delegation much more flexible. The main model no longer has to fully specify the subtask+hints/context in text before passing it along.
Team players and strategic thinkers
Prateek believes some of the most important domains for long-horizon agents are agentic coding, particularly for machine learning and LLMs, and STEM research, especially in biotech and material science research. Success in these areas requires judgment, “this kind of work requires the development of taste and personality itself in the model, along with domain expertise and general intelligence,” noted Prateek.
Five years out, Prateek expects the main job of these systems to be organizational. “What I see is the base models becoming not only very intelligent and powerful, but also having a lot of capabilities around the ability to plan, delegate, coordinate. The model can be put in a variety of harnesses. You can attach memory to them, you can attach knowledge bases to them, and make them work.”
And underneath all of that, the model has to know its own limits. “Does it know that this is beyond its capability so it can bring in other ideas? Is it able to recognize that maybe it should spin up a new subagent with focus on specific tools?” Prateek sees these as the key questions. Every routing decision depends on that: sizing a sub-agent, setting a budget, deciding whether to escalate. A model that can’t tell hard from easy, can’t ask for help from humans or call potentially expensive tools, and can’t ration effectively.
Today, the models are great ICs, especially when supplied with clear tasks and strategies to solve the task. What they’re still learning is how to be, as Prateek put it, “really good team players, strategic thinkers, and leaders.”
Thanks to Divy Thakkar for making this connection!
Author’s note: An LLM was used for light copy editing only (spelling, grammar, and clarity). Content, meaning, tone, and structure remain unchanged. Prateek is speaking in his personal capacity. The views expressed here are his own and do not represent those of his company.