Dream-RSI turns past discovery runs into replayable worlds, letting AI agents test search strategies before running costly experiments. We explain how it works, what the research found, where ModularRSI and ScienceBuddy fit, and why recorded decision paths may matter for future enterprise AI agents.
An AI agent tackling a hard problem can make hundreds of attempts. It writes code, tests it, studies a failure, and tries another route. Some attempts lead somewhere. Many do not. Usually the final answer survives, while the route taken to find it becomes a pile of logs.
That route may be valuable in its own right. A recent paper, Dream-RSI: Recursive Self-Improvement through Evolving Worlds, asks whether an agent can use the record of its earlier searches to improve how it searches next time. The authors, affiliated with Google, Google DeepMind and two US universities, turn recorded discovery paths into places where new search strategies can be tested cheaply. Related projects, including ModularRSI and ScienceBuddy, explore other ways an agent system can learn from its own work.
The idea is easier to understand once we set aside the dramatic image of an AI rewriting itself overnight. Here, self-improvement starts with something fairly practical: learning which trails are worth following, how many to explore at once, and when to stop.
Why AI Discovery Agents Waste Time Repeating the Search
Imagine a coding agent looking for a faster algorithm. It creates one candidate, runs a test, and gets a score. It might refine that candidate or start a fresh branch. If ten branches are open, it also has to decide where the next hour of computing time goes.
These choices matter. An agent can find a good solution and still spend most of its budget checking weak ideas. A fixed rule, such as “keep ten branches running for the same number of steps,” cannot respond well when one branch looks promising or several have stalled.
Improving the rule is hard for a reason: to find out whether a new strategy works, you would normally run the discovery task again. That could mean many more code generations, tests and model calls. Now imagine testing hundreds of possible strategies. The bill grows quickly, and useful feedback arrives late.
This is the opening Dream-RSI takes. Past searches already contain decisions, results and costs. Could those records become a testing ground for the next search policy?
What Recursive Self-Improvement Means in Dream-RSI
“Recursive self-improvement” is a broad term. It can describe a system that changes its model, its tools, its instructions or the process it uses to solve problems. Dream-RSI focuses on one layer: the exploration policy, meaning the rules that decide where the agent searches and how it spends its available effort.
The underlying coding agent stays fixed in the paper’s setup. So does the evaluator that scores candidates. What changes is the code controlling search: which branch to extend, how many attempts to run in parallel, and whether it is time to finish. A separate policy-development agent proposes changes to that code and checks them against saved searches.
It is recursive because the revised policy goes back into a live discovery run. That run produces another record, giving the system more experience to use when it revises the policy again. The cycle can repeat: explore, record, rehearse, improve, explore again.
An agent finding a better algorithm has improved the answer to one task. An agent finding a better way to allocate its search effort has improved the process it uses across attempts. Dream-RSI studies that second kind of improvement.
How Dream-RSI Turns Past Runs Into Replay Worlds
The system first records a live search as a discovery tree. Think of the starting workspace as the trunk. Each attempt creates a branch or extends one that already exists. The record keeps the candidate, the test result, the score, the feedback and the work state from which that attempt began.
Once this tree exists, a different exploration policy can move through the recorded branches. It may choose to inspect one line of work earlier, open another branch later, run available attempts in parallel or stop after finding a strong result. The outcomes have already been measured, so replay can reveal them without asking the coding agent to regenerate each candidate and rerun every test.
The paper calls these saved trees “worlds” and the offline policy trials “dreaming.” The words sound grander than the mechanism. In practice, the agent is rehearsing search decisions against recorded results.
Consider an example. A coding agent tried four routes to speed up a program, spending equal effort on each. In the saved tree, route A showed steady gains, route B failed early, and routes C and D were mixed. A proposed policy can be replayed to see whether shifting effort toward A and stopping B earlier would have found the best recorded result sooner. Repeating this across many trees is cheaper than repeating every live experiment.
But the word “recorded” matters. Replay cannot tell us what an untried fifth route would have produced. Nor does it guarantee that the same coding agent would generate the same result if a branch were run again; live generation can vary. The simulator covers the part of the search space that was actually observed. New live runs are needed to test whether a policy learned from history really helps outside it.
Dream-RSI scores candidate policies using the quality of the best result they uncover in replay, while accounting for the number of attempts and the use of parallel workers. The selected policy then guides another live run. Its new tree joins the pool of replay worlds, so the next round can draw on a broader record.
What the Dream-RSI Experiments Found
The researchers tested the approach on eight discovery tasks across algorithm engineering, mathematical optimization and GPU kernel engineering. They compared it with a setup that used the same discovery agent and initial search rule but kept that rule fixed in later rounds.
The reported gains vary by task, which is worth saying plainly. On one algorithm engineering comparison, the paper reports up to 162 times fewer discovery-agent calls than a separate baseline called SimpleTES. Against the more closely controlled fixed-exploration setup, it reports up to 1.7 times fewer calls for that task. In GPU kernel work, two tasks reached comparable performance with 2.43 times and 1.79 times fewer generations; two others showed higher performance under comparable search budgets. On one mathematical task, Dream-RSI improved the reported score over the fixed-search comparison. On circle packing it matched the strongest reported result, while on an autocorrelation task its result remained competitive without being the best in the comparison table.
Those are research results in defined test settings, not a promise that every agent will become faster by the same factor. The stronger point is the pattern: changing the search policy with feedback from earlier searches can improve the use of a discovery budget, even when the underlying coding model does not change.
How ModularRSI and ScienceBuddy Approach Self-Improvement
Dream-RSI sits within a wider line of research, but the projects do not all improve the same thing.
ModularRSI looks at the agent harness: the software and procedures around a model that handle its work. The researchers split that harness into five parts: the agent loop, tool use, observation management, context management and task-completion detection. They compare successful and failed runs, look for repeated faults, change a relevant part, and then combine the changes. Their tests use coding and terminal tasks kept separate from the tasks used to evolve the harness. That separation helps check whether an improvement travels beyond the examples that inspired it.
ScienceBuddy, meanwhile, is built around interaction with researchers. It turns requests, follow-up feedback and execution evidence into tasks and ways to assess them. Its proposed inner loop improves the agent’s working procedures while keeping the model fixed. An outer loop then trains the model under those improved procedures. The paper presents case studies and experiments on scientific task families. Its scope is broader than Dream-RSI’s search controller because it studies both procedure changes and model learning.
These three efforts share a question: what should an agent keep from its previous work so that the next round goes better? Their answers differ. Dream-RSI reuses the shape and outcomes of past search. ModularRSI diagnoses recurring faults in the machinery around a model. ScienceBuddy uses researcher interaction to improve both that machinery and, in its outer loop, the model itself.
Could Replay-Based Learning Help Enterprise AI Agents?
The immediate evidence comes from research and coding tasks. Still, the design suggests a useful question for enterprise systems: how much of an agent’s decision path is being kept, and can that record be safely used to improve the next run?
Take a quality-investigation agent in a factory as an illustration, not a tested Dream-RSI application. It might review inspection results, batch history and machine events to identify likely causes of a defect. If it records which evidence it checked, which hypotheses it rejected, how long each path took and what a quality engineer later confirmed, that record could help evaluate future investigation strategies. Perhaps one sequence finds verified answers with fewer queries. Perhaps another repeatedly misses a shift-change note. The practical gain would come from comparing those paths against confirmed outcomes, then testing any revised procedure on fresh cases.
Other settings introduce difficulties that a tidy benchmark may hide. Data changes. Human decisions may lack a simple score. A past result may reflect an unusual case, and one plant’s process may differ from another’s. An organisation would need clear records, reliable evaluation criteria and fresh checks before using revised policies.
The Limits of Learning From Yesterday’s Search
Replay has a natural blind spot: it can only reveal what earlier runs recorded. If those runs mostly explored one kind of idea, a policy might get very good at navigating that history while missing better ideas elsewhere. Reusing the same trees for repeated policy selection can also reward rules that suit those particular trees. Success in replay is therefore a reason to try a new policy live, not proof that it will work everywhere.
There is a measurement problem, too. A system tends to improve what it scores. If the score values speed but misses correctness, safety or the cost of a bad recommendation, the search policy may move in the wrong direction. In an enterprise setting, a fast but weak investigation is hardly an improvement.
ModularRSI’s emphasis on held-out tasks points to one response: evaluate changes on work the system has not used to learn its new habits. Dream-RSI itself redeploys policies online, where fresh experience expands the replay pool. Human review still matters whenever the outcome is hard to measure or the decision has real consequences.
A More Practical View of Self-Improving AI
The immediate development is quieter: agents are beginning to inspect, test against and learn from their past work. Dream-RSI offers one concrete version of that idea. Save the discovery process, rehearse different ways through it, and carry the most promising search policy into another real run. ModularRSI asks which parts of an agent’s working setup should change. ScienceBuddy asks how research collaboration can feed both better procedures and model learning.
For teams building agents today, the lesson is modest but useful. Keep the evidence behind each attempt. Measure whether changes help on new tasks. And remember that an agent’s route to an answer may teach as much as the answer itself.