One in every of my basic beliefs concerning the world is that Ramez Naam must weblog extra. Ramez is without doubt one of the world’s best futurists — he predicted the photo voltaic and battery revolutions lengthy earlier than these had been extensively understood. For those who had been studying Ramez in 2011, you had been capable of perceive the way forward for each power know-how and local weather change, lengthy earlier than different individuals did. His earlier guide Greater than Human remains to be an incredible information to the form of organic enhancements that AI would possibly make attainable. Ramez can be a superb science fiction creator, having written a trilogy of novels wherein nanotechnological telepathy is distributed as a celebration drug (I’m unsure if he truly expects that to occur, nevertheless it’s a really cool thought).
Sadly, though he does have a Substack (which it is best to completely observe), Ramez doesn’t weblog frequently. Nonetheless, after having a prolonged personal debate with him about Recursive Self-Enchancment, I used to be capable of prevail upon him to jot down up his ideas for my weblog.
To say that RSI is an enormous deal within the AI world could be a colossal understatement. Amongst AI researchers, entrepreneurs, and AI security individuals, there’s a widespread perception that as AI will get higher at enhancing itself, there will probably be a “quick takeoff” or “FOOM”, wherein AI’s capabilities “take off” and create a technological Singularity. This occasion is a staple of science fiction, together with works by my favourite sci-fi creator, Vernor Vinge.
Lots of people within the business imagine that this second is now shut at hand, and are racing towards that prize:
Tanay Jaipuria@tanayj
Noam Brown says that recursive self-improvement / the flexibility of AI fashions to do AI analysis is the highest precedence for OpenAI by a large margin.
OpenAI ranks it above making fashions that promote.
(by way of @theinformation podcast)
12:24 AM · Sep 27, 2026 · 4.01K Views
4 Replies · 2 Reposts · 44 Likes
However Ramez — usually among the many most wide-eyed of techno-optimists — is extremely skeptical that we’ll see something just like the “FOOM” of Vernor Vinge novels. On this prolonged, well-researched put up, he explains his skepticism.
Personally, I’m agnostic. Ramez’s case essentially rests on loads of assumptions; though it’s cogently laid out, I believe the actual reply is that we’ll simply have to attend and see whether or not the Singularity arrives. However much more essentially, I don’t know the way a lot this debate issues within the sensible sense — even with out the form of Singularity depicted in sci-fi novels, AI capabilities are enhancing so quickly that they’re already superhuman in lots of respects, and shortly will in all probability be strongly superhuman in most or all dimensions. The AI of 2040 goes to look godlike, whether or not or not it explodes into an precise god in 2027.
Nonetheless, it’s a really attention-grabbing argument, and Ramez’s ideas on the way forward for know-how are at all times value listening to.
AI is already serving to enhance itself. The query is whether or not even absolutely autonomous recursive self-improvement (RSI) would trigger a runaway intelligence explosion.
The speculation is that every era of AI might construct a greater successor, sooner than the final era did. That would result in a “quick takeoff,” with capabilities surging to synthetic superintelligence (ASI) in a 12 months, months, and even days.
Right here’s my take: Given our greatest present knowledge, the AI self-improvement loop would must be roughly 5–10× stronger to maintain itself, not to mention run away. I’ll clarify this math in part 8. I anticipate extremely fast AI progress by the requirements of practically another know-how. However the proof we’ve got doesn’t recommend a sudden explosion to incomprehensible superintelligence anytime quickly.
I could possibly be fallacious. Forecasters have repeatedly underestimated AI progress! I might properly be subsequent. One factor that’s clear is that we’d like higher knowledge. For now, let’s work with what we are able to measure, and keep open to breakthroughs that would change the image.
Determine 1. How sturdy is the self-improvement loop? Mannequin.
Right here’s the case, with hyperlinks to every half:
Key charts: The suggestions loop · Measured vs. forecast progress · Diminishing returns
Individuals use “recursive self-improvement” to imply the whole lot from AI boosting the productiveness of human researchers to AI bootstrapping itself to incomprehensible intelligence. Right here’s my taxonomy: productiveness features (Kind 1), rising autonomy whereas nonetheless going through diminishing returns (Sorts 2–4), and a runaway loop to superintelligence if we are able to ever discover accelerating returns (Kind 5).
Determine 2. 5 kinds of AI self-improvement.
We’ve made actual progress on Sorts 1 and a pair of: AI helps each researchers and engineers within AI corporations, and highly effective fashions can practice and enhance smaller ones. We haven’t but seen clear proof for Kind 3 (although Alibaba simply made some sturdy claims) and positively not for Kind 4. I do anticipate autonomous self-improvement to reach in some unspecified time in the future. I’m skeptical that it results in Kind 5 – runaway super-intelligence – and not using a main conceptual breakthrough.
There are many different definitions of RSI, which is usually a bit complicated. Weco’s 4 ranges of RSI are near mine. For a broader tour of all of the issues individuals imply once they say ‘RSI’, learn Tom Cunningham’s complete information.
I do anticipate slender superintelligence in extremely verifiable domains. Assume chess, Go, formal math, elements of laptop science and coding. Extremely verifiable domains are largely formal and structured kinds of work the place machines can generate limitless coaching knowledge, with excellent or near-perfect verification of right vs incorrect, and accomplish that solely in software program with out ready on the bodily world or people. That’s an excellent setting for AI studying.
Determine 3. What makes a website extremely verifiable?
The truth is, we have already got slender superintelligence in recreation plang. We’re seeing it occur now in essentially the most formal elements of math, particularly in proofs and find counter-examples that disprove main conjectures. For instance, OpenAI not too long ago reported an AI-generated proof resolving the Navier–Stokes existence and smoothness downside. Components of software program growth are additionally extraordinarily verifiable, whereas others are a bit much less crisp (resembling understanding what people need).
That isn’t the identical as broad superintelligence. Even our strongest fashions want way more coaching knowledge than people, battle to study reliably from ongoing expertise, and fail in shocking methods on duties individuals discover easy. Superhuman math doesn’t robotically imply superhuman judgment all over the place else.
Benchmarks and forecasts recommend that AI fashions ought to reliably succeed at coding duties that take people hours, with out human assist. The true world is messier. OpenAI’s inner knowledge reveals a lot shorter stretches of autonomous work on analysis duties.
In its Analysis Acceleration / RSI report, OpenAI confirmed how typically its fashions accomplished duties with and with out human assist, grouped by how lengthy a human would want to do the work.
Determine 4. OpenAI’s inner analysis duties. Supply.
Even on duties that might take a human lower than quarter-hour, OpenAI’s fashions succeeded with out human intervention solely 86% of the time. The estimated activity size at 80% success was roughly quarter-hour over the primary seven months of the 12 months. July’s outcomes had been just like the entire interval common.
Totally autonomous RSI would require an AI to string collectively an incredible many analysis duties reliably, stretching out over advanced duties that people want weeks or months to perform. OpenAI’s knowledge means that we aren’t shut.
Anthropic additionally launched a graph exhibiting how Claude accelerates AI analysis. It reveals that inner AI fashions collaborate on and even lead greater than 90% of R&D duties. That’s objectively spectacular. On the similar time, the graph experiences zero instances of AI autonomously finishing AI R&D duties.
Determine 5. Claude’s function in inner AI R&D. Supply.
These are unimaginable instruments. However they nonetheless want expert individuals to set course and get them again on monitor.
For years, METR has been publishing a chart exhibiting what size of coding activity (measured in human hours to finish) best-in-class AI fashions can obtain. It’s been referred to as a very powerful graph in AI. METR’s Mythos Preview analysis estimated that the mannequin might succeed at 80% of coding duties that took people three hours.
Determine 6. METR’s 80% activity horizons. Supply.
Epoch’s personal rule of thumb is that each 5 further factors of ECI (their general benchmark of AI functionality) correspond to roughly a doubling of METR’s activity horizon. Utilizing that system, we’d anticipate GPT 5.6 Sol and GPT 6 Astra to be 80% profitable at finishing duties of round 4 hours and 11 hours of human size, respectively.
One other estimate (a forecast) of AI activity size comes from the AI 2027 situation, which estimated that by July 2026, frontier AIs could be 80% profitable undertaking duties of round 11 hours. Pretty comparable.
The AI 2027 Tracker charts all of those.
Determine 7. The AI 2027 Tracker. Supply.
Inside OpenAI, although, the July research-task horizon at 80% success was roughly quarter-hour.
Right here’s the hole:
Determine 8. Forecasts, benchmarks, and actual AI analysis. Tracker · OpenAI.
A four-hour benchmark horizon is about 16 instances longer than OpenAI’s analysis horizon. AI 2027’s 11-hour forecast is about 44 instances longer. In fact, the duties being carried out by researchers at OpenAI aren’t the identical as these within the METR benchmark. So we should always anticipate some discrepancy. This, nonetheless, goes properly past that.
Precise AI analysis at OpenAI is an order of magnitude or extra more durable than metrics, benchmarks, or forecasts recommend. That ought to make us cautious of relying an excessive amount of on benchmarks, or of claiming that future situations like AI 2027 are ‘on monitor.’ The authors of the associated AI 2040 undertaking nonetheless describe AI 2027 as roughly the long run they anticipate, and say actuality is monitoring nearer to it than even they anticipated. That’s not what we see from inside OpenAI. This isn’t an apples-to-apples comparability, however the distinction is exceptional. AI 2027 seems to be considerably over-optimistic on this regard.
In January of this 12 months, Nathan Witkin made a case that the METR graph was exaggerating progress. The true world knowledge means that not less than a few of his critiques had been right. The hole between benchmarks, forecasts, and knowledge gleaned from precise use of AI ought to affect our expectations concerning the future.
OpenAI’s report additionally reveals spectacular will increase in AI token utilization, in compute spend per researcher, and in strains of code written. However these aren’t outcomes. They’re intermediate measures. How a lot progress do they really drive?
Researchers used 124x extra tokens per individual. Engineers shipped roughly 7x as many strains of code per individual. Researchers ran 1.6x as many experiments per researcher vs OpenAI’s 2025 entire 12 months common.
Determine 9. Token use inside OpenAI. Supply.
Determine 10. Experiment tempo inside OpenAI. Supply.
Determine 11. From tokens to code to experiments. Supply.
Extra tokens and code don’t inform us a lot on their very own. The 1.6× experiment tempo is nearer to helpful analysis output. Even that doesn’t imply AI is enhancing 1.6× sooner.
An unlimited improve in AI output has accompanied a a lot smaller improve in experiments run.
This isn’t a managed experiment. We don’t know what would occur if researchers switched again to an older mannequin. However it provides us a helpful view of AI-assisted analysis inside a frontier lab.
It’s not simply OpenAI. Anthropic experiences that their engineers are actually producing 8x as many strains of code per individual as they did in 2024 – considerably just like OpenAI. Anthropic additionally sees vital diminishing returns between productiveness and AI progress. Right here’s a direct quote from its Mythos Preview system card:
“Productiveness uplift doesn’t translate one-for-one to capabilities progress. We surveyed technical employees on the productiveness uplift they expertise from Claude Mythos Preview relative to zero AI help. The distribution is broad and the geometric imply is on the order of 4×. […] We estimate that reaching 2× on general progress by way of this channel would require uplift roughly an order of magnitude bigger than what we observe.”- Anthropic, Claude Mythos Preview System Card; emphasis mine
Translation: To double the tempo of AI progress, Anthropic estimates that AI would want to extend the productiveness of their staff by roughly an element of 40 relative to no AI help.
Determine 12. Anthropic’s productivity-to-progress estimate. Supply.
That is an estimate, not a measurement of progress. Even the 4× productiveness determine comes from an opt-in survey of 130 Anthropic employees. I put extra weight on OpenAI’s logged experiments, although the 2 sources measure various things.
We don’t but know the way a lot these further experiments are accelerating AI enchancment, if in any respect. Typically, there are additionally steeply diminishing returns of extra experiments in most branches of science. That signifies that a 60% improve in experiment tempo could possibly be on the order of a ten% enhance to AI enchancment tempo. (An influence regulation exponent of 0.2, for many who need to do the maths.) That’s hypothesis for now. We’ll study extra because the labs publish outcomes.
What about giving the identical AI mannequin extra time to assume?
That scales badly additionally. In OpenAI’s not too long ago publicized outcomes on unsolved math issues, success rises roughly with the log of compute over the vary proven. It reveals logarithmic diminishing returns. In plain English, every further doubling of compute for a mannequin buys roughly the identical acquire in success fee, whereas costing twice as a lot.
Determine 13. Take a look at-time compute and math efficiency. Supply.
What if we throw extra brokers at it as a substitute? A standard RSI / ASI thought is that when we’ve got AIs at a sure functionality stage, we are able to simply spawn extra copies and put them to work.
Including brokers can get duties performed sooner and typically attain the next functionality stage. However on the three benchmarks in Toby Ord’s evaluation, increasing a swarm buys much less enchancment per token than letting one agent assume longer.
His tough rule of thumb is a sq. root. If one agent can accomplish a activity in 10 hours, then 100 brokers might accomplish it in a single hour. The speedup is 10, the sq. root of the variety of brokers (100). However to get this speedup, you improve the whole value in tokens or run time compute by the identical issue. So going from one to 100 brokers can get a activity performed in a single tenth the time. However it’ll be ten instances as costly.
Parallel brokers can save time, at a a lot greater compute value.
One other problem is that brokers typically assume alike. In a examine evaluating LLMs with 467 individuals, the primary ten AI responses supplied collective creativity corresponding to about eight to 10 individuals. After that, roughly two further AI responses added as a lot as one further human response. A separate examine throughout mannequin households additionally discovered much less variety in AI responses. That doesn’t imply each agent has the identical thought. However 100 copies might supply much less selection than 100 totally different researchers.
None of this makes swarms useless-or protected. Lisan al-Gaib makes a powerful case for parallel agent swarms as a potent cyber-weapon in “Unintentional Scaling.” I don’t share all of his evaluation of what swarms have completed. In math, for instance, I believe he provides far an excessive amount of credit score to the swarm and never sufficient to the higher inner mannequin that OpenAI used.
OpenAI says the mannequin behind its Navier–Stokes end result was developed by way of “large-scale reinforcement studying on high of a beforehand pretrained mannequin.” Formal math is a extremely verifiable area, which makes it a very good match for that strategy: Machines can generate practically limitless quantities of coaching knowledge, and confirm that options are right or incorrect, all in software program. My guess is that this mannequin’s full outcomes will present an particularly massive enchancment in math.
OpenAI’s Noam Brown made the central level explicitly: he wouldn’t give multi-agent strategies even 10% of the credit score for the Navier–Stokes end result.
I do assume Lisan makes good factors about cybersecurity. For those who’re trying to find a safety vulnerability at a goal web site and might divide the search amongst brokers, velocity might justify an enormous token invoice. Swarms could be harmful even once they’re inefficient.
I’m much less satisfied that this scales to analysis breakthroughs. Inventing one thing just like the transformer in all probability takes greater than looking an area somebody has already outlined.
Constructing a greater mannequin can convey features that further pondering time or extra copies of the previous mannequin can’t. Have a look at the hole between Astra and OpenAI’s inner mannequin on the identical math issues.
Determine 14. Higher fashions versus extra pondering time. Supply.
That’s the strongest model of the RSI argument: a extra succesful AI might do analysis that as we speak’s mannequin can’t do, nonetheless many copies we run.
However constructing that higher mannequin additionally runs into diminishing returns. Extra coaching knowledge, extra coaching compute, bigger fashions, and extra reinforcement-learning (RL) compute all present diminishing returns in printed scaling research. Making dense fashions bigger normally raises the compute wanted for every output token, too. None of those routes provides us a free cross round the issue.
Determine 15. Diminishing returns to scaling. Chinchilla · ScaleRL · OpenAI.
These scaling outcomes give us motive to anticipate diminishing returns when AI helps construct the following mannequin, too.
AI capabilities are rising shortly. However the public knowledge doesn’t present a sustained acceleration. To the extent that AI instruments are boosting productiveness, they could be being offset by the issues rising more durable. Or we might merely be early. Both method, the pattern isn’t exhibiting a quick takeoff.
Determine 16. Frontier ECI features since January 2024. Supply.
The public ECI frontier-the finest rating amongst fashions launched by every date-has gained about 16 factors a 12 months on a pattern fitted from January 2024 by way of September 2026. That’s blisteringly quick progress, however this era doesn’t present a runaway surge.
Right here’s the identical frontier in absolute ECI factors, by way of July 2026, to place it in perspective.
Determine 17. Absolutely the frontier ECI rating. Supply.
The general public frontier can also’t inform us the whole lot occurring contained in the labs. Anthropic provides us a more in-depth look within the Opus 5.5 system card, utilizing its personal model of the index, AECI.
Determine 18. Anthropic’s fitted functionality pattern. Supply.
Eli Lifland, a co-author of AI 2027 and AI 2040, noticed the obvious pattern break as a warning that we had been heading towards an intelligence explosion:
“Anthropic might be proper right here [that they hadn’t reached dangerous levels of AI self-improvement], however alarm bells needs to be going off! Our processes should not able to deal with an intelligence explosion and we look like going full-steam forward towards one.”
– Eli Lifland, On Mythos’s AI R&D Capabilities
What regarded like acceleration now seems extra in line with a one-time leap. The extent went up. The speed hasn’t stored climbing.
Attaining these features has required an infinite improve within the inputs to AI. For instance, think about computing energy. Epoch’s estimates of AI chip capability, measured in NVIDIA H100 equivalents, present roughly 127-fold progress in simply over three years (together with projections on the finish of this era).
Determine 19. AI chip capability and frontier ECI. Supply: Epoch AI.
That is whole AI chip capability, together with inference. Nonetheless, the rise is placing: vastly extra computing capability has accompanied a lot steadier features in measured functionality.
The broader image seems comparable. Listed here are six inputs alongside functionality features, going again to February 2023.
Determine 20. Six inputs alongside frontier ECI. Epoch chip knowledge · SemiAnalysis workload shares.
All over the place we glance, AI has diminishing returns. It will get costlier in treasure and expertise to make every step ahead. Extra of each enter has been required to keep up regular features in AI capabilities.
We’ve been capable of scale these inputs as a result of, till not too long ago, the fee was inside the scope of what hyperscalers might pay from their earnings. That’s not the case. From this level ahead, future AI funding will more and more depend upon AI revenues going up. And the dimensions of the numbers – 3% of US GDP is now going into AI infrastructure – means that finally the expansion fee will decline. If funding progress does sluggish, to something lower than its present blistering exponential tempo, functionality progress might sluggish too. Even when funding progress continues (which I anticipate for the foreseeable future) a slowdown from its present exponential progress fee to a extra modest one (which I additionally anticipate) might result in a slower tempo of progress. Higher AI analysis instruments could also be wanted to offset that.
The day once we want higher AI instruments simply to proceed the tempo of AI progress might have already got arrived. Not as a result of funding is slowing, however as a result of the issue of enhancing AI itself will get more durable at every step.
Right here’s Anthropic within the Mythos 5.1 system card:
“we imagine that inner utilization of latest AI fashions has been a key consider sustaining the present fee of progress, however we don’t but see clear indicators of dramatic acceleration past that fee.”- Anthropic, Claude Fable 5.1 & Claude Mythos 5.1 System Card, part 2.3 – emphasis theirs.
The important thing phrase is maintaining-and Anthropic italicized that phrase in its personal system card. More and more succesful AI could also be important simply to maintain the tempo of enchancment the place it’s.
Opus 5.5 improves considerably on a number of coding and laptop use benchmarks. However on CoBench, Anthropic’s benchmark constructed from historic AI R&D issues, it features simply 2.6 proportion factors over Opus 5, inside the reported error bars.
Determine 21. Opus 5.5 benchmark features. Supply.
Why the smaller acquire right here? Possibly AI analysis is solely more durable than different duties. Keep in mind that CoBench isn’t testing the flexibility to supply vital discoveries. It’s rather more restricted in scope. It asks fashions to research historic AI R&D issues utilizing code, logs, and paperwork. That’s helpful analysis debugging and productiveness work, nevertheless it doesn’t straight take a look at whether or not a mannequin can invent a brand new structure or make a conceptual breakthrough.
The proof on open-ended analysis suggests one other impediment: arising with helpful concepts that haven’t already been tried.
Why do helpful new concepts typically get more durable to search out?
Tom Cunningham and Manish Shetty have a helpful apple-picking metaphor. An AI can decide the low-hanging fruit shortly, whereas people can nonetheless attain concepts the AI can’t.
As soon as these apples are picked, one other copy of the identical agent discovering them once more doesn’t assist. A stronger mannequin can attain greater. So as to add my very own flourish, the apples may additionally get sparser and farther aside as you climb. The RSI query is whether or not every harvest provides us sufficient to construct a greater apple-picker.
Determine 22. The apple-picking mannequin of AI R&D. Supply.
This sample reveals up throughout R&D. Bloom and colleagues doc fields the place analysis effort grows whereas analysis productiveness falls. A well-known instance is Eroom’s Legislation: within the historic drug-development knowledge, the inflation-adjusted R&D value per new permitted drug roughly doubled each 9 years.
Determine 23. Eroom’s Legislation in drug growth. Supply.
Pharma has different issues, together with regulation, tough medical trials, and rising expectations for security. Current therapies also can increase the bar for a helpful new drug. However a few of this problem may additionally be that the low-hanging fruit has been picked.
Stockfish, the chess engine, provides us a extra direct have a look at software program analysis. We have now data of experiments geared toward enhancing it and the features that adopted. This offers us a real-world dataset to have a look at the features of experimentation in software program. Consequently, a number of RSI fashions draw on this knowledge. That stated, not all of the enhancements got here from these experiments. A number of necessary concepts additionally got here from exterior the undertaking, so we shouldn’t give its experiments all of the credit score.
Epoch’s evaluation of software program R&D estimates returns to analysis effort at about 0.83 for Stockfish, a bit slower than linear. These are diminishing returns, however mild ones. These returns, nonetheless, are enhancements in computational effectivity. And extra compute doesn’t flip straight into extra AI functionality. As we noticed earlier, AI functionality additionally has steep diminishing returns from including extra computational energy. So we shouldn’t learn that 0.83 because the return from experimentation to AI functionality itself. AI functionality grows rather more slowly than compute, as we’ve seen already.
Andrej Karpathy’s autoresearch demonstration will get nearer to the method we need to perceive. A “trainer” AI agent adjustments a smaller “scholar” AI mannequin’s coaching code, runs it, checks the end result, and tries once more. The trainer agent itself doesn’t enhance, nevertheless it is ready to enhance the “learner”. That is my Kind 2: A stronger AI improves a weaker one.
One public run, posted by an agent working on Karpathy’s behalf, reported 89 experiments over roughly 7.5 hours. About 92% of that session’s acquire arrived by run 44. Features got here shortly, then slowed. The setup was intentionally small, with a five-minute coaching funds per experiment. However the agent might change the structure, optimizer, and coaching settings; it wasn’t restricted to a handful of knobs.
Determine 24. Features in a single autoresearch run. Supply.
A later public run received additional, so the primary run hadn’t hit a tough ceiling. This can be a helpful early instance of autonomous analysis, and yet one more place the place we see the diminishing returns endemic in AI analysis. That stated, this was a really early experiment. I anticipate future programs to do significantly better. This specific AI enchancment loop will doubtless develop stronger.
That is the place the excellence issues. Extra tokens should purchase extra code, and extra code might help us run extra experiments. However experiments solely enhance AI in the event that they uncover one thing helpful.
Determine 25. From AI exercise to helpful enhancements.
The larger query is whether or not AI can provide you with formidable new analysis concepts or conceptual breakthroughs.
Anthropic’s description of Opus 5.5 is blunt:
“As with earlier fashions, it’s weaker on open-ended analysis: inner customers report that it principally exams incremental concepts and prefers much less formidable hypotheses, and in our human-run biology train, it deferred to the printed literature and struggled to develop novel concepts (Part 2.2.2).”- Anthropic, Claude Opus 5.5 System Card, part 2.3.3; emphasis mine
METR’s evaluation in the identical card identifies what should still be lacking:
“That is extremely unsure, however we anticipate that full automation of AI R&D would require massive enhancements in foresight, prediction, creating one’s personal suggestions loops, and usually different expertise that may sometimes be known as researcher ‘judgement’ or ‘style’.”- METR, quoted within the Claude Opus 5.5 System Card, part 2.3.6
In these examples, people nonetheless provide a lot of the course and judgment.
Future fashions will in all probability get higher at this. However on the planet’s stockpile of potential coaching knowledge, we’ve got many extra examples of incremental work than of breakthroughs. I ponder whether that makes novelty more durable to study. That’s hypothesis, however value watching.
That is additionally robust to deal with by merely working extra copies of the AI. An enormous variety of parallel brokers might help with the incremental enhancements or looking over a big set of parameters, however for breakthrough concepts they could run into the homogeneity downside: Extra parallel brokers nonetheless assume alike.
How far are we from the self-improvement loop being sturdy sufficient to maintain itself, or to propel itself into runaway super-intelligence? Can we quantify this?
We are able to make a tough estimate. Higher AI helps with analysis; helpful analysis produces higher AI. For the loop to maintain itself, every spherical should produce sufficient features to propel the system by way of the following loop, at the same time as enhancements get more durable to find.
Determine 26. The AI self-improvement loop. Mannequin.
In a latest paper, The Economics of Recursive Self-Enchancment, Tom Cunningham and colleagues modeled this from the standpoint of how rather more productiveness each level of further ECI produces from an AI. They ask initially what that quantity would must be to create a self-sustaining suggestions loop. And secondly, they attempt to decide what that productivity-per-ECI-point quantity is as we speak.
First, they discover a self-sustaining RSI threshold of roughly 15% extra analysis productiveness per further ECI level. Of their mannequin, that’s about the place higher AI would generate sufficient progress to maintain the loop.
The image under reveals the thought. On the threshold, every cycle of features powers the following. Above the edge, the suggestions loop accelerates. Under the edge, the suggestions loop is simply too weak, and the speed of enchancment it brings drops on every cycle. This mannequin isolates the software program loop; exterior funding can nonetheless drive fast progress.
Determine 27. Three illustrative suggestions paths. Supply.
Updating this barely with knowledge from the Stockfish experiments places the edge a bit of greater, at roughly 19% per ECI level. I wouldn’t put a lot weight on that exact distinction. Each estimates are unsure. However they provide us a method to consider the energy of the suggestions loop and a tough band at which self-sustaining or runaway RSI might start.
The second factor Cunningham and staff do is make a tough estimate that the present AI productiveness acquire is about 9% per ECI level. That’s under their self-sustaining threshold.
I just like the mannequin. OpenAI’s newer knowledge, nonetheless, suggests the loop could also be fairly a bit weaker.
Cunningham’s estimate of 9% productiveness acquire per ECI level relies on Anthropic’s survey of 130 employees, who reported roughly 4× the productiveness they’d have with out AI. Cunningham and colleagues examine that with a 16-point functionality acquire since early Claude Code.
That comparability assumes the sooner instruments added little or no productiveness, so ‘no AI’ is an affordable place to begin. The authors say this explicitly. I’m unsure the idea holds for a similar researchers doing the identical work, however that’s a smaller situation.
The authors themselves know that it is a tough calculation, and warn that the 4× survey estimate might be too excessive.
OpenAI’s newer knowledge provides us a firmer approach to verify the quantity: Precise logged experiments over time, slightly than human estimates of their very own productiveness with and with out AI. I put extra weight on this for 3 causes:
-
Direct and broad measurement. As a substitute of counting on surveys, OpenAI truly tracked and measured experiments run on their infrastructure. Meaning they didn’t depend on researchers estimating their very own productiveness, which could be far off.
-
Full pattern, not opt-in. Equally, OpenAI’s knowledge catches each lively experimenter, whereas Anthropic’s solely displays the 130 staff who took the time to reply the survey – and who subsequently will not be a consultant set.
-
Enormously extra knowledge. We don’t know what number of experiments are within the 32 weeks of OpenAI knowledge, nevertheless it’s doubtless not less than tens of 1000’s of particular person examples and presumably a whole lot of 1000’s.
Any method you slice it, the brand new OpenAI knowledge, launched after Cunningham’s paper was drafted, is a bigger, extra complete, extra consultant, and nearly definitely extra correct dataset than Anthropic’s inner opt-in survey of staff.
Now let’s use OpenAI’s experiment knowledge to calibrate the productiveness acquire per ECI level. We all know that in August, OpenAI researchers ran ~1.6× as many experiments per individual per thirty days because the 2025 common. If we pair that with roughly 16 factors of frontier ECI enchancment, it really works backward to about 3% productiveness acquire per level of ECI. Against this, 9% compounded over 16 factors would imply roughly 4× productiveness.
Determine 28. Evaluating productiveness estimates. OpenAI strategies.
Right here’s OpenAI’s printed weekly sequence alongside that hypothetical path of 9% extra productiveness per further ECI level. The blue line ends at ~1.6×. The crimson line reveals what 9% per level would indicate if 16 ECI factors had been unfold throughout this era. That doesn’t match what we see from OpenAI’s knowledge. I need to be clear right here that every one knowledge units are noisy. We don’t know precisely what mannequin researchers had been utilizing on what days, or whether or not the brand new experiments had been additionally greater high quality than previous experiments. We want extra experiments and extra knowledge to additional calibrate these numbers. Working with what we do have, what we see is a fairly low enhance to productiveness from every further ECI level.
Determine 29. Experiment tempo versus a hypothetical path. Supply.
Even that 3% might give higher fashions an excessive amount of credit score. OpenAI additionally used way more tokens and had extra compute for experiments. These might account for among the improve in experiment tempo. So the vary might be a bit decrease.
I exploit 2–3% productiveness acquire per ECI level as a working assumption, permitting for some assist from these different inputs. That is nonetheless a tough estimate, albeit one which’s primarily based on the very best real-world knowledge we’ve got.
Determine 30. Productiveness estimates and the takeoff threshold. Supply.
With these assumptions, 2–3% per ECI level in opposition to a 15–19% threshold leaves a roughly five- to tenfold hole. That’s an enormous hole, although its dimension will depend on how properly experiment counts seize helpful analysis and whether or not the assumed functionality change is true.
Determine 31. Diminishing returns across the loop. Supply.
AI helps construct higher AI. Below this estimate, although, every flip of the loop provides lower than the final. The suggestions must turn into a lot stronger to maintain itself.
This software program loop sits alongside sooner chips, greater knowledge facilities, extra coaching knowledge, and larger funding. These can maintain driving fast progress even when the loop can’t maintain itself.
The loop itself might strengthen too. Higher coaching knowledge, reminiscence, and analysis judgment might all assist.
A breakthrough on the dimensions of the Transformer structure in 2017 might change the image rather more. That may be a superb motive to revisit these estimates.
Higher researchers may additionally run fewer experiments and study extra from each. A handful of higher concepts can matter greater than a mountain of routine runs.
Nonetheless, diminishing returns in machine studying aren’t new. Cortes and colleagues had been becoming machine studying scaling curves in 1993: Extra examples lowered error, following an influence regulation with diminishing returns. These diminishing returns and harsh scaling legal guidelines are as previous as machine studying. They didn’t seem for the primary time with transformers or LLMs or deep studying. That doesn’t show as we speak’s relationships will final without end. However till we see proof that we’ve discovered a brand new strategy that scales with out these inhibitors, we should always plan for diminishing returns as more likely to be with us for a while.
That stated, the world is extra than simply software program. Tom Davidson, Basil Halperin, Thomas Houlden, and Anton Korinek mannequin software program progress, {hardware} progress, and financial suggestions collectively. Higher AI helps design higher chips; higher chips help higher AI; financial progress funds extra funding in each. A number of suggestions loops can mix to beat diminishing returns even when one loop alone can’t. I believe it’s unbelievable that somebody has tried a mannequin that integrates all these totally different avenues of enhancing AI by way of software program, {hardware}, and economics.
However I’ve questions concerning the software program loop itself. Of their central calibration, absolutely automating software program analysis places that loop roughly on the threshold for explosive progress, even with out assist from higher {hardware} or broader financial progress. Recall that Cunningham’s mannequin places the self-sustaining threshold at roughly 15% extra analysis productiveness per further ECI level, whereas our estimate utilizing OpenAI’s experimental knowledge places as we speak’s features at solely 2-3%. These fashions use totally different measures, so we are able to’t equate their numbers straight. However the distinction issues: their absolutely automated software program loop reaches the edge, whereas our greatest estimate from present knowledge places as we speak’s loop far under it.
Having AI do all of the analysis doesn’t get rid of the diminishing returns inherent to enhancing AI, or the broader downside of helpful concepts getting more durable to search out. That is the excellence between Kind 4 and Kind 5 within the taxonomy above. An AI would possibly autonomously design, practice, and take a look at its successor, and nonetheless want exponentially extra assets to make every further step ahead. Closing the loop doesn’t inform us whether or not it’s sturdy sufficient to maintain itself.
The authors do account for diminishing returns. The priority is whether or not their calibration overestimates how a lot helpful AI analysis every spherical of software program enchancment produces. Diminishing returns look like basic to machine studying. We see them in coaching, in test-time compute, and within the seek for higher algorithms. Full autonomy might take away human bottlenecks with out eradicating any of these constraints.
We’ve already seen this inside autonomous analysis. Within the Karpathy autoresearch instance above, a lot of the features arrived early, and extra experiments purchased progressively much less enchancment. That was a small experiment with a set trainer mannequin, not a take a look at of absolutely autonomous RSI. It doesn’t settle the query. However it illustrates why eradicating the human from an experiment loop doesn’t, by itself, take away diminishing returns.
I do anticipate the suggestions loop to get stronger over time. Higher AI ought to turn into higher at analysis. However primarily based on our greatest present knowledge, reaching self-sustaining suggestions requires a loop roughly 5 to 10 instances stronger than as we speak’s. Treating absolutely automated software program analysis as already at that threshold is a considerable leap, earlier than we add the advantages of {hardware} enhancements or financial progress. I could possibly be fallacious, however I’d wish to see proof that autonomy brings sufficient further helpful discoveries to shut that hole.
On {hardware}, I’ve some additional reservations. The mannequin doesn’t explicitly embrace the years it will possibly take to show a chip design into deployed {hardware}. The authors focus on bodily bottlenecks, and I’d wish to see manufacturing and building delays constructed into the predictions.
I additionally marvel how a lot previous chip progress got here from higher concepts, and the way a lot relied on ever costlier factories and gear. If we give researchers an excessive amount of credit score for features that additionally wanted these investments, we might overestimate what sooner AI analysis alone would produce.
Even with these reservations, that is essentially the most compelling paper and mannequin I’ve seen for combining suggestions loops in software program, {hardware}, and economics to grasp how briskly they might push AI ahead. I’m not satisfied it establishes {that a} quick AI takeoff is feasible underneath sensible situations. Extra knowledge might assist us calibrate that judgment. However it provides us a helpful framework for understanding what might occur past the software program layer alone.
This is a crucial paper that helps us mannequin AI as a part of a broader financial system that may have bigger suggestions loops round it. I respect it, and I’m glad they wrote it.
These estimates relaxation on much less knowledge than I’d like. I is perhaps placing an excessive amount of weight on a couple of observations and reaching a comforting conclusion I need to imagine. We want higher measurements, shared typically sufficient to catch adjustments as they occur.
When OpenAI launched its analysis knowledge, Cheryl Wu welcomed the disclosure and identified how a lot was nonetheless lacking. Extra tokens and experiments are helpful issues to find out about. We additionally have to see how they flip into higher algorithms and extra succesful AI.
Cheryl Wu@cherylwoooo
I respect this as an preliminary step towards extra clear reporting on RSI. However there may be nonetheless a lot additional knowledge we have to absolutely perceive RSI.
Particularly, OAI disclosed some proof concerning the inference compute utilization, variety of experiments/researcher, and to a lesser…
Kevin Liu @kliu128
At this time we’re releasing knowledge on fashions accelerating analysis at OpenAI.
Recursive self-improvement could possibly be a very powerful contributor to AI capabilities over the following few years, however by default it is going to solely be seen inside a couple of frontier AI labs. Being clear is extra
11:25 PM · Sep 6, 2026 · 19.7K Views
9 Replies · 16 Reposts · 121 Likes
Determine 32. Cheryl Wu on OpenAI’s analysis knowledge. Supply.
Now Wu, Arjun Ramani, and Basil Halperin, with their colleagues on the Elasticity Institute, have written a concrete proposal: Tips on how to Measure RSI. It lists eight issues the labs might share to assist reply these questions. Test it out.
Determine 33. Eight proposals for measuring RSI. Supply.
I’d particularly wish to see how a lot helpful analysis every new mannequin provides, holding assets roughly fixed, and the way that analysis interprets into higher AI. That’s how we’ll study whether or not the loop is getting stronger.
AI is already serving to construct higher AI. It’s enhancing at a stupendous tempo, and I anticipate that to proceed. We have already got slender superintelligence in chess and Go. I anticipate more and more superhuman efficiency in elements of formal math, coding, and cybersecurity, and another verifiable area the place machines can generate coaching knowledge and confirm success at machine velocity. These are highly effective capabilities. That doesn’t imply we’re near super-intelligence for much less verifiable, messier, open-ended work – or to a normal ASI.
I’m skeptical of a quick takeoff to super-intelligence, however proof issues greater than hunches. Let’s accumulate the information we have to get a clearer image of what’s occurring. Together with proof that would change our minds. If higher AI begins producing sufficient helpful analysis to make the following spherical simpler, I need to know. If the features maintain shrinking, I need to know that too.