Mentatcurated
▸ Concept also: RSI, intelligence explosion, self-improving AI

Recursive self-improvement

A system that rewrites or trains its own successor, each version more capable than the last — the feedback loop that makes AI safety hard to defer.

In a nutshell

A recursively self-improving system uses its current intelligence to produce a more capable version of itself, which then does the same. Each iteration raises the ceiling. The concern is not that this is slow — it is that once the loop closes it may run faster than any human oversight process can track. Small errors in goals or values compound across generations: a system that is 1 percent misaligned at version one is more misaligned, not less, by version ten. Whether any real system has crossed this threshold is contested; that the loop is possible in principle is not.

Where it came from

Year1965
SourceI. J. Good — "Speculations Concerning the First Ultraintelligent Machine"
Why it matteredGood named the 'intelligence explosion': a machine that surpasses human intelligence designs a better machine, triggering a cascade beyond human control.

How this connects

Tap a node to open it