+ "markdown": "---\ntitle: Recursive Self-Improvement\nauthor: Tom Cunningham\ndate: today\ndraft: true\nengine: knitr\nexecute:\n echo: false\n warning: false\n error: false\n cache: true # caches chunk output\neditor:\n render-on-save: true\n---\n<style>\n h1 { border-bottom: 4px solid black; }\n h2 { border-bottom: 1px solid gray; padding-bottom: 0px; color: black; }\n dl {display: grid;}\n dt {grid-column-start: 1; width: 4cm;}\n dd {grid-column-start: 2; margin-left: 2em;}\n</style>\n\n# Summary\n\nTL;DR: an explosion is AI capabilities is *possible* but it's intrinsically difficult to forecasat.\n\n: Define an explosion as a significant acceleration of the historical rates of progress in AI, e.g. measured by effective compute required for a given level of accuracy (e.g. perplexity).\n\nProgress in AI is rapid but smooth.\n\n: We can measure progress in various ways, algorithmic progress has accelerated over the last 10-15 years. Epoch estimate that algorithmic efficiency has been growing at 3X/year between 2012-2024, measured by effective compute.\n\nForecasters expect an 8-20% chance of an explosion by 2030.\n: They define an explosion as a 3X speedup in the historical rate of progress by some measure.\n\nAI has already been accelerating AI research.\n\n: AI has been self-accelerating for decades: (1) automatic differentiation; (2) bayesian optimization of hyperparameters; (3) neural architecture search; (4) LLM coding autocomplete and chatbots; (5) LLM coding agents.\n\nThere's good theoretical reason to expect an explosion.\n\n: If AI increases AI, then it will.\n\nAI usefulness for optimization depends on features of the setup.\n\n: Hassabis says AI will make progress wherever there's (1) combinatorial search space; (2) clear feedback; (3) lots of data or an automatic validator. But there's clearly a fourth condition: that the data has some lower-dimensional latent structure. There are many problems that satisfy the first 3 but where we don't expect substantial progress from autonomous NNs: (A) the telephone book; (B) mapping out stars in the sky; (C) documenting the genome.\n\n\n\n# Models\n\nAK model of recursive growth.\n\n: Suppose we have $Y=AK^\\alpha$, and also we use some share $s$ of output on R&D, which increases productivity $A=sY^\\beta$, then if $\\beta+\\alpha>1$ we get a continuously increasing growth rate. Aghion, Jones and Jones (2017) elaborate on this model, where AI gradually is able to take over human tasks, & they get a similar condition.\n\n: The critical question is whether the returns to R&D effort are sufficiently steep. Some classic papers: Bloom et al. \"are we running out of ideas\"; and Erdil. \n\n# Figure / Evals\n\nQ: what evals would be useful? Suppose we can plot the following:\n\n\n::: {.cell}\n::: {.cell-output-display}\n{width=672}\n:::\n:::\n\n\nWe could plot data from a variety of differnt experiments on this plot, & ask some interesting questions:\n\n1. At what time horizon do we forecast models beating human-level ability at training models?\n2. Do we see grokking, i.e. discontinuous jumps in model-training ability?\n3. Are some models better than others at model-training?\n\n\n\n# References\n\nFRI/METR forecasts of AI progress.\n: (...)\n\nKortum (1997) [\"Research, Patenting, and Technological Change\"](https://egc.yale.edu/sites/default/files/Kortum_1997.pdf)\n: He assumes that technological progress comes from taking random draws from a distribution (undirected search). Then growth over time will be characterized by the extreme value distribution of the underlying distribution, & returns to experience will depend on the thickeness of tails. Alternative assumptions:\n\n - Bounded: growth slows as it approaches a ceiling.\n - Exponential (thin tails): $A=\\ln n$\n - Pareto (thick tails): $A = n^\\gamma$\n\nErdil, Besiroglu, Ho (2024) [\"Estimating Idea Production: A Methodological Survey\"](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4814445)\n: They discuss the difficulty of estimating the returns to R&D on productivity across a few different domains. They have nice estimates for chess inputs & outputs. They also estimate algorithmic progress:\n\n - Computer vision: doubling time 9 months 2012-2022.\n - RL sample efficiency: doubling time 11 months, 2015-2019.\n - SAT solvers: doubling time 2 years, 1997-2018.\n - Linear programming: doubling time 6 years, 1998-2018.\n\n They include a nice derivation of the critical threshold for hyperbolic growth:\n\n $$\\begin{aligned}\n Y &= AK^\\alpha \\\\\n \\dot{K} &\\propto Y && \\text{(constant savings rate)}\\\\\n \\dot{A}/A &\\propto A^{-\\beta}K^\\lambda && \\text{(productivity growth)}\n \\end{aligned}$$\n\n You'll get steady-state growth if and only if $\\lambda/\\beta+\\alpha=1$. You'll get hyperbolic growth if $\\lambda/\\beta+\\alpha>1$.\n\n\nAghion, Jones, and Jones (2019) \"Artificial Intelligence and Economic Growth\"\n: They give conditions under which automating R&D will cause an explosion in GDP growth (i.e. a steadily increasing rate of growth, AKA hyperbolic growth).\n\n\n<!-- $$\\begin{aligned}\n Y_t &= A_t^{\\sigma} K_t^{\\alpha} L^{1-\\alpha} \\\\\n \\dot A_t &= K_t^{\\beta} S^{\\lambda} A_t^{\\phi}\n \\end{aligned}$$\n\n In task-based automation interpretations, ($\\alpha$) and ($\\beta$) reflect the fractions of tasks automated in goods and idea production. Aghion, Jones and Jones analyze when AI-driven automation can yield accelerating growth (“Type I”) or even a singularity (“Type II”). in their Cobb–Douglas setup a key threshold is ($\\gamma = \\frac{\\sigma}{1-\\alpha}\\cdot\\frac{\\beta}{1-\\phi}$), with a Type II singularity when ($\\gamma>1$). -->\n\n\nEric Drexler (2025) [\"The Reality of Recursive Self-Improvement\"](https://aiprospects.substack.com/p/the-reality-of-recursive-improvement)\n: He argues that . Automating steps in the optimization loop:\n - Automating differentiation.\n - Neural architecture search.\n - Bayesian optimization.\n\n > \"The fundamental mechanism is systemic friction reduction — aggregate improvements expand possibilities by enabling faster progress and more ambitious goals\n \n > \"In hyperparameter optimization, advances in Bayesian and multi-fidelity methods often achieve order-of-magnitude savings compared to naive grid search. What once required thousands of full model trainings can now be accomplished with fewer and more intelligent probes. As daunting costs of innovation fall, research becomes faster and more ambitious.\n\n > \"The trajectory toward comprehensive AI capabilities makes these developments predictable, not in detail, but in outline.\n\n\n\nEric Drexler (Aug 2025) [\"The Reality of Recursive Self-Improvement\"](https://aiprospects.substack.com/p/the-reality-of-recursive-improvement)\n\n: Automating steps in the optimization loop:\n\n - Automating differentiation.\n - Neural architecture search.\n - Bayesian optimization.\n\n: > \"The fundamental mechanism is systemic friction reduction — aggregate improvements expand possibilities by enabling faster progress and more ambitious goals\n\n > \"In hyperparameter optimization, advances in Bayesian and multi-fidelity methods often achieve order-of-magnitude savings compared to naive grid search. What once required thousands of full model trainings can now be accomplished with fewer and more intelligent probes. As daunting costs of innovation fall, research becomes faster and more ambitious.\n\n > \"The trajectory toward comprehensive AI capabilities makes these developments predictable, not in detail, but in outline.\n\n\n# We need more bottom-up modelling of AI's economic effect [UNFINISHED]\n\nThere are two approaches to modelling AI's effect.\n\n: From a lot of conversations about recursive self-improvement I realized it's useful to distinguish between two qualitatively different ways of thinking about AI's ability to do work:\n\n 1. _Top down:_ AI replaces each of the human subtasks.\n 2. _Bottom-up:_ AI just does the entire procedure from first principles.\n \n I think 80% of discussion of economic impacts was of the top-down type.\n\nWhat would bottom-up modelling look like?\n: There are some papers with \"model organisms\" of recursive self-improvement: Grefenstette (1986) [genetic algorithm to learn parameters for genetic algorithsm](https://ui.adsabs.harvard.edu/abs/1986ITSMC..16..122G/abstract); [AutoML-Zero](https://arxiv.org/pdf/2003.03384) (2020); Schmidhuber (2003) [Godel Machines](https://arxiv.org/abs/cs/0309048?utm_source=chatgpt.com). \n\nWhat are the implications for recursive self-improvement?\n: AI treats it as a pure optimization problem, already it's better at chip design, algorithm design.\n\n\nSome related discussion:\n: Nathan Lambert recapitulates some discussions from the Curve about recursive self-improvement [here](https://www.interconnects.ai/p/thoughts-on-the-curve).\n\n This is related to the Bresnahan/systems view of AI. He talks about the first wave of ML models: \"The transition to ICT-based production has largely proceeded at the production system level, not the task level.\"[^bresnahan] \n\n\n[^bresnahan]: \"My empirical conclusion about these applications is that the lazy idea of AI – that is, of computer systems that are able to perform productive tasks previously done by humans– is irrelevant to understanding how these technologies create value. Here“irrelevant” does not mean that substitution of machine for human tasks is less important than other determinants of the value in use of AITs. It means irrelevant: task-level substitution of machine for human plays no role in these highly valuable systems.\"\n",
0 commit comments