posts

Author: tecunningham

_freeze/posts/2025-09-13-recursive-self-improvement-explosion-optimization/execute-results/html.json

@@ -271,18 +271,18 @@ <h1 class="title">Recursive Self-Improvement, Literature Review</h1>
 <ul>
 <li>Basic introduction to the problem</li>
 <li>Survey of data: training expenditure (3X/year); training efficiency (4X/year); R&amp;D effort (2X/year).</li>
-<li>An overview of ~10 different models of RSI.</li>
+<li>A single canonical model, &amp; then an overview of ~10 different models of RSI</li>
 </ul></li>
 <li>TO DO:
 <ul>
-<li>A clearer distinction between two stages: (1) now-&gt;R&amp;D automation; (2) automation-&gt;singularity.</li>
-<li>A single canonical model upfront, &amp; then the review can talk about departures from that model.</li>
 <li>Survey of evidence on speedup &amp; autonomous optimization ability.</li>
 <li>Crisper statement of the balance of evidence.</li>
 </ul></li>
 </ol>
 </div>
 </div>
+<section id="summary" class="level1">
+<h1>Summary</h1>
 <dl>
 <dt>Summary.</dt>
 <dd>
@@ -294,18 +294,34 @@ <h1 class="title">Recursive Self-Improvement, Literature Review</h1>
 <p>We are now beginning to see signs of feedback, where AI is sufficiently capable to accelerate progress in R&amp;D: <span class="math display">\[
 \xymatrix@C=3em@R=1.4em{
     *++[F]{\text{R\&amp;D}}\ar[r]|(0.4)r
-    &amp; *++[F]{\text{AI Capabilities}}\ar@{.&gt;}@/_3em/[l]|{?}
+    &amp; *++[F]{\text{AI Capabilities}}\ar@{.&gt;}@/_3em/[l]|{a}
 }
 \]</span></p>
-<p>We have reason to believe that capability progress is fairly sensitive to the quantity of R&amp;D input. We can quantify the effect with <span class="math inline">\(r\)</span>, representing the . (<span class="math inline">\(r\approx 1\)</span>). However there is a great deal of uncertainty on how capabilities will affect R&amp;D.</p>
-<p>We can break the effects into two types, and make some very tentative guesses as of April 2026:<a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a></p>
+</dd>
+<dt>In the pre-automation world we care about <span class="math inline">\(a\)</span>.</dt>
+<dd>
+<p>The discussion of recursive self-improvement breaks down into two very distinct questions depending on whether we have already achieved AI research automation, i.e.&nbsp;agents that are at least as good as human researchers.<a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a></p>
+<p>In the pre-automation world we are most interested in the degree to which AI accelerates human researchers, the variable <span class="math inline">\(a\)</span> in the diagram above. We can distinguish between augmentation of human researchers (“uplift”, “acceleration”), and partial-automation of research (autonomous research agents).</p>
+<p>We can make a very rough guess at our current state, as of April 2026:<a href="#fn2" class="footnote-ref" id="fnref2" role="doc-noteref"><sup>2</sup></a></p>
 <ol type="1">
 <li>Augmentation: agents are accelerating researcher productivity by 50%.</li>
-<li>Automation: agents are able to make autonomous contributions equivalent to around 1 month of a researcher’s work.</li>
+<li>Partial Automation: agents are able to make autonomous contributions equivalent to around 1 month of a researcher’s work.</li>
 </ol>
-<p>The critical uncertainty is how these effects <em>scale</em>. If R&amp;D acceleration continues to scale with capabilities growth then we would expect an imminent substantial acceleration over the historical rate of capabilities growth.</p>
+<p>Even if we confidently knew the levels of augmentation and partial-automation, it’s also important to know how these scale with AI capabilities progress.</p>
+</dd>
+<dt>In the post-automation world we care about <span class="math inline">\(r\)</span>.</dt>
+<dd>
+<p>In the post-automation world the effective stock of R&amp;D labor will suddenly expand, and so there will clearly be a rapid advance in AI capabilities, the question is how quickly that will explode. Many formal models assume that <span class="math inline">\(a=1\)</span>, and switch their attention to <span class="math inline">\(r\)</span>. A justification of <span class="math inline">\(a=1\)</span> is, if we interpret AI capabilities as algorithmic efficiency, then a 1% increase in efficiency implies at least a 1% increase in effective R&amp;D labor.</p>
+<p>The critical questions for <span class="math inline">\(r\)</span> then become potential bottlenecks on R&amp;D inputs:</p>
+<ol type="1">
+<li>The limited supply of inference compute for automated running researchers;</li>
+<li>The limited supply of compute for running research experiments;</li>
+<li>Logical limits on research progress, e.g.&nbsp;statistical ceilings on algorithmic efficiency.</li>
+</ol>
+<p>Most work on estimating post-automation effects doesn’t focus on measuring the abilites of AI, but instead measuring historical returns to R&amp;D inputs, and the substitutability between different types of inputs.</p>
 </dd>
 </dl>
+</section>
 <section id="argument" class="level1">
 <h1>Argument</h1>
 <dl>
@@ -328,7 +344,7 @@ <h1>Argument</h1>
 <dd>
 <p>Until recently AI R&amp;D was mostly done without significant help from AI, but we now see evidence for two channels:</p>
 <ol type="1">
-<li><em>Augmenting researchers:</em> AI researchers self-report big efficiency gains, e.g. <span class="citation" data-cites="anthropic2025claude_work">Anthropic (<a href="#ref-anthropic2025claude_work" role="doc-biblioref">2025</a>)</span> self-report approximately 50% productivity gains, and <span class="citation" data-cites="anthropic2026risk">Anthropic (<a href="#ref-anthropic2026risk" role="doc-biblioref">2026</a>)</span> estimate 100% productivity gains.<a href="#fn2" class="footnote-ref" id="fnref2" role="doc-noteref"><sup>2</sup></a></li>
+<li><em>Augmenting researchers:</em> AI researchers self-report big efficiency gains, e.g. <span class="citation" data-cites="anthropic2025claude_work">Anthropic (<a href="#ref-anthropic2025claude_work" role="doc-biblioref">2025</a>)</span> self-report approximately 50% productivity gains, and <span class="citation" data-cites="anthropic2026risk">Anthropic (<a href="#ref-anthropic2026risk" role="doc-biblioref">2026</a>)</span> estimate 100% productivity gains.<a href="#fn3" class="footnote-ref" id="fnref3" role="doc-noteref"><sup>3</sup></a></li>
 <li><em>Automating research:</em> Autonomous systems are making contributions to frontier R&amp;D, e.g.&nbsp;AlphaEvolve, TTT-Discover, autoresearch.</li>
 </ol>
 <p>Both of these effects are hard to measure, &amp; we have a great deal of uncertainty.</p>
@@ -1043,7 +1059,7 @@ <h2 class="anchored" data-anchor-id="jones1995rd-rd-based-models-of-economic-gro
 \end{gathered}
 \]</span></p>
 <p>Then <span class="math inline">\(g_A = \frac{\dot{A}}{A} = R^\lambda A^{-\beta}.\)</span></p>
-<p>So if <span class="math inline">\(R\)</span> is constant, <span class="math inline">\(g_A\)</span> declines as <span class="math inline">\(A\)</span> rises.<a href="#fn3" class="footnote-ref" id="fnref3" role="doc-noteref"><sup>3</sup></a> If <span class="math inline">\(R\)</span> grows at rate <span class="math inline">\(g_R\)</span> along a balanced growth path, then <span class="math inline">\(g_A = \frac{\lambda}{\beta}g_R.\)</span></p></li>
+<p>So if <span class="math inline">\(R\)</span> is constant, <span class="math inline">\(g_A\)</span> declines as <span class="math inline">\(A\)</span> rises.<a href="#fn4" class="footnote-ref" id="fnref4" role="doc-noteref"><sup>4</sup></a> If <span class="math inline">\(R\)</span> grows at rate <span class="math inline">\(g_R\)</span> along a balanced growth path, then <span class="math inline">\(g_A = \frac{\lambda}{\beta}g_R.\)</span></p></li>
 <li><p>Recursive self-improvement, where knowledge directly raises research input: <span class="math display">\[\begin{gathered}
 R=A^\kappa\\
 \dot{A}=(A^\kappa)^\lambda A^{1-\beta}=A^{\lambda\kappa+1-\beta}\\
@@ -1839,9 +1855,10 @@ <h2 class="anchored" data-anchor-id="david-rein-2025-modeling-an-intelligence-ex
 </div></section><section id="footnotes" class="footnotes footnotes-end-of-document" role="doc-endnotes"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>
 
 <ol>
-<li id="fn1"><p>Ryan Greenblatt in April 2026 <a href="https://www.lesswrong.com/posts/WjaGAA4xCAXeFpyWm/my-picture-of-the-present-in-ai">estimates</a> the speedup to engineering to be around 1.6x, and the autonomous capability to be around 5 hours (“the task duration at which AIs match a randomly selected AI company engineer (who is familiar with that part of the code base) is around 5 hours”). Note that a more robust statistic would be the number of hours.<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
-<li id="fn2"><p>“Productivity uplift estimates ranged from 30% to 700%, with a mean of 152% and median of 100%.”<a href="#fnref2" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
-<li id="fn3"><p><span class="citation" data-cites="jones1995rd">Charles I. Jones (<a href="#ref-jones1995rd" role="doc-biblioref">1995</a>)</span> introduced diminishing returns to knowledge, whereas Romer (1990) had assumed no diminishing returns to knowledge, <span class="math inline">\(\beta=0\)</span>.<a href="#fnref3" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
+<li id="fn1"><p><span class="citation" data-cites="davidson2025howquickandbigwo">Davidson and Houlden (<a href="#ref-davidson2025howquickandbigwo" role="doc-biblioref">2025</a>)</span> calls this ASARA, “AI System for AI R&amp;D Automation”, <span class="citation" data-cites="kokotajlo2025aifuturesmodel">Kokotajlo et al. (<a href="#ref-kokotajlo2025aifuturesmodel" role="doc-biblioref">2025</a>)</span> calls this SAR, “superhuman AI researcher”.<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
+<li id="fn2"><p>Ryan Greenblatt in April 2026 <a href="https://www.lesswrong.com/posts/WjaGAA4xCAXeFpyWm/my-picture-of-the-present-in-ai">estimates</a> the speedup to engineering to be around 1.6x, and the autonomous capability to be around 5 hours (“the task duration at which AIs match a randomly selected AI company engineer (who is familiar with that part of the code base) is around 5 hours”). Note that a more robust statistic would be the number of hours.<a href="#fnref2" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
+<li id="fn3"><p>“Productivity uplift estimates ranged from 30% to 700%, with a mean of 152% and median of 100%.”<a href="#fnref3" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
+<li id="fn4"><p><span class="citation" data-cites="jones1995rd">Charles I. Jones (<a href="#ref-jones1995rd" role="doc-biblioref">1995</a>)</span> introduced diminishing returns to knowledge, whereas Romer (1990) had assumed no diminishing returns to knowledge, <span class="math inline">\(\beta=0\)</span>.<a href="#fnref4" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
 </ol>
 </section></div></main> <!-- /main -->
 <script id="quarto-html-after-body" type="application/javascript">