You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<li>Suppose higher levels of model capability <spanclass="math inline">\(A\)</span> increases the effective R&D labor, <spanclass="math inline">\(R\)</span>, by some amount, <spanclass="math inline">\(R=\bar{R}A^\theta\)</span>.</li>
274
+
<li>We want to measure how <spanclass="math inline">\(A\)</span> affects <spanclass="math inline">\(R\)</span>, e.g. by running an uplift study with different levels of <spanclass="math inline">\(A\)</span>.</li>
275
+
<li>We can then plug this into a standard R&D model to estimate the net degree of acceleration, with <spanclass="math inline">\(\dot{A}=R^\gamma A^{1-\beta}\)</span>.</li>
276
+
<li>However it’s really hard to estimate the relationship between model capability and R&D worker speedup. People are poor at introspecting uplift, and it’s difficult to run experiments.</li>
277
+
<li>Notes:
282
278
<ul>
283
-
<li>There’s a distribution of tasks, with probability f(t).</li>
284
-
<li>An LLM is just a lookup table, you slowly expand coverage, collecting a larger fraction of tasks t.</li>
285
-
</ul>
286
-
</dd>
287
-
<dt>Model D: layer cake.</dt>
288
-
<dd>
279
+
<li>The augmentation could take place through automation of subtasks, where the subtasks are complements. E.g. if they’re perfect complements then the acceleration will be <spanclass="math inline">\(\frac{1}{1-f}\)</span>, where <spanclass="math inline">\(f\)</span> is the share of tasks you’ve automated.</li>
280
+
<li>Any <spanclass="math inline">\(\theta>0\)</span> will make growth super-exponential, AKA hyperbolic, but the magnitude matters.</li>
281
+
<li>If <spanclass="math inline">\(\theta\)</span> is sufficiently high (<spanclass="math inline">\(\theta>\gamma-\beta\)</span>) then there will be self-sustaining growth, i.e. growth even if R&D labor was constant.</li>
282
+
<li>We ignore expenditure on inference, assume that it asymptotes relatively quickly (seems reasonable).</li>
283
+
</ul></li>
284
+
</ul></li>
285
+
<li><p><strong>AI automation of R&D:</strong></p>
289
286
<ul>
290
-
<li><p>You have a function that turns money->intelligence, & that function gets more efficient over time, it’s falling at about 8X/year.</p></li>
291
-
<li><p>You can separate the optimization problem into layers, they are partially separable:</p>
287
+
<li>Suppose instead that model capability <spanclass="math inline">\(A\)</span> can <em>autonomously</em> do AI R&D work, i.e. it’s a perfect substitute for all human labor.</li>
288
+
<li>Then we should expect a rapid adjustment in wages and model prices until they’re in equilibrium.</li>
289
+
</ul>
292
290
<oltype="1">
293
-
<li>GPU design</li>
294
-
<li>Model architecture</li>
295
-
<li>Pretraining optimizer</li>
296
-
<li>Pretraining data</li>
297
-
<li>Posttraining algorithm (RL)</li>
298
-
<li>Posttraining data</li>
299
-
<li>Elicitation/harness</li>
291
+
<li>Augmentation: it makes each R&D worker more productive.</li>
292
+
<li>Automation:</li>
300
293
</ol></li>
301
-
<li><p>The cost of an experiment is different for different parts. Some parts scale additively (GPU design), other parts its hard to predict effect of scale (architecture). If they scale additively then the cost of experimentation is small (but expenditure could still be large).</p></li>
302
-
</ul>
303
-
</dd>
304
-
</dl>
294
+
<li><p><strong>Complications:</strong></p>
295
+
<ul>
296
+
<li><strong>Measuring capability.</strong></li>
297
+
<li><strong>Scale-dependent algorithmic progress.</strong><spanclass="citation" data-cites="gundlach2025algorithmicprogressai">Gundlach et al. (<ahref="#ref-gundlach2025algorithmicprogressai" role="doc-biblioref">2025</a>)</span> argue that algorithmic progress has contributed much less.</li>
298
+
<li><strong>Data contribution.</strong> Berren Millidge argues <ahref="https://www.beren.io/2025-08-02-Most-Algorithmic-Progress-is-Data-Progress/">“Most Algorithmic Progress is Data Progress”</a></li>
<li>Based on history of AI R&D, what’s a reasonable estimate of how AI R&D will change the future trajectory?</li>
873
+
<li>What is the best capability metric for estimating AI R&D contribution?</li>
874
+
</ol>
875
+
<p>Basic argument:</p>
875
876
<oltype="1">
876
-
<li>Algorithmic progress has been proceeding at around 5X/year.</li>
877
+
<li>Say algorithmic efficiency has been growing at 5X/year, so we can get the same intelligence for 5X lower cost.</li>
878
+
<li>We are starting to see glimmers of AI contributing to R&D, & want to predict how this will change the system.</li>
879
+
<li>Some alternatives in how to model this:
880
+
<ul>
881
+
<li><p><strong>Accelerant.</strong> Suppose a certain level of AI can speed up a human researcher by a certain proportion, <spanclass="math inline">\(\lambda\)</span>. This feedback loop should accelerate AI development, but it’s sensitive to how strong this connection is.</p></li>
882
+
<li><p><strong>Perfect substitutes.</strong> Suppose AI can do entirely autonomous research, i.e. it’s a perfect substitute for a human. Then we are only constrained on the cost of AI compute. Unless the cost is very high, this implies a massive acceleration in AI progress, constrained only by (a) experiment bottlenecks; (b) a ceiling to AI progress.</p></li>
<li>Endogenous growth setup: (1) diminishing returns to R&D about 1/2; (2) increasing returns to knowledge about 1/2; these balance and get balanced growth.</li>
878
886
<li>Now we add link back from algorithms to R&D, but very little guidance on how to calibrate it.</li>
879
887
</ol>
888
+
<p>Capability metrics:</p>
889
+
<oltype="1">
890
+
<li>Human researcher uplift.</li>
891
+
<li>Ability to push forward the frontier.</li>
892
+
</ol>
893
+
<p>Wrinkles:</p>
894
+
<oltype="1">
895
+
<li>Estimating algorithmic progress.</li>
896
+
<li>Scale-dependent algorithmic progress.</li>
897
+
<li>Bottlenecks on experiment compute.</li>
898
+
<li>Shape of the landscape.</li>
899
+
<li>Adding capital.</li>
900
+
</ol>
880
901
<p>Ajeya’s questions:</p>
881
902
<oltype="1">
882
-
<li>When will we have automated AI R&D?</li>
903
+
<li>When will we have automated AI R&D?
904
+
<ul>
905
+
<li>Answer: we already have automated R&D.</li>
906
+
</ul></li>
883
907
<li>When we have automated AI R&D, will this trigger super-exponential growth?</li>
@@ -896,7 +932,7 @@ <h1>Literature Review on RSI Models</h1>
896
932
</thead>
897
933
<tbody>
898
934
<trclass="odd">
899
-
<td><spanclass="citation" data-cites="jones1995rd">C. I. Jones (<ahref="#ref-jones1995rd" role="doc-biblioref">1995</a>)</span> “R&D-Based Models of Economic Growth.”</td>
935
+
<td><spanclass="citation" data-cites="jones1995rd">Charles I. Jones (<ahref="#ref-jones1995rd" role="doc-biblioref">1995</a>)</span> “R&D-Based Models of Economic Growth.”</td>
900
936
<td>Standard endogenous growth model: (1) diminishing returns to R&D; (2) positive spillovers from knowledge.</td>
901
937
</tr>
902
938
<trclass="even">
@@ -983,23 +1019,26 @@ <h1>Literature Review on RSI Models</h1>
<h2class="anchored" data-anchor-id="jones1995rd-rd-based-models-of-economic-growth"><spanclass="citation" data-cites="jones1995rd">C. I. Jones (<ahref="#ref-jones1995rd" role="doc-biblioref">1995</a>)</span> “R&D-Based Models of Economic Growth”</h2>
988
-
<p>Basic model with accumulating ideas: <spanclass="math display">\[\begin{gathered}\dot{A}=R^{\gamma}\\
989
-
\xymatrix{*++[F]{R\&D} \ar[r] & *++[F]{\Delta knowledge}\ar[r] & *++[F]{knowledge}}
1024
+
<h2class="anchored" data-anchor-id="jones1995rd-rd-based-models-of-economic-growth"><spanclass="citation" data-cites="jones1995rd">Charles I. Jones (<ahref="#ref-jones1995rd" role="doc-biblioref">1995</a>)</span> “R&D-Based Models of Economic Growth”</h2>
1025
+
<p>Best reference is the review article in <spanclass="citation" data-cites="jones2022semiendogenous">Charles I. Jones (<ahref="#ref-jones2022semiendogenous" role="doc-biblioref">2022</a>)</span>.</p>
1026
+
<p>Basic model with diminishing returns to R&D: <spanclass="math display">\[\begin{gathered}\dot{A}=R^{\gamma}\\
1027
+
\xymatrix{*++[F]{R\&D} \ar[r]|(0.4)\gamma & *++[F]{\Delta knowledge}\ar[r] & *++[F]{knowledge}}
990
1028
\end{gathered}
991
1029
\]</span></p>
992
-
<p>Model with shoulders/armpits of giants (Jones model):<ahref="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a><spanclass="math display">\[\begin{gathered}
1030
+
<p>Now we add positive spillovers (“shoulders of giants”). If <spanclass="math inline">\(R\)</span> is constant then you’ll get declining growth rate. If <spanclass="math inline">\(R\)</span> is growing at <spanclass="math inline">\(g_R\)</span> then you’ll get <spanclass="math inline">\(g_A=\frac{\alpha}{\beta}g_R\)</span>.<ahref="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a><spanclass="math display">\[\begin{gathered}
993
1031
\dot{A}=R^{\gamma}A^{1-\beta}\\
994
-
\xymatrix{*++[F]{R\&D} \ar[r] & *++[F]{\Delta knowledge}\ar[r] & *++[F]{knowledge}\ar@/^2em/[l]}
1032
+
\xymatrix{*++[F]{R\&D} \ar[r]|(0.4)\gamma & *++[F]{\Delta knowledge}\ar[r] & *++[F]{knowledge}\ar@/^2em/[l]|{1-\beta}}
995
1033
\end{gathered}
996
1034
\]</span></p>
997
-
<p>Recursive self-improvement, where knowledge actually helps R&D: <spanclass="math display">\[\begin{gathered}
1035
+
<p>Recursive self-improvement, where knowledge directly helps R&D: <spanclass="math display">\[\begin{gathered}
998
1036
R=A^{\kappa}\\
999
1037
\dot{A}=A^{1-\beta+\kappa}\\
1000
-
\xymatrix{*++[F]{R\&D} \ar[r] & *++[F]{\Delta knowledge}\ar[r] & *++[F]{knowledge}\ar@/^2em/[l]\ar@/^3em/[ll]}
1038
+
\xymatrix{*++[F]{R\&D} \ar[r]|(0.4)\gamma & *++[F]{\Delta knowledge}\ar[r] & *++[F]{knowledge}\ar@/^2em/[l]|{1-\beta}\ar@/^4em/[ll]|\kappa}
1001
1039
\end{gathered}
1002
1040
\]</span></p>
1041
+
<p>Here you will get constant exponential growth if and only if <spanclass="math inline">\(\beta=\kappa\)</span>.</p>
1003
1042
<dl>
1004
1043
<dt>Extrapolating from these models.</dt>
1005
1044
<dd>
@@ -1329,6 +1368,9 @@ <h2 class="anchored" data-anchor-id="david-rein-notes">David Rein notes</h2>
Jones, Charles I. 1995. <span>“R&d-Based Models of Economic Growth.”</span><em>Journal of Political Economy</em> 103 (4): 759–84. https://doi.org/<ahref="https://doi.org/10.1086/262002">https://doi.org/10.1086/262002</a>.
Jones, Charles I. 2022. <span>“The Past and Future of Economic Growth: A Semi-Endogenous Perspective.”</span><em>Annual Review of Economics</em> 14 (1): 125–52. <ahref="https://doi.org/10.1146/annurev-economics-080321-033317">https://doi.org/10.1146/annurev-economics-080321-033317</a>.
<liid="fn1"><p>This was introduced by <spanclass="citation" data-cites="jones1995rd">C. I. Jones (<ahref="#ref-jones1995rd" role="doc-biblioref">1995</a>)</span>, where there are diminishing returns to knowledge, whereas Romer (1990) had assumed no diminishing returns to knowledge, <spanclass="math inline">\(\beta=0\)</span>.<ahref="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
1401
+
<liid="fn1"><p>This was introduced by <spanclass="citation" data-cites="jones1995rd">Charles I. Jones (<ahref="#ref-jones1995rd" role="doc-biblioref">1995</a>)</span>, where there are diminishing returns to knowledge, whereas Romer (1990) had assumed no diminishing returns to knowledge, <spanclass="math inline">\(\beta=0\)</span>.<ahref="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
@@ -270,7 +278,7 @@ <h1 class="title">An Apple-Picking Model of AI R&D</h1>
270
278
<dt>A simple model for AI R&D.</dt>
271
279
<dd>
272
280
<p>There are signs that AI agents are able to make genuine contributions to to AI R&D, how should we interpret this? Is recursive self-improvement imminent?</p>
273
-
<p>Here I describe a simple “apple-picking” model of optimization problems in which AI agents can make genuine autonomous contributions, without being able to replace human researchers. I’m not confident that the model is correct, maybe we are on the verge of human takeover, but I find this model helpful for thinking about what evidence we would need.</p>
281
+
<p>Here I describe a simple “apple-picking” model of optimization problems in which AI agents can make genuine autonomous contributions, without being able to replace human researchers. I’m not confident that the model is correct, maybe we are indeed on the verge of AI takeover, but I find the model helpful for thinking about what evidence we would need.</p>
274
282
<p>An agent helping you to optimize an algorithm is like a robot helping you pick apples. It will take care of all the apples up to a certain height, and it may find apples you haven’t found yet, but there will still be apples out of its reach.</p>
275
283
<p>Below I give a formal model but the basic ideas can all be seen in the drawing below. Here both the human and robot have picked four apples, but they’ve left the tree in a very different state, so the robot isn’t ready to replace the human yet:</p>
Copy file name to clipboardExpand all lines: posts/2025-09-13-recursive-self-improvement-explosion-optimization-offcuts.qmd
+33Lines changed: 33 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -18,6 +18,7 @@ execute:
18
18
19
19
==OFFCUTS FILE==
20
20
21
+
21
22
# Summary
22
23
23
24
@@ -70,6 +71,38 @@ AK model of recursive growth.
70
71
The critical question is whether the returns to R&D effort are sufficiently steep. Some classic papers: Bloom et al. "are we running out of ideas"; and Erdil.
71
72
72
73
74
+
# Models
75
+
76
+
See also `posts/2026-02-10-model-of-labs.qmd`
77
+
78
+
Model A: Chinchilla. You just choose model size.
79
+
:
80
+
- One choice variable: model size (holding compute fixed, this trades-off against training runs).
81
+
- You can do some low-cost derisks, but extrapolating to a large compute scale is imperfect.
82
+
- *Convex but expensive to evaluate.*
83
+
84
+
Model B: algorithm search.
85
+
: - Each algorithm is a mapping from compute->loss, and you can summarize it with a scalar efficiency (just like nanoGPT).
86
+
- *medium-cost evaluation; non-convex.*
87
+
88
+
Model C: training data.
89
+
: - There's a distribution of tasks, with probability f(t).
90
+
- An LLM is just a lookup table, you slowly expand coverage, collecting a larger fraction of tasks t.
91
+
92
+
Model D: layer cake.
93
+
: - You have a function that turns money->intelligence, & that function gets more efficient over time, it's falling at about 8X/year.
94
+
95
+
- You can separate the optimization problem into layers, they are partially separable:
96
+
1. GPU design
97
+
2. Model architecture
98
+
3. Pretraining optimizer
99
+
4. Pretraining data
100
+
5. Posttraining algorithm (RL)
101
+
6. Posttraining data
102
+
7. Elicitation/harness
103
+
104
+
- The cost of an experiment is different for different parts. Some parts scale additively (GPU design), other parts its hard to predict effect of scale (architecture). If they scale additively then the cost of experimentation is small (but expenditure could still be large).
105
+
73
106
# Trainee:Trainer Curves
74
107
75
108
Q: what evals would be useful in learning about RSI? Suppose we can plot the following:
0 commit comments