robots

Author: tecunningham

_freeze/posts/2026-03-13-apple-picking-ai/execute-results/html.json

@@ -263,11 +263,7 @@ <h1 class="title">An Apple-Picking Model of Agents</h1>
 <dt>A simple model for AI R&amp;D.</dt>
 <dd>
 <p>An agent helping you to optimize an algorithm is like a robot helping you pick apples. It will take care of all the apples up to a certain height, and it may find apples you haven’t found yet, but there will still be apples out of its reach.</p>
-<p>The motivation for writing this model was to help think through the implications of recent evidence that AI can push forward the frontier on various optimization and AI R&amp;D problems (see my earlier post on <a href="https://tecunningham.github.io/posts/2026-01-29-knowledge-creating-llms.html">AI knowledge creation</a>). If you can spend $100 in tokens to increase the efficiency of an AI training algorithm by 0.1% then, on its surface, this looks like the path to self-improvement, and you can replace humans with AI. But realistically the agents have been discovering <em>shallow</em> improvements to algorithms. This apple-picking model is my attempt to help think through the distinction, and figure out how to measure agents’ optimization ability.</p>
 <p>Below I give a formal model but the basic ideas can all be seen with a drawing: here both the human and robot have picked four apples, but they’ve left the tree in a very different state, so the robot isn’t ready to replace the human yet:</p>
-</dd>
-</dl>
-<p>&nbsp;</p>
 <div class="cell" data-layout-align="center">
 <div class="cell-output-display">
 <div class="quarto-figure quarto-figure-center">
@@ -277,37 +273,40 @@ <h1 class="title">An Apple-Picking Model of Agents</h1>
 </div>
 </div>
 </div>
+<p>The motivation for writing this model was to help think through the implications of recent evidence that AI can push forward the frontier on various optimization and AI R&amp;D problems (see my earlier post on <a href="https://tecunningham.github.io/posts/2026-01-29-knowledge-creating-llms.html">AI knowledge creation</a>). If you can spend $100 in tokens to increase the efficiency of an AI training algorithm by 0.1% then, on its surface, this looks like the path to self-improvement, and you can replace humans with AI. But realistically the agents have been discovering <em>shallow</em> improvements to algorithms. This apple-picking model is my attempt to help think through the distinction, and figure out how to measure agents’ optimization ability.</p>
+</dd>
+</dl>
 <table class="caption-top table">
 <colgroup>
 <col style="width: 50%">
 <col style="width: 50%">
 </colgroup>
 <thead>
 <tr class="header">
-<th>Agents finding optimizations</th>
 <th>Robots picking apples</th>
+<th>Agents finding optimizations</th>
 </tr>
 </thead>
 <tbody>
 <tr class="odd">
-<td>Agents can autonomously advance the state-of-the-art on an optimization problem</td>
 <td>Robots can find low apples that humans have not picked yet</td>
+<td>Agents can autonomously advance the state-of-the-art on optimization problems</td>
 </tr>
 <tr class="even">
-<td>Agents will asymptote to a lower maximum value</td>
 <td>Robots will not be able to pick all the apples humans can</td>
+<td>Agents will asymptote to a lower maximum value</td>
 </tr>
 <tr class="odd">
-<td>Agents will have greater relative value for problems that are not yet heavily optimized</td>
 <td>Robots are more useful for trees that have never been picked</td>
+<td>Agents will have greater relative value for problems that are not yet heavily optimized</td>
 </tr>
 <tr class="even">
-<td>An algorithm will be more optimized if agents are started after some rounds of human optimization</td>
 <td>A tree will yield more apples if it’s harvested by both a human and robots</td>
+<td>An algorithm will be more optimized if agents are started after some rounds of human optimization</td>
 </tr>
 <tr class="odd">
-<td>To gauge the value of agents we want to test for the maximum <em>depth</em> of optimization they can do</td>
 <td>To gauge the ability of robots we want to measure the highest apple they can reach</td>
+<td>To gauge the value of agents we want to test for the maximum <em>depth</em> of optimization they can do</td>
 </tr>
 </tbody>
 </table>
@@ -318,8 +317,8 @@ <h1 class="title">An Apple-Picking Model of Agents</h1>
 </dd>
 <dt>Relation to other models of AI R&amp;D.</dt>
 <dd>
-<p>The critical distinction between this model and others is how we represent the state. Most existing models summarize the level of productivity (or stock of knowledge) with a single number, meaning there’s no distinction between a shallow and deep contribution. The model I’m using here allows us to represent the state with two numbers: the share of apples picked above and below <span class="math inline">\(\lambda\)</span>. For the RSI version of the model, we track <span class="math inline">\(\lambda\)</span> and the share of apples picked between <span class="math inline">\(\lambda\)</span> and 1.</p>
-<p>Most existing models of recurse self-improvement assume that either AI accelerates or replaces human R&amp;D researchers. I believe this is roughly true for <span class="citation" data-cites="aghion2019artificial">Aghion, Jones, and Jones (<a href="#ref-aghion2019artificial" role="doc-biblioref">2019</a>)</span>, <span class="citation" data-cites="davidson2021could">Davidson (<a href="#ref-davidson2021could" role="doc-biblioref">2021</a>)</span>, <span class="citation" data-cites="erdil2025gate">Erdil et al. (<a href="#ref-erdil2025gate" role="doc-biblioref">2025</a>)</span>, <span class="citation" data-cites="davidson2026automatingairesearch">Davidson et al. (<a href="#ref-davidson2026automatingairesearch" role="doc-biblioref">2026</a>)</span>, <span class="citation" data-cites="jones2025aird">Jones (<a href="#ref-jones2025aird" role="doc-biblioref">2025</a>)</span>, <span class="citation" data-cites="kwa2026simpleraitimelines">Kwa (<a href="#ref-kwa2026simpleraitimelines" role="doc-biblioref">2026</a>)</span>. These models then calibrate the effect through (1) how much does AI accelerate R&amp;D workers; (2) how much do R&amp;D workers contribute to our stock of knowledge.</p>
+<p>The critical distinction between this model and others is how we represent the state. Most existing models summarize the level of productivity (or stock of knowledge) with a single number, meaning there’s no distinction between a shallow and deep contribution. The model I’m using here allows us to represent the state with two numbers: the share of apples picked above <span class="math inline">\(\lambda\)</span>, and the share below <span class="math inline">\(\lambda\)</span>. For the RSI version of the model we instead track the current level of capabilities (<span class="math inline">\(\lambda_n\)</span>) and the share of apples picked above that threshold.</p>
+<p>Most existing models of recursive self-improvement assume that either AI accelerates or replaces human R&amp;D researchers. I believe this is roughly true for <span class="citation" data-cites="aghion2019artificial">Aghion, Jones, and Jones (<a href="#ref-aghion2019artificial" role="doc-biblioref">2019</a>)</span>, <span class="citation" data-cites="davidson2021could">Davidson (<a href="#ref-davidson2021could" role="doc-biblioref">2021</a>)</span>, <span class="citation" data-cites="erdil2025gate">Erdil et al. (<a href="#ref-erdil2025gate" role="doc-biblioref">2025</a>)</span>, <span class="citation" data-cites="davidson2026automatingairesearch">Davidson et al. (<a href="#ref-davidson2026automatingairesearch" role="doc-biblioref">2026</a>)</span>, <span class="citation" data-cites="jones2025aird">Jones (<a href="#ref-jones2025aird" role="doc-biblioref">2025</a>)</span>, <span class="citation" data-cites="kwa2026simpleraitimelines">Kwa (<a href="#ref-kwa2026simpleraitimelines" role="doc-biblioref">2026</a>)</span>. These models then calibrate the effect through (1) how much does AI accelerate R&amp;D workers; (2) how much do R&amp;D workers contribute to our stock of knowledge.</p>
 <p>However there is some awkwardness in fitting these models to the data:</p>
 <ol type="1">
 <li>It is hard to reconcile these models with evidence that AI is already autonomously contributing to AI research, yet still not replacing humans (i.e.&nbsp;autonomous agents are not perfect substitutes for humans).</li>
@@ -331,11 +330,11 @@ <h1 class="title">An Apple-Picking Model of Agents</h1>
 <dd>
 <ul>
 <li><p><em>Landscape.</em> A more general version would model the entire <em>landscape</em>. You can represent an optimization problem as <span class="math inline">\(y=f(\bm{x})\)</span>, where you’re trying to choose an <span class="math inline">\(\bm{x}\)</span> to maximize <span class="math inline">\(y\)</span>, given some unknown <span class="math inline">\(f(\cdot)\)</span>. (talk about non-additivity of optimizations; talk about conditions under which landscape is separable, and so each subspace is an independent apple; talk about path dependence).</p></li>
-<li><p><em>Relation to time horizon.</em> You can think of the high apples as long-time-horizon tasks.</p></li>
+<li><p><em>Relation to time horizon.</em> You could think of the high apples as long-time-horizon tasks, i.e.&nbsp;tasks that AI agents are relatively worse at performing.</p></li>
 <li><p><em>Shape of the tree.</em> You can extend the model such that apples are non-uniformly distributed, then we can replace <span class="math inline">\(\lambda\)</span> with <span class="math inline">\(F(\lambda)\)</span> below. We can then talk about types of domain which are bottom-heavy (most optimizations are pretty easy to find) vs top-heavy (most optimizations are hard to find). It then becomes important to know whether AI R&amp;D is relatively more bottom-heavy or top-heavy, if the former then we might already be on the brink of an intelligence explosion.</p></li>
-<li><p><em>Directed search.</em> We assume that the probability of finding an apple/optimization is independent of other apples already found.</p></li>
-<li><p><em>Other bottlenecks.</em> – e.g.&nbsp;compute bottlenecks, experiment bottlenecks.</p></li>
-<li><p><em>Sketch of a quantitative model of LLM training.</em> LLM training is a big stack of algorithms, which we’ve been optimizing at perhaps 10X/year. [add some speculation about which parts of the stack have low-hanging fruit]</p></li>
+<li><p><em>Directed search.</em> We assumed that the probability of finding an apple is independent of other apples already found. Realistically people have an ability to put direct their attention to finding new innovations. This implies lower diminishing returns to expenditure, and higher complementarity between agents and humans, I’m not sure whether it would change the qualitative conclusions of the model.</p></li>
+<li><p><em>Bottlenecks.</em> Some R&amp;D is bottlenecked not just by thinking (which agents can do), but also by running experiments.</p></li>
+<li><p><em>Sketch of a quantitative model of LLM training.</em> LLM training is a big stack of algorithms, which we’ve been optimizing at perhaps 10X/year. Would be useful to add some speculation about which parts of the stack have low-hanging fruit.</p></li>
 </ul>
 </dd>
 </dl>
@@ -346,11 +345,11 @@ <h1>Open Model</h1>
 <dd>
 <p>There is a continuum of apples spread uniformly on the real line.</p>
 <p>A human can find apples over <span class="math inline">\([0,1]\)</span>, but an agent can only find apples over <span class="math inline">\([0,\lambda]\)</span>, with <span class="math inline">\(\lambda &lt; 1\)</span> (at least for now).</p>
-<p>Humans find apples at rate <span class="math inline">\(r_H\)</span>, agents find apples at rate <span class="math inline">\(r_A\)</span>, and we use <span class="math inline">\(t_H\)</span> and <span class="math inline">\(t_A\)</span> to represent the time humans and agents spend searching (you can also interpret <span class="math inline">\(t_H\)</span> and <span class="math inline">\(t_A\)</span> as expenditure on the problem).</p>
+<p>Humans find apples at rate <span class="math inline">\(r_H\)</span>, agents find apples at rate <span class="math inline">\(r_A\)</span>, and we use <span class="math inline">\(t_H\)</span> and <span class="math inline">\(t_A\)</span> to represent the time humans and agents spend searching (you can also interpret <span class="math inline">\(t_H\)</span> and <span class="math inline">\(t_A\)</span> as monetary expenditure on the problem).</p>
 </dd>
 </dl>
 <p><strong>We can then derive apples found:</strong></p>
-<p><span class="math display">\[\text{share apples found}= \underbrace{\lambda(1-e^{-r_Ht_H-r_At_A})}_{\text{apples from bottom of tree}}+\underbrace{(1-\lambda)(1-e^{-r_Ht_H})}_{\text{apples from top of tree}}.\]</span></p>
+<p><span class="math display">\[\text{apples picked}= \underbrace{\lambda(1-e^{-r_Ht_H-r_At_A})}_{\text{apples from bottom of tree}}+\underbrace{(1-\lambda)(1-e^{-r_Ht_H})}_{\text{apples from top of tree}}.\]</span></p>
 <dl>
 <dt>Implication: agents asymptote to a lower level than humans.</dt>
 <dd>