- "markdown": "---\ntitle: AI cost curves\ndraft: true\nengine: knitr\nexecute:\n echo: false\n warning: false\n error: false\n cache: true # caches chunk output\n---\n\n\n\nIt's useful to draw plots showing achievement vs expenditure, comparing humans & agents.\n\nYou can read the y-axis in a few ways: (1) score on a benchmark; (2) quality of the output; (3) score on an optimization problem.\n\nObservations:\n\n1. In most cases agents are cheaper than humans but hit a ceiling in capability.\n2. Can simplify to say agents are free, without much loss.\n3. We can see three types of agent growth: (A) cheaper inference; (B) expanded capabilities; (C) test-time growth. \n4. Distillation shifts cost curves left.\n5. This observation is a nice fit for time horizon (more discussion required)\n6. Q: can you derive these curves from a theory of task complexity?\n\nTODO:\n\n1. Probably plot ln(expenditure).\n2. Show separate graphs for inference cost & training cost.\n3. Observation: for many things it doesn't matter if we don't have cardinal scale of quality, as long as we have *ordinal* scale. E.g. we can still talk about cost reduction, & something about scale effects.\n\nBasic plot:\n\n\n\n\n::: {.cell}\n::: {.cell-output-display}\n{width=672}\n:::\n:::\n\n\n\n\nThree types of agent change:\n\n\n\n::: {.cell}\n::: {.cell-output-display}\n{width=672}\n:::\n:::\n\n\n\n# additional plots\n\nExtra plots:\n\n\n\n\n::: {.cell}\n::: {.cell-output-display}\n{width=672}\n:::\n:::\n",
0 commit comments