autosave: 2026-02-07 11:28:01

Author: tecunningham

posts/2026-01-29-knowledge-creating-llms.qmd

@@ -100,7 +100,7 @@ Knowledge-creating LLMs.
 
     Knowledge-creating LLMs will differ from knowledge-sharing LLMs in a number of ways:
 
-    - Knowledge-creating LLMs will have qualitatively different benchmarks: instead of seeing if they can answer questions which we already know the answer to (most existing benchmarks), we want them to answer *new* questions, e.g. solve an unsolved mathematical problem ([FrontierMath Open Problems](https://epoch.ai/frontiermath/open-problems)) or set a new record on an optimization problem (e.g. GSO-bench, @shetty2025gso). We can use these new frontier benchmarks are indices for capability, but they are more challenging to interpret because the frontier is always moving.
+    - Knowledge-creating LLMs will have qualitatively different benchmarks: instead of seeing if they can answer questions which we already know the answer to (most existing benchmarks), we want them to answer *new* questions, e.g. solve an unsolved mathematical problem ([FrontierMath Open Problems](https://epoch.ai/frontiermath/open-problems)) or set a new record on an optimization problem (e.g. GSO-bench, @shetty2025gso). We can use these new frontier benchmarks as indices for capability, but they are more challenging to interpret because the frontier is always moving.
 
     - Knowledge-creating LLMs have high returns to compute on individual problems, unlike knowledge-sharing LLMs for which returns asymptote quickly. It can be worth spending billions of tokens to solve a single problem if the solution is generally applicable.