Merge pull request #455 from esokullu/main

docs: add Ai2 response to Molmo2 benchmark

Author: esokullu

web/blog/molmo2-8b-tool-calling/index.html

@@ -576,6 +576,9 @@ <h2 id="objective-assessment">Objective assessment</h2>
 <p>Molmo2-8B did not fail because the endpoint crashed in the text-call run. There were zero transport errors there, and median latency was about 1.2 seconds, which is perfectly usable. But the fairer native-tools run did fail at the server/template layer before inference. So we should not treat the 2/100 text-call result as Molmo2's true tool-calling ability.</p>
 <p>Compared with the existing WebBrain planner benchmark, this is a different bucket. In the earlier 100-case runs, viable planner models typically produced tool calls in most cases, but they also received native structured tools. On the frozen baseline text-call path, Molmo2 produced 2 parsed tool calls. Even giving it credit for malformed attempts only raises that to 9 recognizable attempts. That says the compatibility shim is not enough; it does not prove Molmo2 would fail with a proper native-tools serving stack.</p>
 <p>But the result should not be overread. This was a text-only first-tool-call routing test, not a Molmo2 vision or grounding benchmark. Molmo2's headline capability is VLM grounding; this harness did not ask it to point at pixels, track objects in video, or caption a screenshot. Also, because the local chat template forced text-call compatibility, Molmo2 did not get native structured tool schemas. The frozen baseline means the prompt/tool snapshot is controlled; it does <strong>not</strong> mean the benchmark is fair against models that got native tools. A constrained decoder, a model-specific chat template, native tool-schema support, or a small tool-format fine-tune could change this result materially.</p>
+<h2 id="ai2-s-official-response">Ai2's official response</h2>
+<p>The Allen Institute for AI shared this clarification:</p>
+<blockquote><p>We did not do any post-training for tool calling or add it in our SFT data -- it just wasn't the focus of the project. Using Molmo2 for tool calling would probably require some post-training on top of the released model. (We would like to do this in the future though.)</p></blockquote>
 <p>So the fair read is:</p>
 <ul><li><strong>As an open-source VLM release:</strong> excellent, and unusually important.</li><li><strong>As served through this LM Studio text-call path:</strong> not usable as a drop-in WebBrain planner.</li><li><strong>As a native structured-tool model:</strong> unmeasured here; the server rejected that path.</li><li><strong>As a candidate for model-specific integration work:</strong> yes, because the openness gives us something to work with.</li></ul>
 <h2 id="what-we-should-try-next">What we should try next</h2>

web/blog/posts/molmo2-8b-tool-calling.md

@@ -201,6 +201,12 @@ Compared with the existing WebBrain planner benchmark, this is a different bucke
 
 But the result should not be overread. This was a text-only first-tool-call routing test, not a Molmo2 vision or grounding benchmark. Molmo2's headline capability is VLM grounding; this harness did not ask it to point at pixels, track objects in video, or caption a screenshot. Also, because the local chat template forced text-call compatibility, Molmo2 did not get native structured tool schemas. The frozen baseline means the prompt/tool snapshot is controlled; it does **not** mean the benchmark is fair against models that got native tools. A constrained decoder, a model-specific chat template, native tool-schema support, or a small tool-format fine-tune could change this result materially.
 
+## Ai2's official response
+
+The Allen Institute for AI shared this clarification:
+
+> We did not do any post-training for tool calling or add it in our SFT data -- it just wasn't the focus of the project. Using Molmo2 for tool calling would probably require some post-training on top of the released model. (We would like to do this in the future though.)
+
 So the fair read is:
 
 - **As an open-source VLM release:** excellent, and unusually important.