Commit bfebfff
paper: empirical corpus expanded to 5 LLMs / 4 vendors (660 calls)
Previously the empirical study had 295 parseable tool calls from
two LLMs of the same vendor family (DeepSeek + Qwen). Reviewer
pushback that the 'tie OpenAPI' claim rested on a small sample
across one vendor family is fair. This commit grows the corpus
to 660 calls across 5 LLMs from 4 vendors:
DeepSeek/DeepSeek-V3 143 parsed
Alibaba/Qwen-2.5-72B-Instruct 162 parsed
Alibaba/Qwen-3.5-35B-A3B (multimodal MoE) 121 parsed
Zhipu/GLM-4-32B-0414 86 parsed
MiniMax/MiniMax-M2.5 148 parsed
------------------------------------------------------
Total 660 parsed (of 900)
Aggregate measured rejection rates (vs the prior 2-LLM run):
Out Unit Type Unk Cap PI
DCP prior 2-LLM 9% 7% 0% 0% 100% 50%
DCP now 5-LLM 11% 6% 4% 26% 100% 48%
IoT-MCP prior 0% 0% 6% 0% 0% 5%
IoT-MCP now 0% 2% 5% 11% 0% 6%
Raw MCP now 0% 2% 5% 11% 0% 6%
OpenAPI now 11% 6% 5% 26% 100% 48%
n per category 114 119 115 38 138 136
Findings that replicate sharply across all 5 LLMs and 4 vendors:
- Capability escalation: 100% DCP/OpenAPI vs 0% MCP family.
Across 138 'reboot the lamp' tool calls the LLMs emitted,
every single one is rejected by DCP's capability check and
OpenAPI's OAuth2 scope check; MCP/IoT-MCP have no such
concept and let all 138 through.
- Prompt injection: 48% DCP/OpenAPI vs 6% MCP/IoT-MCP.
New finding the smaller sample didn't surface:
- Unknown intent: DCP/OpenAPI 26%, MCP-family 11%. With more
LLMs in the mix we see more variation in how LLMs handle
'reset the wifi password' style prompts; a richer fraction
actually emit unknown tool names and get caught by static
intent-table lookup or additionalProperties:false.
Operational fixes that made the bigger run reliable:
- gen_llm_corpus.py: bumped per-call sleep 0.2 -> 0.6 s.
- 20-second cool-down between models, so a rate-limit blip on
one model doesn't poison the next model's window.
- The previous run lost MiniMax entirely (0/180 parsed) to a
SiliconFlow balance exhaustion mid-run; that's now refilled
and the model lands cleanly.
Paper updates:
- Abstract, intro 'Empirical headlines' paragraph, contributions
list item, §3 prose, figure caption, conclusion, and §6.1
bullet all updated to '5 LLMs across 4 vendors / 660 calls'
with the new headline numbers.
- Tarball rebuilt (dcp-arxiv-v0.3.1.tar.gz, 163 KB).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 9b26b8c commit bfebfff
7 files changed
Lines changed: 10408 additions & 2939 deletions
File tree
- docs/paper
- figures
- tools
Binary file not shown.
Binary file not shown.
Loading
0 commit comments