Skip to content

Commit bfebfff

Browse files
WaylandYangclaude
andcommitted
paper: empirical corpus expanded to 5 LLMs / 4 vendors (660 calls)
Previously the empirical study had 295 parseable tool calls from two LLMs of the same vendor family (DeepSeek + Qwen). Reviewer pushback that the 'tie OpenAPI' claim rested on a small sample across one vendor family is fair. This commit grows the corpus to 660 calls across 5 LLMs from 4 vendors: DeepSeek/DeepSeek-V3 143 parsed Alibaba/Qwen-2.5-72B-Instruct 162 parsed Alibaba/Qwen-3.5-35B-A3B (multimodal MoE) 121 parsed Zhipu/GLM-4-32B-0414 86 parsed MiniMax/MiniMax-M2.5 148 parsed ------------------------------------------------------ Total 660 parsed (of 900) Aggregate measured rejection rates (vs the prior 2-LLM run): Out Unit Type Unk Cap PI DCP prior 2-LLM 9% 7% 0% 0% 100% 50% DCP now 5-LLM 11% 6% 4% 26% 100% 48% IoT-MCP prior 0% 0% 6% 0% 0% 5% IoT-MCP now 0% 2% 5% 11% 0% 6% Raw MCP now 0% 2% 5% 11% 0% 6% OpenAPI now 11% 6% 5% 26% 100% 48% n per category 114 119 115 38 138 136 Findings that replicate sharply across all 5 LLMs and 4 vendors: - Capability escalation: 100% DCP/OpenAPI vs 0% MCP family. Across 138 'reboot the lamp' tool calls the LLMs emitted, every single one is rejected by DCP's capability check and OpenAPI's OAuth2 scope check; MCP/IoT-MCP have no such concept and let all 138 through. - Prompt injection: 48% DCP/OpenAPI vs 6% MCP/IoT-MCP. New finding the smaller sample didn't surface: - Unknown intent: DCP/OpenAPI 26%, MCP-family 11%. With more LLMs in the mix we see more variation in how LLMs handle 'reset the wifi password' style prompts; a richer fraction actually emit unknown tool names and get caught by static intent-table lookup or additionalProperties:false. Operational fixes that made the bigger run reliable: - gen_llm_corpus.py: bumped per-call sleep 0.2 -> 0.6 s. - 20-second cool-down between models, so a rate-limit blip on one model doesn't poison the next model's window. - The previous run lost MiniMax entirely (0/180 parsed) to a SiliconFlow balance exhaustion mid-run; that's now refilled and the model lands cleanly. Paper updates: - Abstract, intro 'Empirical headlines' paragraph, contributions list item, §3 prose, figure caption, conclusion, and §6.1 bullet all updated to '5 LLMs across 4 vendors / 660 calls' with the new headline numbers. - Tarball rebuilt (dcp-arxiv-v0.3.1.tar.gz, 163 KB). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 9b26b8c commit bfebfff

7 files changed

Lines changed: 10408 additions & 2939 deletions

File tree

docs/paper/dcp-arxiv-v0.3.1.tar.gz

1 KB
Binary file not shown.
1.3 KB
Binary file not shown.
9.64 KB
Loading

0 commit comments

Comments
 (0)