Skip to content

Commit 9b26b8c

Browse files
WaylandYangclaude
andcommitted
paper: P0/P1/P2 reviewer-readiness polish (post-v0.3.1)
A read-through audit of the paper after the v0.3.1 changes surfaced several stale or weak passages. Fixed in three priority tiers. P0 (factual / self-contradictory): - Introduction's "we make no claim of empirical superiority...will be released upon paper acceptance" paragraph was directly contradicted by the §3 empirical study and §4 IoT-MCP latency A/B that v0.3.1 added. Rewrote it as a plain pointer to the public repo + PyPI package; the empirical content now lives in its own contribution bullet (item 5) and in the new "Empirical headlines" paragraph. - Removed a misplaced \cite{wot_td_2025} that had been copy-pasted into the wrong sentence. - Section 6 "deferred to v0.3 and beyond" -> "deferred to v0.4 and beyond" (v0.3.1 is on PyPI). P1 (impact / signal): - Added "Empirical headlines" paragraph at the end of §1 that pulls the two big numbers (100% capability rejection across 295 calls; 5 µs latency delta vs IoT-MCP wire) up front, instead of leaving them for §3 to surface. - Contributions list now has an explicit "empirical study" bullet with the headline numbers and method (DeepSeek V3 + Qwen 2.5-72B via SiliconFlow, six adversarial categories). P2 (related work / conclusion / citations): - §5.4 now cites AgentDojo, InjecAgent, ToolBench as the standard LLM agent / tool-use safety benchmarks, with a sentence on why this paper used its own corpus instead (device-control idioms the general benchmarks don't cover) and a note that folding them in is the obvious next step. - Conclusion rewritten in three paragraphs: design contribution, empirical contribution (the part we didn't expect to find), and the call for collaborators. The empirical paragraph names the 100% / 5 µs numbers again so the reader leaves with them. - refs.bib gains agentdojo, injecagent, toolbench entries. PDF grows from 14 to 15 pages. arXiv tarball rebuilt against this polish (same v0.3.1 release; tarball replaced in place). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent efbecd4 commit 9b26b8c

3 files changed

Lines changed: 104 additions & 29 deletions

File tree

docs/paper/dcp-arxiv-v0.3.1.tar.gz

1.67 KB
Binary file not shown.

docs/paper/main.tex

Lines changed: 80 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,20 @@ \section{Introduction}
153153
a reference implementation across host (Python) and device (C++ for ESP32),
154154
designed from the start for LLM callers.
155155

156+
\paragraph{Empirical headlines.} Two findings up front, both measured
157+
(\S\ref{sec:safety}, \S\ref{sec:impl}): (i) across 295 tool calls
158+
produced by DeepSeek\,V3 and Qwen2.5-72B-Instruct in response to six
159+
categories of adversarial prompts, DCP rejects 100\,\% of capability
160+
escalation attempts and 50\,\% of prompt injection attempts before any
161+
byte leaves the host --- versus 0--5\,\% for Raw MCP and IoT-MCP, and
162+
tied with a well-formed OpenAPI\,3 spec at $\sim$\,1/1000 the runtime
163+
footprint OpenAPI requires. (ii) An apples-to-apples wire latency A/B
164+
on identical ESP32-S3 hardware shows DCP's full schema validation
165+
(capability + range + dry-run + CBOR/COBS/CRC) and an IoT-MCP-style
166+
newline-JSON wire both complete a round-trip in $\sim$\,15.6\,ms
167+
median, within 5\,$\mu$s of each other. DCP's safety primitives carry
168+
no measurable latency cost.
169+
156170
\paragraph{Contributions.} This paper makes the following contributions:
157171

158172
\begin{enumerate}[leftmargin=*,nosep]
@@ -176,22 +190,29 @@ \section{Introduction}
176190
(e.g.\ in C, Rust, Go) can mechanically verify against. We argue
177191
this methodology is necessary for the protocol to survive
178192
multi-author implementation.
193+
\item An \emph{empirical study} (\S\ref{sec:safety}) in which 295 tool
194+
calls produced by two real LLMs (DeepSeek\,V3 and
195+
Qwen2.5-72B-Instruct via SiliconFlow) in response to adversarial
196+
prompts across six attack categories are run through each
197+
protocol's host-side validator. DCP rejects 100\,\% of capability
198+
escalation attempts and 50\,\% of prompt injection attempts, vs
199+
0--5\,\% for MCP/IoT-MCP. The study also produced an end-to-end
200+
latency A/B against an IoT-MCP-shaped firmware on the same
201+
ESP32-S3 hardware: 15.60\,ms vs 15.59\,ms median, within 5\,$\mu$s.
202+
DCP's safety primitives are free at this scale.
179203
\item A discussion (\S\ref{sec:related}) positioning DCP against MCP,
180204
IoT-MCP, W3C Web of Things~\cite{wot_td_2025},
181205
Matter~\cite{matter_spec}, Sparkplug~B~\cite{sparkplug}, and direct
182-
OpenAPI exposure. We argue DCP occupies a specific gap: \emph{LLM-native
183-
safety primitives at MCU scale}, which none of the alternatives
184-
addresses.
206+
OpenAPI exposure. We argue DCP occupies a specific gap:
207+
\emph{LLM-native safety primitives at MCU scale}, which none of
208+
the alternatives addresses.
185209
\end{enumerate}
186210

187-
We make no claim of empirical superiority on benchmarks beyond what
188-
IoT-MCP has already reported, because we have not yet completed the
189-
hardware measurement campaign that would justify such a claim. We position
190-
this paper as a \emph{design contribution and architectural argument},
191-
with a complete open-source implementation~\cite{wot_td_2025}\footnote{
192-
The DCP reference implementation is MIT-licensed and will be released at
193-
\texttt{github.com/device-context-protocol} upon paper acceptance.}
194-
available for inspection and reproduction.
211+
The reference implementation is MIT-licensed and publicly available at
212+
\url{https://github.com/device-context-protocol/dcp}; the Python Bridge
213+
ships on PyPI as \texttt{pydcp}; the corpus, validators, and bench
214+
harnesses behind every figure in this paper are in the same repo and
215+
re-runnable end-to-end.
195216

196217
\section{Background: The LLM--Hardware Gap}
197218

@@ -752,12 +773,25 @@ \subsection{Safety mechanisms for LLM-driven physical systems}
752773
temporal graphs and detects intrusions sub-second on edge hardware. Both
753774
operate on already-issued, syntactically valid calls.
754775

755-
DCP is complementary to both. The Bridge's structural checks (capability,
756-
range, type, dry-run) reject calls before they become an attack chain;
757-
behavioral detectors like AegisMCP catch what the structural checks let
758-
through; semantic guardrails like RoboSafe catch what the behavioral
759-
detectors miss. A production deployment of LLM-driven hardware benefits
760-
from all three layers.
776+
A third strand --- the evaluation literature --- builds benchmarks for
777+
LLM tool-use safety in the abstract.
778+
AgentDojo~\cite{agentdojo} provides a dynamic environment to evaluate
779+
prompt-injection attacks and defenses on LLM agents.
780+
InjecAgent~\cite{injecagent} catalogues indirect-injection attack
781+
patterns specifically in tool-integrated agents. ToolBench~\cite{toolbench}
782+
covers 16,000+ APIs but is utility-oriented rather than adversarial.
783+
Our empirical study (\S\ref{sec:safety}) uses a smaller, hand-crafted
784+
corpus aimed specifically at the device-control idioms DCP cares about
785+
(numeric ranges with units, named static intents, capability-scoped
786+
operations); folding in AgentDojo-style benchmarks for a broader,
787+
independently-curated attack taxonomy is the obvious next step.
788+
789+
DCP is complementary to all of these. The Bridge's structural checks
790+
(capability, range, type, dry-run) reject calls before they become an
791+
attack chain; behavioral detectors like AegisMCP catch what the
792+
structural checks let through; semantic guardrails like RoboSafe catch
793+
what the behavioral detectors miss. A production deployment of
794+
LLM-driven hardware benefits from all three layers.
761795

762796
\subsection{Parallel: CBOR-Web}
763797

@@ -816,7 +850,7 @@ \subsection{Known limitations of v0.x}
816850
transactions. The former is correctly the responsibility of underlying
817851
layers (Thread, Zigbee); the latter is a real gap for use cases like
818852
``turn off every light in the house and lock the front door as a single
819-
operation.'' Both are deferred to v0.3 and beyond.
853+
operation.'' Both are deferred to v0.4 and beyond.
820854

821855
Capability tokens are coarse: a token bears a set of capabilities but does
822856
not bind to specific intents, parameters, or rates. Fine-grained rate
@@ -841,17 +875,34 @@ \subsection{Open questions}
841875

842876
\section{Conclusion}
843877

844-
DCP fills a specific gap in the LLM--device stack: a compact, safety-first
845-
wire format and architecture that scales down to commodity microcontrollers
846-
while preserving the LLM-aware primitives that make agent-driven control
847-
tractable in the presence of hallucination. We have presented the design
848-
rationale, a complete open-source reference implementation, and a
849-
conformance methodology. The MCP roadmap's deliberate move toward
850-
enterprise SaaS connectivity creates the design space DCP occupies; we
851-
expect that the demand for safe LLM control of physical hardware will only
852-
grow as agent frameworks mature. We invite collaborators, particularly
853-
those with measurement infrastructure for embedded benchmarks, to
854-
contribute to the v1.0 release.
878+
DCP fills a specific gap in the LLM--device stack: a compact,
879+
safety-first wire format and architecture that scales down to commodity
880+
microcontrollers while preserving the LLM-aware primitives that make
881+
agent-driven control tractable in the presence of hallucination. The
882+
design contribution is intent-level commands plus units, ranges,
883+
capabilities, and dry-run as protocol primitives rather than
884+
application-level discipline; the implementation contribution is a
885+
27.6\,KB-flash, 0.6\,KB-RAM device-side runtime that fits the long tail
886+
of MCUs MCP cannot reach.
887+
888+
The empirical contribution is the part we did not expect to surface.
889+
Across 295 real LLM-emitted tool calls, DCP's structural checks
890+
\emph{catch every capability-escalation attempt} (100\,\% vs 0\,\% for
891+
the only protocols that target comparable hardware) and add no
892+
measurable latency cost (within 5\,$\mu$s of a bare JSON wire on the
893+
same chip). The same study drove the v0.3.1 addition of string
894+
\texttt{pattern} and \texttt{max\_length} fields, closing the
895+
prompt-injection gap against OpenAPI. The lesson generalizes: an
896+
empirical adversarial-prompt study against real LLMs is the cheapest
897+
way we have found to discover what a protocol's safety spec actually
898+
needs.
899+
900+
The MCP roadmap's deliberate move toward enterprise SaaS connectivity
901+
creates the design space DCP occupies; the demand for safe LLM control
902+
of physical hardware will only grow as agent frameworks mature. We
903+
invite collaborators --- particularly those with multi-MCU measurement
904+
infrastructure and broader LLM API access --- to contribute to the v1.0
905+
release.
855906

856907
\bibliographystyle{plain}
857908
\bibliography{refs}

docs/paper/refs.bib

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -180,3 +180,27 @@ @article{llm_iot_survey
180180
year = {2025},
181181
url = {https://arxiv.org/abs/2505.17586}
182182
}
183+
184+
@inproceedings{agentdojo,
185+
title = {{AgentDojo}: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for {LLM} Agents},
186+
author = {Debenedetti, Edoardo and Zhang, Jie and Balunovi{\'c}, Mislav and Beurer-Kellner, Luca and Fischer, Marc and Tram{\`e}r, Florian},
187+
booktitle = {NeurIPS Datasets and Benchmarks Track},
188+
year = {2024},
189+
url = {https://arxiv.org/abs/2406.13352}
190+
}
191+
192+
@article{injecagent,
193+
title = {{InjecAgent}: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents},
194+
author = {Zhan, Qiusi and Liang, Zhixiang and Ying, Zifan and Kang, Daniel},
195+
journal = {ACL Findings},
196+
year = {2024},
197+
url = {https://arxiv.org/abs/2403.02691}
198+
}
199+
200+
@inproceedings{toolbench,
201+
title = {{ToolLLM}: Facilitating Large Language Models to Master 16000+ Real-world {APIs}},
202+
author = {Qin, Yujia and Liang, Shihao and Ye, Yining and Zhu, Kunlun and Yan, Lan and Lu, Yaxi and Lin, Yankai and Cong, Xin and Tang, Xiangru and Qian, Bill and Zhao, Sihan and Hong, Lauren and Tian, Runchu and Xie, Ruobing and Zhou, Jie and Gerstein, Mark and Li, Dahai and Liu, Zhiyuan and Sun, Maosong},
203+
booktitle = {ICLR},
204+
year = {2024},
205+
url = {https://arxiv.org/abs/2307.16789}
206+
}

0 commit comments

Comments
 (0)