Skip to content

Commit 9cca502

Browse files
WaylandYangclaude
andcommitted
paper: abstract + discussion now reflect empirical study
Two stale spots cleaned up after the hallucination-figure overhaul: 1. The abstract previously listed contributions as "design rationale, reference implementations, and conformance suite" -- omitting the empirical study entirely. It now ends with one sentence summarizing the headline numbers: 295 tool calls from DeepSeek V3 and Qwen 2.5-72B, DCP at 100% capability-escalation rejection and 50% prompt injection vs 0-5% for MCP-family protocols. 2. Section 6.1 ("What this paper does not prove") had a bullet point saying the LLM-safety adversarial-prompt study was still future work. It is no longer future work. The bullet is rewritten to honestly bound what the study DID cover (2 LLMs from 2 families, 295 calls, 6 attack categories) and what is still open (broader model and category coverage). Also added a sentence noting the excluded third LLM (Qwen 2.5-7B did not emit parseable tool calls). Pdf grows from 13 to 14 pages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent c7647d7 commit 9cca502

1 file changed

Lines changed: 31 additions & 21 deletions

File tree

docs/paper/main.tex

Lines changed: 31 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -91,10 +91,17 @@
9191
and post-hoc behavioral intrusion detection~\cite{aegismcp}: DCP prevents
9292
structurally-invalid calls; those approaches catch what slips through. We
9393
describe DCP's design rationale, MIT-licensed reference implementations
94-
(Python Bridge, ESP32 firmware), and a language-neutral conformance suite.
95-
We position DCP as the missing layer between MCP (which is moving toward
96-
enterprise SaaS connectivity~\cite{mcp_roadmap_2026}) and the physical
97-
devices it does not reach.
94+
(Python Bridge, ESP32 firmware), a language-neutral conformance suite,
95+
and an empirical evaluation in which 295~tool calls produced by two
96+
LLMs (DeepSeek~V3, Qwen2.5-72B-Instruct) in response to six categories
97+
of adversarial prompts are run through the host-side validators of
98+
DCP, Raw MCP, IoT-MCP, and a well-formed OpenAPI~3 schema; DCP
99+
rejects 100\,\% of capability-escalation attempts and 50\,\% of
100+
prompt-injection attempts versus 0--5\,\% for MCP-family protocols,
101+
matching OpenAPI's expressiveness at three orders of magnitude less
102+
firmware footprint. We position DCP as the missing layer between MCP
103+
(which is moving toward enterprise SaaS connectivity
104+
\cite{mcp_roadmap_2026}) and the physical devices it does not reach.
98105
\end{abstract}
99106

100107
\section{Introduction}
@@ -754,31 +761,34 @@ \section{Discussion and Limitations}\label{sec:discussion}
754761

755762
\subsection{What this paper does not prove}
756763

757-
This is a design and architecture paper with limited measurement. We have
758-
validated the reference implementation on one MCU (ESP32-WROOM-32) over
759-
one transport (UART), and reported its compiled footprint. We have not
760-
yet established:
764+
We have validated the reference implementation on one MCU (ESP32-WROOM-32)
765+
over one transport (UART), reported its compiled footprint, measured
766+
its end-to-end latency, and run an empirical adversarial-prompt study
767+
against two LLMs (Section~\ref{sec:safety}, Figure~\ref{fig:halluc}).
768+
We have \emph{not} established:
761769

762770
\begin{itemize}[leftmargin=*,nosep]
763771
\item Footprint and latency across the multi-MCU matrix that IoT-MCP
764-
covers (Cortex-M0+, nRF52840, ESP32-C3 etc.).
772+
covers (Cortex-M0+, nRF52840, ESP32-C3 etc.). The DCP firmware
773+
is portable Arduino C++, but only ESP32 is measured here.
765774
\item Comparative end-to-end latency vs IoT-MCP's reported
766775
$\sim$\,205\,ms (we have qualitative confirmation of similar
767776
order-of-magnitude but no formal A/B study).
768-
\item Quantitative LLM-safety improvement: the adversarial-prompt
769-
study that would empirically establish the Bridge's
770-
hallucination-rejection rate against baselines.
777+
\item LLM-safety results beyond two LLMs and six attack categories.
778+
Our corpus is 295~tool calls from DeepSeek~V3 and
779+
Qwen2.5-72B-Instruct (a third candidate, Qwen2.5-7B-Instruct,
780+
emitted no parseable OpenAI-style tool calls and was excluded).
781+
Broader coverage across model families and a larger,
782+
independently-sourced attack taxonomy is left to follow-up work.
771783
\end{itemize}
772784

773-
These belong in a follow-on paper after the hardware campaign. The present
774-
contribution is the architectural argument and the open-source
775-
implementation that makes it reproducible. For the empirical evaluation we
776-
plan to leverage the recently-released IoT-SkillsBench
777-
benchmark~\cite{iot_skillsbench}, which spans three representative
778-
platform--framework combinations (ATmega2560+Arduino, ESP32-S3+ESP-IDF,
779-
nRF52840+Zephyr) and 42 tasks across three difficulty levels. Reusing this
780-
benchmark lets future DCP results compare directly with prior LLM--IoT
781-
work~\cite{iotmcp2025} without re-litigating evaluation methodology.
785+
For the broader hardware and benchmark evaluation we plan to leverage
786+
the recently-released IoT-SkillsBench~\cite{iot_skillsbench}, which
787+
spans three representative platform--framework combinations
788+
(ATmega2560+Arduino, ESP32-S3+ESP-IDF, nRF52840+Zephyr) and 42 tasks
789+
across three difficulty levels. Reusing this benchmark lets future DCP
790+
results compare directly with prior LLM--IoT work~\cite{iotmcp2025}
791+
without re-litigating evaluation methodology.
782792

783793
\subsection{Known limitations of v0.x}
784794

0 commit comments

Comments
 (0)