Skip to content

Commit c396d0d

Browse files
committed
Evals: record decision to leave credential-hygiene as a known gap
Capable models route to the handoff correctly and the smuggled key still lands encrypted in the vault, so the residual harm is narrow. Surveyed approaches logged in the findings log so they don't get re-litigated; not fixing now.
1 parent ff7b2e9 commit c396d0d

1 file changed

Lines changed: 13 additions & 0 deletions

File tree

e2e/evals/EVALS.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,3 +113,16 @@ description tweaks made because of it):
113113
provider-item refs, so the model was smuggling the raw key INTO a ref
114114
field: `from: { provider: "encrypted", id: "Bearer re_…" }`). Description
115115
text can't fix a model that's determined to use what's in context.
116+
- **2026-06-12 · DECISION: not fixing credential-hygiene for now.** Rationale:
117+
capable models already route to the handoff correctly (connect-handoff
118+
passes), the descriptions already steer the right way, and the residual
119+
harm is narrow — the smuggled key still lands ENCRYPTED in the vault; the
120+
real leak is plaintext in our logs/traces + the provider's logs, which the
121+
user's paste already incurred. Approaches surveyed but deliberately NOT
122+
taken (so they don't get re-litigated): fail-helpfully redirect (reject raw
123+
secret → return the handoff URL in the error), field-agnostic secret-shape
124+
redaction in tool middleware, output/log scrubbing, ref-resolution
125+
validation, removing the tool from the agent surface. If this resurfaces,
126+
output/log scrubbing is the pure-upside piece and the cred-hygiene task can
127+
A/B any candidate (it already separates "secret in tool call" vs "secret in
128+
answer"). Tracking this as a known gap, not a bug to fix now.

0 commit comments

Comments
 (0)