Dev-time inference for agents + real-inference eval tier#972
Closed
RhysSullivan wants to merge 4 commits into
Closed
Dev-time inference for agents + real-inference eval tier#972RhysSullivan wants to merge 4 commits into
RhysSullivan wants to merge 4 commits into
Conversation
…ing? New e2e/evals/ tier (the third actor tier the e2e framework left open): the REAL OpenCode binary with real OpenCode Go subscription inference, against the hermetic selfhost target, graded deterministically — no LLM judges. EVAL=1-gated vitest project, never on the PR path. - harness.ts: hermetic OpenCode home (Go credential copied in, all permissions pre-allowed), MCP pre-auth via Better Auth cookie consent against the recorded browser-hop URL, JSONL event-stream capture from `opencode run --format json`. - tasks.ts: two seed tasks. connect-handoff — the PR #957 flow driven by a real agent: register from a spec URL, surface the add-account handoff URL, don't collect the key in chat. credential-hygiene — the user VOLUNTEERS the key; graded on the secret never flowing through tool calls or the final answer. - evals.test.ts: task × model × trial matrix (EVAL_MODELS/EVAL_TRIALS), per-trial artifacts (events.jsonl + grade.json) under runs/evals/, pass-rate report.{md,json}; the only hard gate is pass-at-least-once per task×model. - EVALS.md: philosophy, Go model/quota table, findings log. First findings (logged in EVALS.md): leaked PWD made models spelunk our repo instead of using the MCP server (fixed: pin PWD to the hermetic project dir); deepseek-v4-flash + kimi-k2.5 PASS connect-handoff; minimax-m2.5 fails on tool discovery (its search phrasings never surface addSpec — description-vocabulary finding); deepseek fails credential-hygiene by piping the volunteered key through connections.create (description-strength finding).
Renamed connections.create to connections.dangerousCreateWithPlainTextSecret (DANGEROUS-prefixed description) and re-ran credential-hygiene 3v3 against the baseline name on deepseek-v4-flash. Secret-through-tool-call went from 2/3 to 1/3 — a nudge, not a guarantee — and one scary-name trial leaked the key into the final answer instead. Also: the model smuggles the raw key into the provider-ref field, so the refs-only schema doesn't contain it either. Logged in EVALS.md; the rename itself is reverted — naming can't carry the invariant.
Capable models route to the handoff correctly and the smuggled key still lands encrypted in the vault, so the residual harm is narrow. Surveyed approaches logged in the findings log so they don't get re-litigated; not fixing now.
The eval harness's core capability — drive the real OpenCode binary with
the machine's subscription, hermetically, and read structured output — is
useful on its own, so pull it out of the eval-specific harness.
- e2e/src/clients/inference.ts: runInference({ model, prompt, mcp? }) →
{ answerText, events, toolNames, ... }. Hermetic OpenCode home with the
subscription credential copied in; MCP optional with a pluggable consent
strategy. No effect on the developer's own OpenCode state.
- e2e/scripts/infer.ts (+ `bun run infer`): the CLI an agent runs while
developing — `bun e2e/scripts/infer.ts -m opencode/glm-5.1 "..."`.
- evals/harness.ts is now a thin consumer: runTrial = runInference + the
selfhost cookie-consent strategy; grading vocabulary unchanged.
Also: drop the connect-handoff 'baseUrl resolved' check — running against
current main surfaced that integration-level baseUrl is override-only now
(per-operation baseUrl from the #968-970 storage refactors), so the check
asserted an obsolete field. Documented in EVALS.md. Switch the eval test
import to @effect/vitest per the no-vitest-import rule.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-marketing | 0295eec | Commit Preview URL Branch Preview URL |
Jun 12 2026, 06:56 AM |
Contributor
Cloudflare preview
Sign-in is Cloudflare Access (one-time PIN to an allowed email). The preview has its own database and encryption key; it is destroyed when this PR closes. |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-cloud | 0295eec | Jun 12 2026, 06:57 AM |
@executor-js/cli
@executor-js/config
@executor-js/execution
@executor-js/sdk
@executor-js/codemode-core
@executor-js/runtime-quickjs
@executor-js/plugin-file-secrets
@executor-js/plugin-graphql
@executor-js/plugin-keychain
@executor-js/plugin-mcp
@executor-js/plugin-onepassword
@executor-js/plugin-openapi
executor
commit: |
RhysSullivan
added a commit
that referenced
this pull request
Jun 12, 2026
bun run cli infer [-m model] [--tools] [--json] "prompt" — ask a real model a question (or let it use coding tools in a scratch dir) through an actual agent harness rather than a bespoke loop. pi (@earendil-works/pi-coding-agent) runs headless (--mode json -p, clean typed JSONL events, --no-tools by default) with a throwaway config dir and no session, so nothing touches anyone's pi state. The model is reached through the machine's OpenCode Zen subscription: the gateway is OpenAI-compatible, so a generated pi provider extension points at it with the key from opencode's auth.json via env — no new credentials. runAgent() in e2e/src/clients/agent.ts is the programmatic primitive (answerText, thinkingText, toolCalls, usage, full event stream) for future eval harnesses. The OpenCode client (src/clients/opencode.ts) keeps its role as the MCP-native real-client actor — pi is the inference workhorse, OpenCode is the client-behavior probe. Supersedes the inference half of PR #972's approach (the OpenCode binary as a one-shot inference vehicle): same concept, but through a harness whose headless mode is built for it — no auth.json copying, no open(1) shim, no PWD-leak workarounds, ~2s per call. Verified live: plain answers on deepseek-v4-flash + glm-5.1, a tool-using run (write + read in the scratch dir), the missing-binary and missing-credential error paths, repo lint/typecheck/format.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Two things, stacked: a reusable way for agents (and tests) to run real inference while developing in this repo, and the real-inference eval tier built on top of it. Opening as draft — the inference primitive is the part worth landing; the evals are along for the ride and run on-demand only.
The headline: dev-time inference, committed
e2e/src/clients/inference.ts+ a CLI ate2e/scripts/infer.tsdrive the real OpenCode binary with the machine's OpenCode subscription, in a throwaway home, so an agent working in the repo can ask a real model a question without touching the developer's own OpenCode state/history.Programmatic:
runInference({ model, prompt, mcp? }) → { answerText, events, toolNames, ... }. Passmcpto expose one of our MCP servers to the model (with a pluggable consent strategy); omit it for plain question→answer. Verified end-to-end (~2.7s for a one-word reply on the cheap model).The eval tier (on-demand, never on the PR path)
EVAL=1 npm run test:evals— the real OpenCode binary against the hermetic selfhost target, real subscription inference, deterministic grading (workspace state via the typed API + the emulator's request ledger + transcript content; no LLM judges). Task × model × trial matrix, artifacts + pass-rate report underruns/evals/. Two seed tasks:connect-handoffandcredential-hygiene. Philosophy, the model/quota table, and a findings log live ine2e/evals/EVALS.md.Findings the eval already produced (in EVALS.md)
PWDmade models spelunk the repo instead of using MCP — fixed by pinning the hermetic project dir.minimax-m2.5can't discoveraddSpecvia "add connection"-flavored search queries (tool-description vocabulary gap);deepseek+kimifind it.deepseekpipes a volunteered API key straight throughconnections.create(credential-hygiene). Ran a rename/"scary name" experiment (3v3): it nudges but doesn't hold. Decided not to fix now — capable models route correctly and the key still lands encrypted; logged the surveyed approaches so they don't get re-litigated.mainsurfaced that integration-levelbaseUrlis now override-only (per-operation baseUrl from the Move inline specs to the blob table at boot on the libSQL hosts #968–970 storage refactors) — the eval's old check asserted an obsolete field; removed and documented.Not included / out of scope
No product code changes.
connections.createis untouched (the rename experiment was reverted). The runs-viewer integration forruns/evals/and a nightly workflow are natural follow-ups.