Fix interception proxy capturing 0 token ids on current vLLM

[sergiopaniego]

What does this PR do?

The interception proxy (opencode_env/sandbox/interception.py) captures per-token ids and logprobs from the upstream vLLM response so GRPO can train on the exact tokens the policy emitted. On current vLLM this capture returned zero token ids.

vLLM with --return-tokens-as-token-ids encodes the id in the logprobs token field as "token_id:<int>" rather than a separate token_id key, which is what the capture read. So completion_token_ids came back empty while per_token_logps were fine, and every env that trains on the captured trace (opencode, pi, and the in-progress Claude Code env) loses the token ids GRPO needs.

The fix parses the id from the token field.

Validation

• New unit test tests/envs/test_interception_capture.py covers the vLLM prefixed-token format and the plain-OpenAI (no ids) case.
• Observed live in an AsyncGRPO loop-owning run: with the fix the trace captures real token ids (e.g. [40, 3278, 1855, ...]) end to end, without it the trace had 0.

Note

Independent of the sandbox-to-core move (#1022). When that lands, the same fix needs to travel with the moved interception.py.

AI-assisted.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Optional hardening (non-blocking). If an upstream build ever emits a malformed prefix — e.g. "token_id:" (empty suffix) or a non-numeric suffix — the int(token_id) call two lines below will raise ValueError and abort building the turn record. vLLM's --return-tokens-as-token-ids always produces a valid integer here, and this matches the existing behavior for the separate token_id key, so it's low priority. A small guard (e.g. wrap the int(...) in try/except ValueError, or check the suffix .isdigit()) would make capture resilient to unexpected upstream output rather than dropping the whole turn.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit d6227f3. Configure here.}

[cursor]

Legacy token_id key support dropped

Medium Severity

_build_turn_record only parses ids from a token value with the token_id: prefix and no longer reads a separate token_id field. Builds that still emit that key end up with empty completion_token_ids, which breaks GRPO training on those traces. The PR claims both formats are supported, but the legacy path and its test are missing.

Additional Locations (1)

• tests/envs/test_interception_capture.py#L30-L51

 

^{Reviewed by Cursor Bugbot for commit d6227f3. Configure here.}

[sergiopaniego]

Closing this after investigating whether the empty completion_token_ids ever actually broke training. It does not.

The TRL rollout worker's _trace_output_ids already recovers the ids from completion_tokens when completion_token_ids is empty, parsing the "token_id:<int>" strings vLLM writes into the token field. So the only consumer of the trace already handles this, and populating the field in the proxy would just duplicate that parse in a second place.

For the record, vLLM's _get_decoded_token returns f"token_id:{token_id}" in the token field (not a separate token_id key) across every version checked (v0.6.3, v0.6.6, v0.8.5, v0.9.2, main), which is why the original entry.get("token_id") read nothing. But since the downstream fallback covers it, this is cosmetic rather than a training bug, so leaving the shared proxy untouched.