Skip to content

[codex] Add RL trajectory capture#83

Open
Lixtt wants to merge 6 commits into
AI45Lab:mainfrom
Lixtt:codex/rl-trajectory-capture
Open

[codex] Add RL trajectory capture#83
Lixtt wants to merge 6 commits into
AI45Lab:mainfrom
Lixtt:codex/rl-trajectory-capture

Conversation

@Lixtt

@Lixtt Lixtt commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds opt-in RL trajectory capture to a3s-code so agent runs can emit structured JSONL records for downstream RL/SFT data construction. The latest update also captures token-level logprobs and full tool schemas so saved traces are usable as training examples without relying on the OpenClaw-RL wrapper to reconstruct missing fields.

Changes

  • add RlTrajectoryRecorder and a3s.rl_trajectory.v1 JSONL event schema
  • record execution lifecycle, LLM request/response, context compaction, tool calls, tool results, and parse/duplicate-call errors
  • preserve available_tools as tool-name compatibility data and add full tool_definitions to LLM request events
  • add TokenLogProb / TopTokenLogProb to LlmResponse and write token_logprobs to trajectory response events
  • support opt-in OpenAI-compatible logprobs / top_logprobs for non-streaming and streaming responses
  • expose trajectory and logprob controls through Rust, Python SDK, and Node SDK session options
  • refresh stale Rust lockfiles to match the current local manifests

Validation

  • cargo fmt
  • git diff --check
  • cargo test -p a3s-code-core openai
  • cargo test -p a3s-code-core rl_trajectory
  • cargo test --manifest-path sdk/python/Cargo.toml session_options_map
  • cargo test --manifest-path sdk/node/Cargo.toml session_options_maps

Notes

The logprob request remains off by default because some public providers reject logprob requests when tools are present. Training and evaluation harnesses can enable it explicitly via session options or A3S_CODE_LLM_LOGPROBS / A3S_CODE_LLM_TOP_LOGPROBS.

@Lixtt
Lixtt marked this pull request as ready for review July 6, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant