fix: utf-8 reads, cost-only csv/jsonl rows, cross-file copilot dedup#7
Merged
Conversation
Every data-file open() relied on the locale encoding, so native Windows (cp1252) read transcripts and logs as mojibake and split non-ASCII cwds into separate projects. Pin encoding="utf-8" on every read of user data (the parallel-read helper feeding the JSONL backends, the csv/jsonl request logs, pi, openclaw, vscode, the availability probe, and the .git/wsl.conf/proc reads), keeping the existing errors= handling.
A row logging only cost_usd/credits (no token counts) was skipped by the empty-row check, yet _probe_records_cost still counted it -- the source claimed records_cost=True while showing $0. Ingest rows whose cost is positive even with zero tokens; genuinely empty rows (no tokens AND no cost) stay skipped.
The trace context, coverage sets, and seen keys were local to _parse_file, but OTEL exporters write spans and logs to different files: one call logged as a chat span in file A and an inference log in file B was counted twice. Collect every file's candidates first, then dedup and emit at _parse scope, preserving the chat span > inference log > agent-turn log > agent-summary span priority. The trace/span/response ids in the dedup keys are globally unique, so widening the scope never merges distinct calls.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three store correctness fixes (separate commits):
open()relied on the locale encoding, so native Windows (cp1252) read transcripts as mojibake and split non-ASCII cwds into separate projects.encoding="utf-8"is now pinned on every user-data read (_read_textfeeding the JSONL backends, csv/jsonl request logs, pi, openclaw, vscode, the availability probe), keeping existingerrors=handling.cost_usd/credits with no token counts was dropped by the empty-row check — yet_probe_records_costcounted it, so the source claimedrecords_cost=Truewhile showing $0. Rows with positive cost are now ingested; genuinely empty rows stay skipped._parse_file, but OTEL exporters write spans and logs to different files — one call logged as a chat span in file A and an inference log in file B counted twice. Candidates are now collected across all files, then deduped and emitted at_parsescope, preserving the existing record-shape priority.Tested: 3 new tests (cost-only csv row, cost-only jsonl line, span+log split across two files → one row); suite 307/307, ruff clean.