fix: sanitize LLM-derived text before logging, JSON serialization, and database writes

[GrigoryEvko]

LLM responses and compiler stderr from heterogeneous toolchains (Python tracebacks, Triton MLIR diagnostics, nvcc / ptxas / CUTLASS template explosions, Mojo error formatter output, Pallas / JAX jaxpr traces, CuTe layout errors) flow through gigaevo into loguru sinks, orjson serialization for Redis Streams and the program archive, asyncpg TEXT columns, langfuse trace metadata, file paths, and back into subsequent LLM prompts as part of multi-agent loops. The text was previously passed through verbatim. A single lone UTF-16 surrogate anywhere in a stage error raises UnicodeEncodeError from inside orjson, aborts the Program write to Redis, and stalls the evolution loop. A literal NUL byte in a traceback is rejected by asyncpg with A string literal cannot contain NUL (0x00) characters and aborts the tracker write. ANSI escape sequences from nvcc and clang colorization survive into loguru file sinks and corrupt log readers; CR survives and lets a multi-line LLM justification forge log entries that downstream parsers cannot distinguish from authentic records; BIDI overrides survive and hide content from operator log review.

A minimal reproducer for the orjson crash path, runnable from the repo root before this change:

import openai, httpx, orjson
req = httpx.Request("POST", "https://api.openai.com/v1/chat/completions")
resp = httpx.Response(429, request=req)
err = openai.RateLimitError("rate limited \ud83d", response=resp, body=None)
data = {"error": str(err)}
orjson.dumps(data)
# orjson.JSONEncodeError: surrogates not allowed

The change introduces gigaevo/utils/text_sanitize.py with five pure str -> str functions: sanitize_for_log strips ANSI / BIDI overrides / lone surrogates and escapes C0 and C1 control characters except TAB and LF; sanitize_for_json replaces lone surrogates; sanitize_for_dbtext replaces NUL plus lone surrogates; clean_identifier keeps the conservative charset [A-Za-z0-9._:/+@-]; deep_sanitize_for_json walks JSON-shaped containers and applies sanitize_for_json to every string leaf. Multi-byte Unicode that legitimate LLM-generated cross-language output carries (Greek identifiers in Mojo, U+2192 / U+21D2 arrows in Mojo and Pallas error formatters, CJK comments, math symbols, emoji, Unicode box-drawing in clang carets, CUTLASS template syntax like Layout<Shape<_32,_128>,Stride<_128,_1>>) passes through unchanged.

Integration is applied at the boundaries where the unsanitized text would crash or corrupt. gigaevo/utils/json.py dumps wraps obj in deep_sanitize_for_json before orjson and the stdlib fallback (covers every _dumps caller in Redis storage transparently). StageError in gigaevo/programs/core_types.py grows pydantic field_validators on type, message, and traceback so every construction path (from_exception, direct construction, JSON revalidation) yields sanitized values, which propagates through error.pretty(), program.format_errors() re-injection into LLM prompts, and downstream log lines. MigrantEnvelope.to_stream_fields in gigaevo/evolution/bus/transport.py runs program_data through deep_sanitize_for_json before json.dumps. MultiModelRouter in gigaevo/llm/models.py validates model_name once at construction through clean_identifier, redacts userinfo from base_url via a local _redact_url helper before any log call, and sanitizes server-returned model ids from /models probes. MutationStructuredOutput and MutationChange schemas in gigaevo/llm/agents/mutation.py grow field_validators on archetype, justification, code, description, explanation, and insights_used. TokenTracker.track wraps TokenUsage.from_response in try / except so a hostile token_usage payload no longer escapes ainvoke and kills the LangGraph node. clean_identifier is applied to ParamSpec.name in gigaevo/programs/stages/optimization/optuna/stage.py before the value flows into trial.suggest_* and embeds in Optuna's storage key. sanitize_for_log wraps the exception interpolations in gigaevo/runner/dag_runner.py, gigaevo/programs/stages/python_executors/execution.py, gigaevo/programs/stages/optimization/utils.py, gigaevo/programs/stages/validation.py, gigaevo/programs/dag/dag.py, gigaevo/database/redis_program_storage.py, gigaevo/database/state_manager.py, gigaevo/evolution/mutation/mutation_operator.py, gigaevo/prompts/coevolution/stages.py, gigaevo/prompts/coevolution/stats.py, gigaevo/prompts/fetcher.py, gigaevo/llm/agents/memory_selector.py, gigaevo/llm/bandit.py. gigaevo/utils/trackers/backends/redis.py replaces an ad-hoc tag normalization with clean_identifier and applies sanitize_for_dbtext on the history value plus deep_sanitize_for_json on the history record before json.dumps.

Tests add the sanitizer unit suite (sanitize_for_log ANSI / C0 / C1 / BIDI / surrogate coverage; sanitize_for_json minimum-viable surrogate handling; sanitize_for_dbtext NUL plus surrogate; clean_identifier charset and max_len; deep_sanitize_for_json recursive walk) plus three adversarial suites generated against the same module: a Unicode suite covering confusables, normalization invariance, zero-width characters, weak versus strong BIDI marks, variation selectors, tag characters, line and paragraph separators, BOM, soft hyphen, Zalgo combining stacks, CJK script families, RTL scripts, emoji ZWJ sequences, Fitzpatrick skin-tone modifiers, regional-indicator flags, mathematical alphanumerics, halfwidth and fullwidth forms, and non-Latin digit forms; a regex-bypass suite covering malformed CSI, intermediate bytes, private parameters, OSC family including security-relevant OSC 52, DCS / SOS / PM / APC, direct C1 introducers, bare ESC plus Fp / Fs gaps, adjacent surrogates, ANSI inside emoji ZWJ sequences, perf-bound DoS tests; a downstream-consumer suite that pipes sanitized output through json.dumps both encoding modes, pydantic.BaseModel.model_dump_json, pydantic.TypeAdapter.dump_python and dump_json, str.encode("utf-8"), loguru file sinks, subprocess argv, fakeredis SET / GET, sqlite3 TEXT round-trips, csv.writer round-trips, and a realistic openai 2.x BadRequestError carrying an ANSI-colorized nvcc error plus an embedded NUL plus a Greek-letter Mojo identifier. Integration tests under tests/llm/test_sanitize_wiring.py, tests/utils/test_text_sanitize_wiring.py, tests/stages/test_sanitize_integration.py, and tests/dag/test_sanitize_integration.py exercise each modified production module with a hostile fixture combining ANSI, NUL, CR, BEL, lone surrogate, and BIDI RLO, and assert the captured loguru output contains no raw hostile bytes, encodes cleanly as UTF-8, and round-trips through json.dumps. Two defects were discovered during the sanitizer audit and fixed: the lone-surrogate regex previously used a lookahead / lookbehind that mistakenly treated adjacent independent surrogates as a valid pair (chr(0xD800) + chr(0xDC00) survived and broke UTF-8 encoding downstream); clean_identifier with a negative max_len silently dropped a trailing character via the Python slice quirk and now raises ValueError. Total new test count is approximately 760, and the full target regression suite (tests/llm, tests/utils, tests/dag, tests/test_program.py, tests/stages, tests/database, tests/evolution, tests/prompts, tests/trackers, tests/infra) passes after the changes.

[GrigoryEvko]

Conflict map for cross-cutting overlaps with other open PRs

This PR introduces gigaevo.utils.text_sanitize and applies sanitize_for_log / deep_sanitize_for_json / field validators across ~22 files. Several open PRs touch the same lines for orthogonal reasons. The table below documents the residual overlap after pre-emptive decoupling commits were pushed where surgical decoupling was possible.

Other PR	Files	Hunks	Notes
#13 fix/llm-call-outcome-classifier	gigaevo/llm/bandit.py	1	Disjoint imports — clean union. Two duplicate-work commits on #13 (local _scrub_str / _scrub_surrogates in transport.py + StageError validators in core_types.py) were pre-emptively reverted on that branch since this PR's centralised helpers supersede them.
#14 loky-executor	gigaevo/programs/stages/python_executors/execution.py	3	#14 removes the memory-detection block; this PR adds sanitize_for_log to the adjacent log args. Resolution: take #14's structure + this PR's wrapping.
#15 fix/error-context-preservation	dag_runner.py (11), optimization/optuna/stage.py (2), optimization/utils.py (1), validation.py (1)	15	#15 converts logger.error → logger.exception and restructures error-construction sites that this PR also sanitises. For dag_runner: take #15's logger.exception (loguru auto-escapes the attached traceback). For the other 3 files: union — keep #15's structural fixes (full stderr, structural SyntaxError) + re-apply sanitize_for_log to interpolated strings. One wrapper.py overlap on #15 was pre-emptively reverted on that branch (it was dead code post-#14 anyway).
#16 fix/pipeline-builder-hygiene	—	0	No overlap.
#17 fix/aiohttp-network-layer	gigaevo/llm/models.py	1	#17 restructures _verify_models (TTL cache + jitter + auth); this PR sanitises the warning args inside the same loop. Resolution: take #17's restructure + this PR's sanitize_for_log on the warning.

Merge order: all overlaps are intrinsic line-level edits (different concerns at same sites), not duplicate work. Order is interchangeable; the second-merged PR needs the small line-level union shown in each PR's own conflict comment. Pre-emptive decoupling pushed where the overlap was duplicate work (#13 × 2 reverts, #15 × 1 revert) — those are documented in each branch's revert commits.

[GrigoryEvko]

Two integration concerns not in the conflict map above, surfaced while reconciling #10 + #14 + #20 on a nightly branch:

vs #14: tests/stages/test_sanitize_integration.py:356 passes max_memory_mb=None to evaluate_single. #14 drops that parameter (replaced by disk spilling — see 0ec8b88). After both merge: TypeError: evaluate_single() got an unexpected keyword argument 'max_memory_mb'. Whichever lands second: drop the kwarg from the test.

vs #20 (feat/dataplane-foundation): #20 renames RedisPromptStatsProvider._get_redis → _get_dataplane and replaces gigaevo.prompts.fetcher._get_sync_redis with a DataPlane handle. Test-side monkeypatches added by this PR break:

• tests/utils/test_text_sanitize_wiring.py: retarget patches at _get_dataplane; the archive-read path uses raw_hash_values (replaces the old hvals).
• tests/llm/test_sanitize_wiring.py: retarget urllib.request.urlopen patch at _fetch_available_models_at — that's the per-base-URL helper after fix(infra): replace httpx with aiohttp + requests for long-running asyncio stability #17 lands.

Verified locally by integrating all three PRs into a single branch — full suite (5862 tests) passes after the test-side adjustments. Parallel notes left on #14 and #20.