fix: sanitize LLM-derived text before logging, JSON serialization, and database writes

[GrigoryEvko]

LLM responses and compiler stderr from heterogeneous toolchains (Python tracebacks, Triton MLIR diagnostics, nvcc / ptxas / CUTLASS template explosions, Mojo error formatter output, Pallas / JAX jaxpr traces, CuTe layout errors) flow through gigaevo into loguru sinks, orjson serialization for Redis Streams and the program archive, asyncpg TEXT columns, langfuse trace metadata, file paths, and back into subsequent LLM prompts as part of multi-agent loops. The text was previously passed through verbatim. A single lone UTF-16 surrogate anywhere in a stage error raises UnicodeEncodeError from inside orjson, aborts the Program write to Redis, and stalls the evolution loop. A literal NUL byte in a traceback is rejected by asyncpg with A string literal cannot contain NUL (0x00) characters and aborts the tracker write. ANSI escape sequences from nvcc and clang colorization survive into loguru file sinks and corrupt log readers; CR survives and lets a multi-line LLM justification forge log entries that downstream parsers cannot distinguish from authentic records; BIDI overrides survive and hide content from operator log review.

A minimal reproducer for the orjson crash path, runnable from the repo root before this change:

import openai, httpx, orjson
req = httpx.Request("POST", "https://api.openai.com/v1/chat/completions")
resp = httpx.Response(429, request=req)
err = openai.RateLimitError("rate limited \ud83d", response=resp, body=None)
data = {"error": str(err)}
orjson.dumps(data)
# orjson.JSONEncodeError: surrogates not allowed

The change introduces gigaevo/utils/text_sanitize.py with five pure str -> str functions: sanitize_for_log strips ANSI / BIDI overrides / lone surrogates and escapes C0 and C1 control characters except TAB and LF; sanitize_for_json replaces lone surrogates; sanitize_for_dbtext replaces NUL plus lone surrogates; clean_identifier keeps the conservative charset [A-Za-z0-9._:/+@-]; deep_sanitize_for_json walks JSON-shaped containers and applies sanitize_for_json to every string leaf. Multi-byte Unicode that legitimate LLM-generated cross-language output carries (Greek identifiers in Mojo, U+2192 / U+21D2 arrows in Mojo and Pallas error formatters, CJK comments, math symbols, emoji, Unicode box-drawing in clang carets, CUTLASS template syntax like Layout<Shape<_32,_128>,Stride<_128,_1>>) passes through unchanged.

Integration is applied at the boundaries where the unsanitized text would crash or corrupt. gigaevo/utils/json.py dumps wraps obj in deep_sanitize_for_json before orjson and the stdlib fallback (covers every _dumps caller in Redis storage transparently). StageError in gigaevo/programs/core_types.py grows pydantic field_validators on type, message, and traceback so every construction path (from_exception, direct construction, JSON revalidation) yields sanitized values, which propagates through error.pretty(), program.format_errors() re-injection into LLM prompts, and downstream log lines. MigrantEnvelope.to_stream_fields in gigaevo/evolution/bus/transport.py runs program_data through deep_sanitize_for_json before json.dumps. MultiModelRouter in gigaevo/llm/models.py validates model_name once at construction through clean_identifier, redacts userinfo from base_url via a local _redact_url helper before any log call, and sanitizes server-returned model ids from /models probes. MutationStructuredOutput and MutationChange schemas in gigaevo/llm/agents/mutation.py grow field_validators on archetype, justification, code, description, explanation, and insights_used. TokenTracker.track wraps TokenUsage.from_response in try / except so a hostile token_usage payload no longer escapes ainvoke and kills the LangGraph node. clean_identifier is applied to ParamSpec.name in gigaevo/programs/stages/optimization/optuna/stage.py before the value flows into trial.suggest_* and embeds in Optuna's storage key. sanitize_for_log wraps the exception interpolations in gigaevo/runner/dag_runner.py, gigaevo/programs/stages/python_executors/execution.py, gigaevo/programs/stages/optimization/utils.py, gigaevo/programs/stages/validation.py, gigaevo/programs/dag/dag.py, gigaevo/database/redis_program_storage.py, gigaevo/database/state_manager.py, gigaevo/evolution/mutation/mutation_operator.py, gigaevo/prompts/coevolution/stages.py, gigaevo/prompts/coevolution/stats.py, gigaevo/prompts/fetcher.py, gigaevo/llm/agents/memory_selector.py, gigaevo/llm/bandit.py. gigaevo/utils/trackers/backends/redis.py replaces an ad-hoc tag normalization with clean_identifier and applies sanitize_for_dbtext on the history value plus deep_sanitize_for_json on the history record before json.dumps.

Tests add the sanitizer unit suite (sanitize_for_log ANSI / C0 / C1 / BIDI / surrogate coverage; sanitize_for_json minimum-viable surrogate handling; sanitize_for_dbtext NUL plus surrogate; clean_identifier charset and max_len; deep_sanitize_for_json recursive walk) plus three adversarial suites generated against the same module: a Unicode suite covering confusables, normalization invariance, zero-width characters, weak versus strong BIDI marks, variation selectors, tag characters, line and paragraph separators, BOM, soft hyphen, Zalgo combining stacks, CJK script families, RTL scripts, emoji ZWJ sequences, Fitzpatrick skin-tone modifiers, regional-indicator flags, mathematical alphanumerics, halfwidth and fullwidth forms, and non-Latin digit forms; a regex-bypass suite covering malformed CSI, intermediate bytes, private parameters, OSC family including security-relevant OSC 52, DCS / SOS / PM / APC, direct C1 introducers, bare ESC plus Fp / Fs gaps, adjacent surrogates, ANSI inside emoji ZWJ sequences, perf-bound DoS tests; a downstream-consumer suite that pipes sanitized output through json.dumps both encoding modes, pydantic.BaseModel.model_dump_json, pydantic.TypeAdapter.dump_python and dump_json, str.encode("utf-8"), loguru file sinks, subprocess argv, fakeredis SET / GET, sqlite3 TEXT round-trips, csv.writer round-trips, and a realistic openai 2.x BadRequestError carrying an ANSI-colorized nvcc error plus an embedded NUL plus a Greek-letter Mojo identifier. Integration tests under tests/llm/test_sanitize_wiring.py, tests/utils/test_text_sanitize_wiring.py, tests/stages/test_sanitize_integration.py, and tests/dag/test_sanitize_integration.py exercise each modified production module with a hostile fixture combining ANSI, NUL, CR, BEL, lone surrogate, and BIDI RLO, and assert the captured loguru output contains no raw hostile bytes, encodes cleanly as UTF-8, and round-trips through json.dumps. Two defects were discovered during the sanitizer audit and fixed: the lone-surrogate regex previously used a lookahead / lookbehind that mistakenly treated adjacent independent surrogates as a valid pair (chr(0xD800) + chr(0xDC00) survived and broke UTF-8 encoding downstream); clean_identifier with a negative max_len silently dropped a trailing character via the Python slice quirk and now raises ValueError. Total new test count is approximately 760, and the full target regression suite (tests/llm, tests/utils, tests/dag, tests/test_program.py, tests/stages, tests/database, tests/evolution, tests/prompts, tests/trackers, tests/infra) passes after the changes.