fix: sanitize LLM-derived text before logging, JSON serialization, and database writes#10
Conversation
…d database writes Introduce gigaevo/utils/text_sanitize.py with five pure str-to-str functions and wire them at the boundaries where unsanitized LLM-derived text would crash or corrupt downstream consumers. StageError gets pydantic field_validators; gigaevo/utils/json.py dumps wraps obj through deep_sanitize_for_json before serializing; MigrantEnvelope and the tracker Redis backend gain serialization belts; MultiModelRouter validates model_name via clean_identifier and redacts userinfo from base_url before logging; mutation, memory_selector, token_tracking, bandit, optuna stage, dag_runner, and prompt subsystems route their LLM-derived interpolations through sanitize_for_log. Two defects discovered during the sanitizer audit are fixed: the lone-surrogate regex previously treated adjacent independent surrogates as a valid pair (chr(0xD800) + chr(0xDC00) survived and broke UTF-8 encoding); clean_identifier with negative max_len silently dropped a trailing character via the slice quirk and now raises ValueError. Adds approximately 760 tests across the sanitizer unit suite plus adversarial Unicode, regex-bypass, and downstream-consumer suites, and integration tests under tests/llm, tests/utils, tests/dag, tests/stages that exercise each modified module with a hostile fixture combining ANSI, NUL, CR, BEL, lone surrogate, and BIDI override.
test_init_log_with_hostile_model_name previously ran the real _verify_models HTTP probe against a fake base_url and spent around seven seconds waiting for the urllib timeout. The test exists to verify the init INFO banner sanitizes model_name and redacts base_url userinfo, not to exercise the server probe (the other two tests in the same class already do that with monkeypatched urlopen). Patching _verify_models to a no-op brings the test from 7.4 seconds down to under ten milliseconds.
…anitization _apply_modifications cleaned ParamSpec.name through clean_identifier but inserted the parameterized_snippet unchanged, so a snippet that originally referenced _optuna_params['old hostile name'] failed with KeyError once the trial dict only carried the cleaned name. Build a name_map alongside the cleaning loop, dedupe collisions so two distinct hostile names that clean to the same identifier do not collapse into one parameter, and rewrite every string-literal _optuna_params[...] reference inside every snippet before splicing. Tests now positively exec the rewritten code against a synthetic _optuna_params dict instead of swallowing exceptions.
…aths
flush() stored latest values under clean_identifier(tag), but
get_latest, get_history, and clear_series read with the raw caller
tag. A write_scalar('loss train', ...) was stored as 'losstrain' and
get_latest('loss train') returned an empty dict. Extract a single
_field_tag() helper applied symmetrically on every read and write
side; fall back to metric_<sha256[:12]> when sanitization yields an
empty tag so distinct hostile inputs do not collide. New
round-trip tests cover both clean and hostile tags through
write_scalar -> flush -> get_latest / list_metrics / get_history.
prompt_text_to_id called blob.encode() on the raw return value of the prompt entrypoint, raising UnicodeEncodeError when the LLM-generated prompt source synthesized a lone UTF-16 surrogate via an escape literal. The error aborted the prompt fitness pipeline. Apply sanitize_for_log inside prompt_text_to_id and at the two entrypoint call sites so the stored prompt text and the hashed prompt text always match. Clean prompts keep their historical sha256[:16] id unchanged (test_clean_system_only_hash_matches_previous_sha256 proves no migration breakage). Empty-after-sanitization input becomes an explicit error in the prompt execution stage and a skip with warning in the archive fetcher rather than hashing an empty blob.
parse_response computed safe_msg through sanitize_for_log for the loguru line and for state['error'] but the value placed in parsed_output['error'] still came from str(e). Downstream consumers that read parsed_output directly (run-state dumps, langfuse traces, re-injection into LLM prompts) now also see the sanitized form.
The structured-output schemas receive LLM responses verbatim. ANSI
escapes, NUL bytes, lone UTF-16 surrogates, and BIDI overrides flow
through bare str fields into reports, JSON dumps, Postgres TEXT
columns, and re-injection back into LLM prompts. Match the pattern
already applied to MutationStructuredOutput: pydantic
field_validators on ProgramInsight.{type, insight, tag, severity}
and TransitionInsight.{strategy, description} pipe each value
through sanitize_for_log at the schema layer. ProgramScore has no
str fields, no change. Tests cover the clean-input identity path,
ANSI stripping, lone-surrogate replacement, CR removal, and
preservation of legitimate Unicode (arrows, math symbols, template
syntax).
…m_fields" This reverts commit 7fbf680. The defence is now redundant against PR FusionBrainLab#10 (fix/llm-output-sanitization), which introduces ``gigaevo.utils.text_sanitize.deep_sanitize_for_json`` and applies it at the same migration-bus boundary with broader coverage (ANSI / BIDI / control characters, not only surrogates). Keeping the local ``_scrub_str`` / ``_scrub_surrogates`` helpers here forces a merge conflict with FusionBrainLab#10 regardless of merge order; removing them lets either PR merge independently against ``main``. If FusionBrainLab#10 is rejected, this commit should be reverted (or the original content cherry-picked back) to restore the defence on the bus.
… fields" This reverts commit a25678f. The defence is now redundant against PR FusionBrainLab#10 (fix/llm-output-sanitization), which adds ``sanitize_for_log`` field validators on ``StageError`` (covering ANSI / BIDI / control characters / surrogates). Keeping the local ``_scrub_surrogates`` validators here forces a merge conflict with FusionBrainLab#10 regardless of merge order; removing them lets either PR merge independently against ``main``. If FusionBrainLab#10 is rejected, this commit should be reverted (or the original content cherry-picked back) to restore the defence on ``StageError``.
|
Conflict map for cross-cutting overlaps with other open PRs This PR introduces
Merge order: all overlaps are intrinsic line-level edits (different concerns at same sites), not duplicate work. Order is interchangeable; the second-merged PR needs the small line-level union shown in each PR's own conflict comment. Pre-emptive decoupling pushed where the overlap was duplicate work (#13 × 2 reverts, #15 × 1 revert) — those are documented in each branch's revert commits. |
LLM responses and compiler stderr from heterogeneous toolchains (Python tracebacks, Triton MLIR diagnostics,
nvcc/ptxas/ CUTLASS template explosions, Mojo error formatter output, Pallas / JAX jaxpr traces, CuTe layout errors) flow through gigaevo intologurusinks,orjsonserialization for Redis Streams and the program archive,asyncpgTEXT columns,langfusetrace metadata, file paths, and back into subsequent LLM prompts as part of multi-agent loops. The text was previously passed through verbatim. A single lone UTF-16 surrogate anywhere in a stage error raisesUnicodeEncodeErrorfrom insideorjson, aborts theProgramwrite to Redis, and stalls the evolution loop. A literal NUL byte in a traceback is rejected byasyncpgwithA string literal cannot contain NUL (0x00) charactersand aborts the tracker write. ANSI escape sequences fromnvccandclangcolorization survive intologurufile sinks and corrupt log readers; CR survives and lets a multi-line LLM justification forge log entries that downstream parsers cannot distinguish from authentic records; BIDI overrides survive and hide content from operator log review.A minimal reproducer for the
orjsoncrash path, runnable from the repo root before this change:The change introduces
gigaevo/utils/text_sanitize.pywith five purestr -> strfunctions:sanitize_for_logstrips ANSI / BIDI overrides / lone surrogates and escapes C0 and C1 control characters except TAB and LF;sanitize_for_jsonreplaces lone surrogates;sanitize_for_dbtextreplaces NUL plus lone surrogates;clean_identifierkeeps the conservative charset[A-Za-z0-9._:/+@-];deep_sanitize_for_jsonwalks JSON-shaped containers and appliessanitize_for_jsonto every string leaf. Multi-byte Unicode that legitimate LLM-generated cross-language output carries (Greek identifiers in Mojo, U+2192 / U+21D2 arrows in Mojo and Pallas error formatters, CJK comments, math symbols, emoji, Unicode box-drawing inclangcarets, CUTLASS template syntax likeLayout<Shape<_32,_128>,Stride<_128,_1>>) passes through unchanged.Integration is applied at the boundaries where the unsanitized text would crash or corrupt.
gigaevo/utils/json.pydumpswrapsobjindeep_sanitize_for_jsonbeforeorjsonand the stdlib fallback (covers every_dumpscaller in Redis storage transparently).StageErroringigaevo/programs/core_types.pygrows pydanticfield_validatorsontype,message, andtracebackso every construction path (from_exception, direct construction, JSON revalidation) yields sanitized values, which propagates througherror.pretty(),program.format_errors()re-injection into LLM prompts, and downstream log lines.MigrantEnvelope.to_stream_fieldsingigaevo/evolution/bus/transport.pyrunsprogram_datathroughdeep_sanitize_for_jsonbeforejson.dumps.MultiModelRouteringigaevo/llm/models.pyvalidatesmodel_nameonce at construction throughclean_identifier, redacts userinfo frombase_urlvia a local_redact_urlhelper before any log call, and sanitizes server-returned model ids from/modelsprobes.MutationStructuredOutputandMutationChangeschemas ingigaevo/llm/agents/mutation.pygrowfield_validatorsonarchetype,justification,code,description,explanation, andinsights_used.TokenTracker.trackwrapsTokenUsage.from_responseintry / exceptso a hostiletoken_usagepayload no longer escapesainvokeand kills the LangGraph node.clean_identifieris applied toParamSpec.nameingigaevo/programs/stages/optimization/optuna/stage.pybefore the value flows intotrial.suggest_*and embeds in Optuna's storage key.sanitize_for_logwraps the exception interpolations ingigaevo/runner/dag_runner.py,gigaevo/programs/stages/python_executors/execution.py,gigaevo/programs/stages/optimization/utils.py,gigaevo/programs/stages/validation.py,gigaevo/programs/dag/dag.py,gigaevo/database/redis_program_storage.py,gigaevo/database/state_manager.py,gigaevo/evolution/mutation/mutation_operator.py,gigaevo/prompts/coevolution/stages.py,gigaevo/prompts/coevolution/stats.py,gigaevo/prompts/fetcher.py,gigaevo/llm/agents/memory_selector.py,gigaevo/llm/bandit.py.gigaevo/utils/trackers/backends/redis.pyreplaces an ad-hoc tag normalization withclean_identifierand appliessanitize_for_dbtexton the history value plusdeep_sanitize_for_jsonon the history record beforejson.dumps.Tests add the sanitizer unit suite (
sanitize_for_logANSI / C0 / C1 / BIDI / surrogate coverage;sanitize_for_jsonminimum-viable surrogate handling;sanitize_for_dbtextNUL plus surrogate;clean_identifiercharset andmax_len;deep_sanitize_for_jsonrecursive walk) plus three adversarial suites generated against the same module: a Unicode suite covering confusables, normalization invariance, zero-width characters, weak versus strong BIDI marks, variation selectors, tag characters, line and paragraph separators, BOM, soft hyphen, Zalgo combining stacks, CJK script families, RTL scripts, emoji ZWJ sequences, Fitzpatrick skin-tone modifiers, regional-indicator flags, mathematical alphanumerics, halfwidth and fullwidth forms, and non-Latin digit forms; a regex-bypass suite covering malformed CSI, intermediate bytes, private parameters, OSC family including security-relevant OSC 52, DCS / SOS / PM / APC, direct C1 introducers, bare ESC plus Fp / Fs gaps, adjacent surrogates, ANSI inside emoji ZWJ sequences, perf-bound DoS tests; a downstream-consumer suite that pipes sanitized output throughjson.dumpsboth encoding modes,pydantic.BaseModel.model_dump_json,pydantic.TypeAdapter.dump_pythonanddump_json,str.encode("utf-8"),logurufile sinks, subprocess argv,fakeredisSET / GET,sqlite3TEXT round-trips,csv.writerround-trips, and a realisticopenai 2.xBadRequestErrorcarrying an ANSI-colorizednvccerror plus an embedded NUL plus a Greek-letter Mojo identifier. Integration tests undertests/llm/test_sanitize_wiring.py,tests/utils/test_text_sanitize_wiring.py,tests/stages/test_sanitize_integration.py, andtests/dag/test_sanitize_integration.pyexercise each modified production module with a hostile fixture combining ANSI, NUL, CR, BEL, lone surrogate, and BIDI RLO, and assert the capturedloguruoutput contains no raw hostile bytes, encodes cleanly as UTF-8, and round-trips throughjson.dumps. Two defects were discovered during the sanitizer audit and fixed: the lone-surrogate regex previously used a lookahead / lookbehind that mistakenly treated adjacent independent surrogates as a valid pair (chr(0xD800) + chr(0xDC00)survived and broke UTF-8 encoding downstream);clean_identifierwith a negativemax_lensilently dropped a trailing character via the Python slice quirk and now raisesValueError. Total new test count is approximately 760, and the full target regression suite (tests/llm,tests/utils,tests/dag,tests/test_program.py,tests/stages,tests/database,tests/evolution,tests/prompts,tests/trackers,tests/infra) passes after the changes.