feat(v1): render wire images as ASCII or braille

[snimu]

Summary

Experimental opt-in image → ASCII/braille rendering at the v1 interception server.

[textify]
enabled = true       # disabled by default
# mode = "ascii"     # default when enabled; "braille" also available
width = 160

The transform runs before dialect parsing, so the provider/train renderer and the trace all see the same text. It catches data-URI images from task prompts, tool results, user simulators, and Chat/Responses/Anthropic wire formats without taskset or harness changes.

Design

• TextifyConfig on EnvConfig; v0/legacy configs reject enabled textify instead of silently ignoring it
• pure deterministic render core: ASCII + braille, explicit width/height, aspect correction, gamma, auto/manual inversion, ramp/threshold, hard character budget
• in-place content-part replacement preserves message ordering and protocol structure
• Responses file-backed images and HTTP(S) URLs pass through unchanged (no hot-path network fetch); computer screenshots become ordinary text observations with paired computer calls removed
• traces record what the model actually saw; original task data remains unchanged for scoring
• describe_textify(config) supports future self-supervised (image, params, rendering) data generation

Safety / performance

• Pillow/NumPy rendering runs in asyncio.to_thread, never on the shared aiohttp loop
• resize-first decoding avoids full-resolution float/RGBA intermediates
• 25 MP decoded-image ceiling, 100 MB payload ceiling
• 40k output-character default budget, 1M hard ceiling
• bounded 32-entry per-rollout resend cache
• malformed images consistently surface as recorded TaskErrors
• fresh-process 4K PNG measurement: ~65 MB RSS over import baseline (vs ~536 MB before the resize-first refactor)

Validation

• 27 focused textify/config tests pass
• changed Python files pass Ruff; lockfile check passes; no type diagnostics in changed files
• final Docker smoke, rebased onto c8ad8b4fc:
  • mmmu-v1, Physics validation, deepseek/deepseek-v4-flash, null harness
  • --textify.enabled true --textify.width 160
  • reward 1.0, no errors
• without textify, this text-only model rejected MMMU image requests upstream (404); with textify, subprocess and Docker rollouts complete normally

Note: full-repo Ruff currently reports an unrelated unused BytesIO import introduced on current main in environments/mmmu_v1/mmmu_v1/taskset.py; this PR does not touch it.

Initial experiments

Six real MMMU images were rendered and tested blind with subagents:

• shuffled image↔ASCII matching: 6/6 at widths 40, 120, 160; 4/6 at 80 (two sparse horizontal diagrams swapped — nearest-neighbor phase matters, not just width)
• ASCII-only descriptions at width 160 retained coarse geometry but lost semantics/text: a leg anatomy cross-section was described as Saturn; current-carrying wires as frying pans; labels/equations were unreadable
• tiny live MMMU slices were highly noisy across generation budgets, so this PR does not claim a universal minimum resolution; width 160 is a conservative initial default, with 120–160 a plausible coarse-geometry range

TEXTIFY_PLAN.md is a living design/experiment log. Future work explicitly covers an opt-in OCR/layout hybrid for text-heavy images (OCR text regions with coordinates/confidence; ASCII non-text regions), rather than merely increasing ASCII resolution.

Intentional limitations

• lossy by design — not vision equivalence
• data-URI images only; HTTP(S) and provider file IDs remain images
• color is reduced to luminance
• OCR/layout mode is future work

Note

Add ASCII and braille image rendering for wire images in v1 evaluations

• Adds TextifyConfig to configure image-to-text rendering (mode, width, height, gamma, invert, threshold, etc.) and exposes it from verifiers.v1.
• Implements image_to_text and render_url in verifiers/v1/utils/textify.py to decode image data URLs and emit fenced ASCII or braille art.
• Extends the ChatDialect, AnthropicDialect, and ResponsesDialect with textify_body methods that replace image parts with rendered text fences before forwarding to the model.
• The interception server applies textify transforms to inbound model-turn and aux requests, serializes opening-message fetches, and caches post-commit errors for replay instead of re-sampling.
• Adds pillow as a project dependency for image decoding.
• Risk: Textify is rejected for legacy v0 environments via a model validator; enabling it for v0 raises ValueError.

^{Macroscope summarized 6d8e123. (Automatic summaries will resume when PR exits draft mode or review begins).}

---

Note

Medium Risk
When enabled, every intercepted request pays thread-pool image work and changes what the model sees; retry/replay behavior on the hot path is more complex, though scoped to native v1 and off by default.

Overview
Textify is an opt-in path for native v1 evals: base64 images in task prompts, tool results, and user-simulator turns are turned into fenced ASCII or braille before dialect parsing, so text-only models and traces see the same rendered text without taskset changes.

TextifyConfig on EnvConfig (--textify.* / [textify]); legacy v0 configs error if enabled. Rendering uses Pillow + NumPy in asyncio.to_thread, with size/output limits and a bounded per-rollout render cache on RolloutSession.

Chat, Responses, and Anthropic dialects implement textify_body; Responses also textifies computer screenshots and drops paired computer calls. The interception server textifies model and aux requests, serializes opening-message textify under a lock, and extends the retry replay cache to post-commit TaskErrors (e.g. bad images after a model turn) so SDK retries do not resample.

Adds pillow, docs/skills reference for TextifyConfig, unit + e2e fixtures/tests, and routine lockfile bumps.

^{Reviewed by Cursor Bugbot for commit 6d8e123. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a substantial new feature (Textify) for rendering images as ASCII/braille text, adding new configuration, utility modules, dialect methods, and interception server integration. New features introducing new processing pipelines and user-facing capabilities warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[snimu]

Resolved all currently open review threads. The two newest correctness issues were valid and are fixed in df085537b: committed model turns are cached/resolved before later simulator or typed-textify failures so retries replay rather than re-sample, and successful serialized opening textification clears any prior failure before an opening @stop. Added deterministic regressions for both paths; 29 focused Textify tests pass. All 14 inline review threads are now resolved.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2d24496. Configure here.}