Skip to content

feat: add TOON output format (--format toon)#3147

Open
davidfierro wants to merge 1 commit into
snowflakedb:mainfrom
davidfierro:feature/toon-output-format
Open

feat: add TOON output format (--format toon)#3147
davidfierro wants to merge 1 commit into
snowflakedb:mainfrom
davidfierro:feature/toon-output-format

Conversation

@davidfierro

@davidfierro davidfierro commented Jul 18, 2026

Copy link
Copy Markdown

Pre-review checklist

  • If my changes add or modify user-facing interface (commands, flags, output formats), I consulted maintainers and got sign-off beforehand (how). — Requesting design sign-off in this PR; see "Design sign-off requested" below.
  • I've confirmed that instructions included in README.md are still correct after my changes in the codebase.
  • I've added or updated unit tests to verify correctness of my new code. (11 TOON tests in tests/output/test_printing.py, round-trip via toon_format.decode)
  • I've added or updated integration tests to verify correctness of my new code. — Happy to add once the design is signed off.
  • Manually tested on macOS
  • Manually tested on Windows
  • I've confirmed my changes are up-to-date with the target branch. (rebased on latest main)
  • I've described my changes in RELEASE-NOTES.md (see when and how).
  • I've updated documentation if behavior changed. — No in-repo user docs mention output formats; help-text snapshots regenerated.

Changes description

Adds TOON (Token-Oriented Object Notation, spec) as a new structured output format: snow sql -q "..." --format toon (available on every command, like CSV/JSON).

Why TOON

TOON is a compact, lossless text format designed for LLM consumption — YAML-style indentation plus CSV-style tabular arrays, which is exactly the shape of snow sql query results. Numbers below are from the project's published benchmarks:

  • Token efficiency: 39.9% fewer tokens than JSON on mixed datasets; up to −59.0% on fully tabular data (9,115 vs 22,245 tokens on time-series rows — the snow sql result shape). E-commerce −33.3%, nested config −31.9%.
  • LLM comprehension is not worse than JSON: mean retrieval accuracy 76.4% (TOON) vs 75.0% (JSON) while using 39.9% fewer tokens; TOON beats JSON on 3 of 4 evaluated models (Claude Haiku 59.8 vs 57.4, GPT-5 Nano 90.9 vs 89.0, Grok-4 58.4 vs 56.5; Gemini 3 Flash essentially tied at 96.7 vs 97.1). Field-retrieval accuracy 99.6%.
  • Use case: snow sql output consumed by agents/LLM pipelines — materially lower context cost with no loss of parseability (encode/decode round-trips are lossless; asserted in the unit tests).

Implementation

  • New OutputFormat.TOON enum value and an is_structured property that consolidates the previously-triplicated "structured format" check (printing dispatch, intermediate-output muting, streamlit log streaming).
  • TOON printer mirrors the CSV dispatch shape. Rows are pre-converted by _to_encodable_value to preserve data fidelity: Decimal as exact string (never float), bytes/bytearray as hex, non-finite floats as NaN/Infinity tokens (matching JSON output, since the toon encoder would collapse them to null), dates as ISO-8601, keys and string values passed through sanitize_for_terminal. Empty single-row results print null, matching JSON semantics.
  • Related fix: several commands gated their structured return on output_format.is_json (stage diff, app diff, app validate, app version create, app release-channel list, apps init, helpers write-project-definition-schema). Under --format csv these emitted no output at all (console muted + None return). They now use is_structured, so CSV and TOON get structured results.
  • Dependency: toon-format==0.9.0b1 (MIT, zero runtime dependencies, Python ≥3.8). Note: this pins a beta — the 0.1.0 release on PyPI is a stale placeholder; 0.9.0b1 is the spec-aligned implementation. Flagging explicitly for dependency review.

Design sign-off requested

Per adding-commands.md, a new output format needs maintainer sign-off — requesting it here:

  1. Format value: --format toon (case-insensitive, like the existing values).
  2. Lifecycle stage: proposed GA alongside CSV/JSON; if you prefer PrPr/PuPr, I'm happy to gate it behind a FeatureFlag (e.g. ENABLE_TOON_OUTPUT_FORMAT) in this PR.
  3. Result mapping: collections → tabular TOON; objects/messages → key-value TOON; multiple results → one TOON document per result separated by a blank line (mirrors CSV); empty single result → null (mirrors JSON).

Proposed follow-ups (out of scope here)

  • Consolidate the four parallel scalar-conversion tables in printing.py (CustomJSONEncoder, StreamingJSONEncoder, _write_csv_row, _to_encodable_value) into one shared converter.
  • Sanitize CSV header keys (same terminal-safety gap, pre-existing).
  • Fix the dead-code type guard in _stream_collection_as_csv (raises after the .keys() call that already crashes).

Adds TOON (Token-Oriented Object Notation, https://github.com/toon-format)
as a new structured output format. TOON is a token-efficient text format
for LLM consumption (30-60% fewer tokens than JSON).

- New OutputFormat.TOON enum value and is_structured property that
  consolidates the previously triplicated "structured format" check
  (printing, intermediate-output muting, streamlit log streaming).
- TOON printer mirrors the CSV dispatch; rows are pre-converted with
  _to_encodable_value to preserve data fidelity (Decimal as exact string,
  bytes/bytearray as hex, non-finite floats as NaN/Infinity tokens
  matching JSON output, terminal-sanitized keys and values). Empty
  single-query results print `null`, matching JSON semantics.
- Related fix: commands gating structured returns on is_json (stage diff,
  app diff, app version create, release-channel list, apps init, helpers
  write-project-definition-schema) now use is_structured, so they emit
  structured results for CSV and TOON instead of no output at all.
- Dependency: toon-format==0.9.0b1 (MIT, zero runtime deps, Python>=3.8).
  Note: 0.9.0b1 is a beta pin; the 0.1.0 release on PyPI is a stale
  placeholder.
@davidfierro
davidfierro force-pushed the feature/toon-output-format branch from 44da427 to 16e5f9a Compare July 18, 2026 18:41
@davidfierro
davidfierro marked this pull request as ready for review July 18, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant