Skip to content

[LQP Meta] Add formatted and non-formatted literal tokens - #228

Merged
nystrom merged 7 commits into
mainfrom
nn-meta-less-hashing
Mar 17, 2026
Merged

[LQP Meta] Add formatted and non-formatted literal tokens#228
nystrom merged 7 commits into
mainfrom
nn-meta-less-hashing

Conversation

@nystrom

@nystrom nystrom commented Mar 6, 2026

Copy link
Copy Markdown
Contributor

Summary

In the Julia pretty-printer, we want to obscure some constants (e.g., by hashing them so they don't appear in logs) but not all. For instance, literal values inside formula fragments should go through a ConstantFormatter hook, while values in config relations and attributes should be printed as-is.

This PR introduces formatted literal tokens (FORMATTED_INT, FORMATTED_STRING, etc.) into the grammar as aliases of their unformatted counterparts (INT, STRING, etc.):

  • Parsing: formatted tokens are aliases — they use the same lexer rules as the base tokens, so parsing behavior is unchanged.
  • Pretty-printing: formatted tokens dispatch to format_*_formatted builtins, which in the Julia SDK go through the ConstantFormatter hook on the PrettyPrinter (allowing RAICode to hash/obscure values). In Go and Python, the formatted variants behave identically to the base formatters.
  • Grammar: the value nonterminal (used in formula term) now uses formatted tokens. A new raw_value nonterminal uses plain tokens and is used in config_key_value, attribute, and specialized_value where constants should not be obscured. date/datetime are similarly split into formatted and raw_* variants.

Changes

  • grammar.y: Add %token_alias declarations, split value/raw_value and date/datetime/raw_date/raw_datetime, remove the constant nonterminal (was just an alias for value)
  • grammar.py, yacc_parser.py: Support %token_alias in grammar representation and parser
  • pretty_gen.py: Table-driven terminal formatting with formatted variant entries
  • codegen_templates.py, target_builtins.py: Register format_*_formatted builtins for all three target languages
  • pretty_printer.jl.template: Add format_int32/format_float32 to the ConstantFormatter protocol; formatted builtins dispatch through pp.constant_formatter, unformatted builtins use DEFAULT_CONSTANT_FORMATTER directly
  • Regenerated parsers and pretty-printers for Python, Go, and Julia SDKs

@nystrom
nystrom marked this pull request as ready for review March 9, 2026 17:11
@nystrom
nystrom requested a review from comnik March 9, 2026 17:11

@staworko staworko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

end

@testitem "Custom formatter - int32" setup=[PrettySetup] begin
struct Int32Formatter <: ConstantFormatter end

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a nice solution. 🎉

@nystrom
nystrom merged commit 1bd2fcf into main Mar 17, 2026
5 checks passed
@nystrom
nystrom deleted the nn-meta-less-hashing branch March 17, 2026 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants