Skip to content

Commit d028fbd

Browse files
authored
BEP-049 M1–M4: backtick strings, interpolation, control flow, tagged templates (#3577)
Implements [BEP-049](https://beps.boundaryml.com/beps/49) (String Interpolation) milestones **M1 through M4**. ## What's in this PR **M1 — backtick string literals** `` `...` ``: new `BACKTICK` token, multi-tick delimiter ladders, escapes, multi-line auto-dedent (§12), usable in any expression position. **M2 — `${expr}` interpolation**: `${...}` as a block expression (§4), implicit `.to_string()` for non-string values (§11), and **strict typing (§7/§11)** — interpolating a nullable or non-stringable value is a compile-time error. **M3 — block control flow**: `${for (let x in xs)}…${endfor}`, `${if (c)}…${else if}…${else}…${endif}` using host syntax, with §13 whitespace control. **M4 — tagged templates** (§10): `` tag`...` `` with the `//baml:tagged_string` marker, `TaggedString { parts, values }`, tag resolution + signature validation, body-lambda params scoping into `${…}`, and **full VM execution** — static layouts (M4e.1a) and `${for}`/`${if}` runtime flattening (M4e.1b). ## Architecture: check-then-lower (clean diagnostics) Both backtick forms are unified behind a first-class `Expr::Template { tag: Default { elaborated } | Custom { tag, body }, segments }` node that survives through TIR. Untagged backticks no longer desugar early — interpolation diagnostics point at the original `${…}` span instead of leaking a synthetic `.to_string()` call. This matches how tagged templates (and TypeScript) check the structured node and lower late. ## Testing Full workspace suite green. Adds unit + runtime coverage for backtick literals, interpolation, strict-type diagnostics, control flow, and tagged-template execution (static, `${for}`/`${if}`, nested, mixed, captures, body-param injection, nested lambdas capturing loop-locals). ## Not in this PR M5 (built-in `prompt` tag), M6 (legacy `#"..."#` interop + migrator), M7 (drop minijinja). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * BEP-049 backtick string literals with `${}` interpolation, including `${for}` / `${if}` control-flow. * Tagged templates with custom tag expressions and structured `TaggedString` template metadata. * Backtick-aware LLM prompts using prompt closures, plus typed prompt/context helpers in the standard library. * **Improvements** * BEP-049 dedentation and block-tag trimming for multiline literals. * Extended escape decoding with backtick-specific line-ending normalization. * Formatter/lexer/parser now properly recognize and preserve backtick delimiter/interpolation syntax. * **Bug Fixes** * More precise template diagnostics and improved diagnostic span stability. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent 2f72dba commit d028fbd

160 files changed

Lines changed: 17618 additions & 2547 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

baml_language/.cargo/size-gate.toml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -36,9 +36,9 @@ package = "baml_cli"
3636
bin = "baml-cli"
3737
policy.max_delta_pct = 3.0
3838

39-
# macOS arm64: baseline 15.12 MB file.
39+
# macOS arm64: baseline 15.59 MB file.
4040
[artifacts.baml-cli.platform.aarch64-apple-darwin]
41-
max_file_bytes = 15_580_000
41+
max_file_bytes = 16_060_000
4242
# Linux x86_64: baseline 20.01 MB file.
4343
[artifacts.baml-cli.platform.x86_64-unknown-linux-gnu]
4444
max_file_bytes = 20_620_000
@@ -50,9 +50,9 @@ max_file_bytes = 16_810_000
5050
kind = "pack"
5151
policy.max_delta_pct = 3.0
5252

53-
# macOS arm64: baseline 11.50 MB file.
53+
# macOS arm64: baseline 12.01 MB file.
5454
[artifacts.packed-program.platform.aarch64-apple-darwin]
55-
max_file_bytes = 11_850_000
55+
max_file_bytes = 12_370_000
5656
# Linux x86_64: baseline 15.12 MB file.
5757
[artifacts.packed-program.platform.x86_64-unknown-linux-gnu]
5858
max_file_bytes = 15_570_000
Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
version = 1
2-
recorded_at = "2026-06-02T00:00:00Z"
3-
git_sha = "465d9d13e"
2+
recorded_at = "2026-06-17T00:00:00Z"
3+
git_sha = "1b7046243"
44

55
[artifacts.baml-cli]
6-
file_bytes = 18180634
7-
stripped_bytes = 18180672
8-
gzip_bytes = 8715275
6+
file_bytes = 15592656
7+
stripped_bytes = 15592704
8+
gzip_bytes = 7520693
99

1010
[artifacts.packed-program]
11-
file_bytes = 11501554
12-
gzip_bytes = 5522485
11+
file_bytes = 12009602
12+
gzip_bytes = 5662276

baml_language/Cargo.lock

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
//! Template-string dedenting.
2+
//!
3+
//! Shared with `sys_llm::jinja::render::preprocess_template`; both implement the
4+
//! same algorithm and should remain in lockstep until that callsite migrates to
5+
//! this one.
6+
//!
7+
//! Algorithm (BEP-049 §12, Kotlin `trimIndent` rule):
8+
//! 1. Compute the longest common leading-whitespace *prefix* across all non-blank
9+
//! lines, compared character-by-character. Tabs and spaces don't mix: a
10+
//! tab-indented line and a space-indented line share no common prefix, so the
11+
//! strip column drops to zero (§12 Rule 2).
12+
//! 2. Strip that common prefix from every line that has it; otherwise keep the
13+
//! line as-is (its content is whitespace-only).
14+
//! 3. Trim leading/trailing whitespace from the overall result.
15+
//!
16+
//! Used by BEP-049 backtick string literals (multi-line auto-dedent, §12). The
17+
//! legacy Jinja prompt pipeline (`sys_llm::preprocess_template`) keeps the older
18+
//! byte-count-min variant until that path is removed (M6) — they intentionally
19+
//! diverge on mixed tab/space indentation, which only the new backtick form specs.
20+
// Walk leading whitespace by *char* so we never split a multi-byte
21+
// Unicode whitespace codepoint (NBSP U+00A0 = 2 bytes, LINE SEPARATOR
22+
// U+2028 = 3 bytes, etc.). A naive `line.len() - line.trim_start().len()`
23+
// mixed with `&line[min_indent..]` panics when an NBSP-indented line is
24+
// sliced at a byte offset derived from a sibling ASCII-indented line.
25+
fn leading_whitespace_bytes(line: &str) -> usize {
26+
line.chars()
27+
.take_while(|c| c.is_whitespace())
28+
.map(char::len_utf8)
29+
.sum()
30+
}
31+
32+
// Strip leading-whitespace chars from a line until we've stripped at
33+
// least `target_bytes` bytes — always stopping on a char boundary, even
34+
// if that means over-stripping by a few bytes when a multi-byte
35+
// whitespace char straddles the target. (Under-stripping into the
36+
// middle of a char would panic; over-stripping at most one whitespace
37+
// char is benign — the line was leading-whitespace anyway.)
38+
fn strip_leading_indent(line: &str, target_bytes: usize) -> &str {
39+
if target_bytes == 0 {
40+
return line;
41+
}
42+
let mut consumed = 0usize;
43+
let mut split_at = 0usize;
44+
for c in line.chars() {
45+
if !c.is_whitespace() {
46+
break;
47+
}
48+
consumed += c.len_utf8();
49+
split_at += c.len_utf8();
50+
if consumed >= target_bytes {
51+
break;
52+
}
53+
}
54+
&line[split_at..]
55+
}
56+
57+
/// Longest common prefix of two strings, ending on a char boundary.
58+
fn common_prefix<'a>(a: &'a str, b: &str) -> &'a str {
59+
let end = a
60+
.char_indices()
61+
.zip(b.chars())
62+
.take_while(|((_, ca), cb)| ca == cb)
63+
.map(|((i, ca), _)| i + ca.len_utf8())
64+
.last()
65+
.unwrap_or(0);
66+
&a[..end]
67+
}
68+
69+
pub fn preprocess_template(template: &str) -> String {
70+
let lines: Vec<&str> = template.lines().collect();
71+
72+
// Longest common leading-whitespace *prefix* across non-blank lines, compared
73+
// char-by-char (§12 Rule 2: tabs and spaces don't mix). The strip column is
74+
// its byte length.
75+
let mut common: Option<&str> = None;
76+
for line in lines.iter().filter(|line| !line.trim().is_empty()) {
77+
let ws = &line[..leading_whitespace_bytes(line)];
78+
common = Some(match common {
79+
None => ws,
80+
Some(prev) => common_prefix(prev, ws),
81+
});
82+
}
83+
let strip = common.map_or(0, str::len);
84+
85+
lines
86+
.iter()
87+
.map(|line| {
88+
if leading_whitespace_bytes(line) >= strip {
89+
strip_leading_indent(line, strip)
90+
} else {
91+
line.trim()
92+
}
93+
})
94+
.collect::<Vec<_>>()
95+
.join("\n")
96+
.trim()
97+
.to_string()
98+
}
99+
100+
#[cfg(test)]
101+
mod tests {
102+
use super::*;
103+
104+
#[test]
105+
fn empty() {
106+
assert_eq!(preprocess_template(""), "");
107+
}
108+
109+
#[test]
110+
fn single_line() {
111+
assert_eq!(preprocess_template("hello"), "hello");
112+
}
113+
114+
#[test]
115+
fn uniform_indent_stripped() {
116+
let input = " hello\n world";
117+
assert_eq!(preprocess_template(input), "hello\nworld");
118+
}
119+
120+
#[test]
121+
fn min_indent_is_smallest() {
122+
// After stripping the smallest leading indent (2), the result is
123+
// " hello\nworld"; the final .trim() removes the leading " ".
124+
let input = " hello\n world";
125+
assert_eq!(preprocess_template(input), "hello\nworld");
126+
}
127+
128+
#[test]
129+
fn blank_lines_ignored_in_min_calc() {
130+
let input = " hello\n\n world";
131+
assert_eq!(preprocess_template(input), "hello\n\nworld");
132+
}
133+
134+
#[test]
135+
fn trims_leading_and_trailing() {
136+
let input = "\n hello\n world\n";
137+
assert_eq!(preprocess_template(input), "hello\nworld");
138+
}
139+
140+
#[test]
141+
fn tab_and_space_indent_do_not_mix() {
142+
// BEP-049 §12 Rule 2: tabs and spaces don't mix — a tab-indented line and
143+
// a four-space-indented line share no common leading-whitespace prefix, so
144+
// the strip column is zero. (Byte-count-min would wrongly strip 1, eating a
145+
// space from the second line.) The leading tab of line 1 is removed by the
146+
// final `.trim()`; line 2 keeps all four spaces.
147+
let input = "\t- foo\n - bar";
148+
assert_eq!(preprocess_template(input), "- foo\n - bar");
149+
}
150+
151+
#[test]
152+
fn nbsp_indent_does_not_panic() {
153+
// BEP-049 / ultrareview bug_001: when one line is indented with
154+
// NBSP (U+00A0, 2 UTF-8 bytes) and another with ASCII space,
155+
// `min_indent` computed in bytes lands inside the NBSP, and a
156+
// naive byte-slice `&line[1..]` panics with "byte index 1 is not
157+
// a char boundary". Realistic trigger: rich-text paste / macOS
158+
// Option+Space.
159+
let input = " hello\n\u{00A0}world";
160+
let _ = preprocess_template(input);
161+
}
162+
163+
#[test]
164+
fn line_separator_indent_does_not_panic() {
165+
// U+2028 LINE SEPARATOR is a 3-byte Unicode whitespace char.
166+
// Mixing with ASCII space exposes the same byte-vs-char bug.
167+
let input = " xy\n\u{2028}xy";
168+
let _ = preprocess_template(input);
169+
}
170+
}

0 commit comments

Comments
 (0)