Merge pull request #1 from regtab/feature/embedded-rtl-dsl

Embedded RTL DSL (pyregtab.dsl), release 0.2.0

Author: shigarov

Cargo.lock

@@ -1,6 +1,6 @@
 [package]
 name = "pyregtab"
-version = "0.1.1"
+version = "0.2.0"
 edition = "2021"
 description = "Native core of pyRegTab: RTL compiler, ATP matcher and table interpreter"
 license = "MIT"

Cargo.toml

@@ -12,7 +12,8 @@ TableSyntax → RtlCompiler/TablePattern → AtpMatcher → TableInterpreter →
 ```
 
 **pyRegTab 0.1.x ≙ jRegTab 0.4.0** (same API, same semantics, same test
-corpus).
+corpus). **0.2.0** adds the embedded RTL DSL (`pyregtab.dsl`), mirroring
+jRegTab's `ru.icc.regtab.dsl`.
 
 ## Installation
 
@@ -52,6 +53,9 @@ Patterns can also be built without RTL, via the fluent spec API
 snake_case method names), and serialized back to RTL with
 `AtpToRtlSerializer.serialize(pattern)`.
 
+For a terser, RTL-like way to build patterns in code, use the **embedded RTL**
+DSL (`pyregtab.dsl`) — see [Embedded RTL](#embedded-rtl) below.
+
 Named Python predicates are attached to RTL via `EXT('name')`:
 
 ```python
@@ -63,6 +67,32 @@ p = RtlCompiler.compile(
 )
 ```
 
+## Embedded RTL
+
+The `pyregtab.dsl` module is a fluent DSL that reads almost like RTL but is
+ordinary Python — with IDE completion, structural typing, pattern composition
+via plain variables, and Python callables as escape-hatch constraints. It builds
+the **same `TablePattern`** objects as the compiler (verified byte-for-byte
+against `RtlCompiler.compile` for a representative set of tasks in
+`tests/test_dsl.py`).
+
+```python
+from pyregtab.dsl import *
+
+# RTL: { [ [VAL : ST*->REC] [VAL]{2} []+ ]
+#        [ []               [VAL]{4} []+ ] }+
+p = table(
+    subtable(
+        row(cell(VAL, rec(ST.unbounded())), cell(VAL).exactly(2), skip().one_or_more()),
+        row(skip(),                         cell(VAL).exactly(4), skip().one_or_more()),
+    ).one_or_more())
+```
+
+Method names are snake_case (`.one_or_more()`, `.and_()`, `.split_by()`); the
+vocabulary constants (`VAL`, `ST`, `COL`, `C(n)`, …) match RTL. See the
+[Embedded RTL guide](docs/embedded-rtl.md) for the full mapping and the
+`where(...)` escape hatch.
+
 ## API mapping (Java → Python)
 
 | Java | Python |
@@ -104,14 +134,17 @@ Rust (`pyregtab._core`, built with [PyO3](https://pyo3.rs) and
 
 ## Testing
 
-`pytest tests` runs (1 878 tests):
+`pytest tests` runs (1 904 tests):
 
 - the full benchmark suite — tasks 001–150 (Foofah, RegTab, Baikal),
   every fixture variant, **both** via RTL patterns and via ATP patterns
   built with the Python spec API (1 500 task variants in total; fixtures
   are copied verbatim from jRegTab into `tests/fixtures/tasks`, ATP
   builders are mechanically translated from the Java tests by
   `tools/translate_atp.py`);
+- embedded RTL DSL parity — 26 representative tasks/constructs built with
+  `pyregtab.dsl` produce byte-identical ATP to `RtlCompiler.compile`
+  (`tests/test_dsl.py`);
 - the RTL conformance corpus (positive canonical forms, fixed points,
   negative rejections);
 - RTL↔ATP round-trip for tasks 001–050;

README.md

@@ -101,6 +101,16 @@ cell predicates receive a `Cell`, item filters receive
 predicates cannot be serialized back to RTL (as in Java) and disable the
 GIL-release fast path.
 
+## Embedded RTL DSL
+
+`pyregtab.dsl` is a fluent, RTL-like layer over the ATP factories above:
+`table/subtable/row/subrow/cell`, atoms `val/attr/aux`, actions `rec/avp/join/
+fill/prefix/suffix`, provider constants (`ST`, `COL`, `C(n)`, …) with
+`.and_()/.or_()/.card()/.unbounded()` and traversal methods, and `where(...)`
+Python-callable escape hatches. It builds ordinary `TablePattern` objects,
+byte-identical to `RtlCompiler.compile` for lambda-free patterns. See the
+[Embedded RTL guide](embedded-rtl.md).
+
 ## Recordset transformations
 
 `WhitespaceNormalization()`, `AnchorAttributeAtPosition(pos)`,

docs/api.md

@@ -0,0 +1,194 @@
+# Embedded RTL
+
+**Embedded RTL** is a Python DSL (module `pyregtab.dsl`) that mirrors RTL syntax while
+remaining ordinary Python code. It combines the brevity of RTL with full Python integration:
+callables as constraints, IDE completion, and pattern composition with plain variables and
+functions.
+
+Patterns built with embedded RTL produce **exactly the same `TablePattern` objects** as
+`RtlCompiler.compile(...)` — this equivalence is verified task-by-task for a representative
+subset of the benchmark corpus (`tests/test_dsl.py`): for every lambda-free pattern the DSL
+builds a structurally identical ATP and serializes to the same canonical RTL.
+
+```python
+from pyregtab.dsl import *
+```
+
+## Quick example
+
+RTL (task 001):
+
+```
+{ [ [VAL : ST*->REC] [VAL]{2} []+ ]
+  [ []               [VAL]{4} []+ ] }+
+```
+
+Embedded RTL:
+
+```python
+p = table(
+    subtable(
+        row(cell(VAL, rec(ST.unbounded())), cell(VAL).exactly(2), skip().one_or_more()),
+        row(skip(),                         cell(VAL).exactly(4), skip().one_or_more()),
+    ).one_or_more())
+```
+
+The result is a plain `pyregtab.TablePattern` — match and interpret it exactly like any other:
+
+```python
+from pyregtab import AtpMatcher, TableInterpreter
+
+itm = AtpMatcher.match(p, syntax)
+rs = TableInterpreter().interpret(itm)
+```
+
+---
+
+## Vocabulary
+
+### Pattern levels and quantifiers
+
+| RTL | Embedded RTL |
+|---|---|
+| table pattern | `table(subtable…)` |
+| `{ rows }` subtable | `subtable(row…)` |
+| `[ cells ]` row | `row(cell…)` |
+| `{ cells }` explicit subrow | `subrow(cell…)`, mixed with implicit runs: `row(subrow(…), subrow(…))` |
+| `+` `*` `?` `{n}` | postfix `.one_or_more()` `.zero_or_more()` `.zero_or_one()` `.exactly(n)` |
+
+A row with plain cells (`row(cell…)`) wraps them into one implicit subrow, exactly like the
+compiler.
+
+### Cells and guards
+
+| RTL | Embedded RTL |
+|---|---|
+| `[]` | `skip()` |
+| `[VAL : acts]` | `cell(VAL, acts…)` |
+| `[!BLANK ? VAL : acts]` | `cell(not_blank(), VAL, acts…)` |
+| `[BLANK]` (condition-only) | `cell(blank())` |
+| guards `BLANK` `"re"` `~"s"` (+ `!`) | `blank()` `re("…")` `contains("…")` / `not_blank()` `not_re("…")` `not_contains("…")` |
+| — (escape hatch) | `cell(where("desc", lambda c: …), VAL)` |
+
+### Content specifications
+
+| RTL | Embedded RTL |
+|---|---|
+| `VAL` / `ATTR` / `AUX` / `_` | constants `VAL ATTR AUX SKIP`; atoms `val(acts…)` `attr(…)` `aux(…)` |
+| `VAL #'tag'` | `val(…).tagged("tag")` |
+| `VAL = NORM` (also `TRIM UC LC REPL SUBSTR`, chains) | `val(…).extract(NORM)`, `repl("rx","rep")`, `substr(b,e)`, `chain(…)` |
+| compound `A "d" B` | `val(…).then(" ", val(…))` (chainable) |
+| delimited `(VAL : …){","}` | `val(…).split_by(",")` |
+| conditional `BLANK ? _ \| VAL` | `when(blank(), SKIP, VAL)` (directives and specs mix freely) |
+
+### Providers and constraints
+
+| RTL | Embedded RTL |
+|---|---|
+| `LT RT AV BW ROW COL SR SC ST NCL CL STR` | constants with the same names (type `Prov`) |
+| `Cn` / `Ca..b` / `C+n`, `C-n` / `C+a..b` / `C+a..*` | `C(n)` / `C(lo,hi)` / `Crel(±n)` / `Crel(lo,hi)` / `CrelFrom(lo)` |
+| `Rn Ra..b R±n`, `Pn Pa..b P±n` | `R(…)`, `Rrel(…)`, `P(…)`, `Prel(…)` |
+| `#'t'` / `!#'t'` | `tag("t")` / `not_tag("t")` |
+| item `"re"`, `~"s"`, `BLANK` (+ `!`) | `item_re item_contains item_blank` / `item_not_re item_not_contains item_not_blank` |
+| `&` conjunction | `.and_(…)` |
+| `\|` disjunction | `.or_(…)`; distribution matches the compiler: `A.and_(B.or_(C))` ≙ `(A&B)\|(A&C)` |
+| `{n}` / `*` cardinality | `.card(n)` / `.unbounded()` |
+| `-` / `^` / `-^` traversal | `.reversed()` / `.col_major()` / `.reversed_col_major()` |
+| — (escape hatch) | `ROW.where("desc", lambda anchor, candidate: …)` |
+
+!!! note "Why `C(n)` and `Crel(n)` are separate"
+    `C(n)` and `C(lo, hi)` are absolute column constraints (one column, or a range); `Crel`
+    takes a signed anchor-relative delta: `Crel(1)` ≙ `C+1`, `Crel(-1)` ≙ `C-1`, and
+    `Crel(lo, hi)` / `CrelFrom(lo)` give relative and open-ended relative ranges. The naming
+    mirrors jRegTab's, where absolute and relative forms must be distinct factories.
+
+### Actions and context providers
+
+| RTL | Embedded RTL |
+|---|---|
+| `(…)->REC` / `REC(n)` / `REC('s')` | `rec(…)` / `rec(n, …)` / `rec_split("s", …)` |
+| `prov->AVP` / `'NAME'->AVP` | `avp(prov)` / `avp("NAME")` |
+| `(…)->JOIN` / `JOIN(k)` | `join(…)` / `join(k, …)` |
+| `(…)->FILL('d')`, `PREFIX`, `SUFFIX` | `fill("d", …)`, `prefix(…)`, `suffix(…)` (delimiter optional) |
+| `'EUR'` context literal | `lit("EUR")` (VALUE under REC/JOIN, ATTRIBUTE otherwise — as in the compiler) |
+| `@'K'='V'` | `ctx_avp("K", "V")` |
+
+Provider kinds (VAL/ATTR/UNRESTRICTED) are inferred from the action, exactly as in the RTL
+compiler — `rec(ST)` builds a VAL provider, `avp(SC)` an ATTR provider.
+
+### Level-scoped (inherited) actions and conditions
+
+RTL allows action specs and a condition at the table, subtable, row, and subrow level; they are
+merged down into every atom below. Embedded RTL mirrors this with `acts(…)` and an optional
+leading `CellPredicate`:
+
+```python
+# RTL: [ BW*->REC { [ATTR] [VAL] }* ]
+row(acts(rec(BW.unbounded())),
+    subrow(cell(ATTR), cell(VAL)).zero_or_more())
+
+# RTL: !BLANK ? BW*->REC [ [VAL] ]+
+table(not_blank(), acts(rec(BW.unbounded())), subtable(row(cell(VAL)).one_or_more()))
+```
+
+### Settings
+
+The RTL settings prefix maps to recordset transformations:
+
+```python
+# RTL: <NORM,ANCH(1),SPLIT(",")> …
+table(…).with_transformations(norm(), anch(1), split(","))
+```
+
+### Fragments are just Python
+
+RTL named fragments `$name=[…]` become plain variables — and gain parameterisation for free:
+
+```python
+# RTL: $V=['\d+' ? VAL: (COL&#'H'*,ROW&#'S'*)->REC]  …  [$V]+
+v = cell(re(r"\d+"), VAL, rec(COL.and_(tag("H")).unbounded(),
+                             ROW.and_(tag("S")).unbounded()))
+row(cell(blank()).one_or_more(), v.one_or_more())
+```
+
+---
+
+## Escape hatches into Python
+
+The reason embedded RTL exists: anywhere the model accepts a predicate, a plain Python callable
+works.
+
+```python
+p = table(subtable(row(
+    cell(where("is_total", lambda c: c.text.startswith("Total")), VAL,
+         rec(ROW.where("is_num", lambda a, c: c.str.isdigit()).unbounded())),
+    cell(VAL).one_or_more())))
+```
+
+Patterns containing `where(...)` **cannot be serialized back to RTL** (an opaque Python callable
+has no RTL analog) and two patterns with distinct callables never compare equal. The serializable
+alternative is a named `EXT('name')` binding in the string compiler, which round-trips:
+
+```python
+from pyregtab import RtlCompiler, Bindings
+
+p = RtlCompiler.compile(
+    "{ [ [EXT('is_total') ? VAL : ST*->REC] []+ ] }+",
+    Bindings.of().cell("is_total", lambda cell: cell.text.startswith("Total")),
+)
+```
+
+---
+
+## Relation to the other two APIs
+
+| | RTL string | Embedded RTL | ATP API |
+|---|---|---|---|
+| Brevity | ✅✅ | ✅ | ❌ |
+| Python callables | via `EXT` + `Bindings` | ✅ direct | ✅ direct |
+| IDE completion / structural typing | ❌ | ✅ | ✅ |
+| Layer | compiles to ATP | thin sugar over ATP | the model itself |
+
+Embedded RTL is a construction layer only — it adds no expressive power beyond ATP, and every
+pattern it builds is an ordinary `TablePattern`. The [ATP API](model/atp.md) remains the
+documented low-level layer.

docs/embedded-rtl.md

@@ -150,6 +150,7 @@ for record in rs.records:
 ## What's next
 
 - [RTL reference](rtl-reference.md) — complete syntax for the RTL DSL.
+- [Embedded RTL](embedded-rtl.md) — build the same patterns fluently in Python (`pyregtab.dsl`).
 - [ITM](model/itm.md) — syntactic and semantic layers, working state, table interpretation.
 - [ATP](model/atp.md) — pattern hierarchy, content specs, matching algorithm.
 - [API reference](api.md) — public classes and methods.

docs/getting-started.md

@@ -39,6 +39,7 @@ nav:
   - Home: index.md
   - Getting started: getting-started.md
   - RTL reference: rtl-reference.md
+  - Embedded RTL: embedded-rtl.md
   - IDE support: ide-support.md
   - Formal model:
       - ITM: model/itm.md