feat(krl): DB-6 Phase A — explain keyword + structured query plan (#34) (#86)

hyperpolymath · claude · web-flow · commit 9c8d705867f1 · 2026-07-28T20:40:25.000+01:00
## Summary Closes the **Phase A** gap on #34. Before this, EXPLAIN was reachable only over HTTP with raw SQL; the KRL surface the issue actually specifies did not exist — `explain` appeared in neither the lexer nor the parser, and there was no selectivity model at all, not even the stub the criteria permit. ### What's added - **`explain` is a reserved word and a statement prefix**: `explain from … | filter … | take …`. - It parses to **`KRLExplainStmt`** — deliberately a *distinct statement type* rather than a flag on `KRLQueryStmt`, so every existing consumer of `KRLQueryStmt` still means "this query will actually execute" and cannot be handed a plan where it expected rows. - **`ExplainPlan.jl`** builds the plan shape the acceptance criteria specify — an ordered `scan` followed by one entry per stage, each filter carrying `column`, `comparator`, `value`, `indexed` and a selectivity estimate. Conjunctions are **split into separate filter operations**, which is what makes the DB-7 reorder meaningful later. ``` explain from knots | filter colouring_count_3 = 9 and crossing < 8 | take 5 {"op" => "scan", "table" => "knots"} {"op" => "filter", "column" => "colouring_count_3", "comparator" => "=", "value" => 9, "indexed" => true, "selectivity_estimate" => 0.5, "selectivity_source" => "stub"} {"op" => "filter", "column" => "crossing", "comparator" => "<", "value" => 8, "indexed" => false, "selectivity_estimate" => 0.5, "selectivity_source" => "stub"} {"op" => "take", "n" => 5} ``` ### Honesty properties (each asserted by a test) - **Selectivity is the stub the criteria permit pending DB-3 (#33) — and says so.** Every estimate carries `"selectivity_source" => "stub"`, and `plan_is_costed` returns `false` while any stub remains. That is the guard stopping a future DB-7 from silently optimising against invented numbers. - **`indexed` is driven by a deliberately short explicit column list.** An over-claiming list would make it a fake signal, which is worse than reporting `false`. - **A predicate the plan cannot describe is not mis-described.** Column-to-column comparisons, nested booleans and calls are reported opaquely (`"predicate" => "<expression>"`, no `column` key) rather than given a fabricated column. ## Verification 25 assertions in `server/krl/test/explain_test.jl`, wired into the dependency-free CI step alongside the other pure-Julia suites (so it runs before `Pkg.instantiate` and cannot be masked by a dependency breakage). The existing lexer / parser / sql / seam suites all still pass. ## Scope Phase A only. **Phase B (cost-based reordering) is deliberately not attempted** — it needs real selectivity, which is DB-3 (#33). The plan returned is execution order *as written*, with no reordering. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
diff --git a/.github/workflows/krl-verification.yml b/.github/workflows/krl-verification.yml
@@ -66,6 +66,7 @@ jobs:
           julia --color=yes server/krl/test/parser_test.jl
           julia --color=yes server/krl/test/sql_test.jl
           julia --color=yes server/krl/test/seam_test.jl
+          julia --color=yes server/krl/test/explain_test.jl
 
       # server/Project.toml [sources] resolves KnotTheory/Skein/AcceleratorGate
       # at server/../../<name>.jl — i.e. one directory above the repo checkout
diff --git a/server/krl/Ast.jl b/server/krl/Ast.jl
@@ -36,7 +36,7 @@ export KRLNode, KRLStatement, KRLSource, KRLPipeStage, KRLReturnItem,
        SortOrder, SortAsc, SortDesc,
        EdgeDir, EdgeForward, EdgeBackward, EdgeUndirected,
        # program + statements
-       KRLProgram, KRLQueryStmt, KRLLetStmt, KRLRuleDef, KRLAxiomDef,
+       KRLProgram, KRLQueryStmt, KRLExplainStmt, KRLLetStmt, KRLRuleDef, KRLAxiomDef,
        # query + sources
        KRLQuery, KRLSourceKnots, KRLSourceDiagrams, KRLSourceInvariants,
        KRLSourceNamed, KRLSourceSubquery,
@@ -123,6 +123,20 @@ struct KRLQueryStmt <: KRLStatement
     col::Int
 end
 
+"""
+    KRLExplainStmt(query, line, col)
+
+`explain from … | …` — DB-6 Phase A. Returns the query PLAN rather than
+running the query. Deliberately a distinct statement type rather than a flag
+on `KRLQueryStmt`, so that every existing consumer of `KRLQueryStmt`
+continues to mean "this query will actually execute".
+"""
+struct KRLExplainStmt <: KRLStatement
+    query::KRLNode   # KRLQuery
+    line::Int
+    col::Int
+end
+
 """
     KRLLetStmt(name, type_ann, expr, line, col)
 
diff --git a/server/krl/ExplainPlan.jl b/server/krl/ExplainPlan.jl
@@ -0,0 +1,152 @@
+# SPDX-License-Identifier: MPL-2.0
+# Copyright (c) 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
+#
+# DB-6 Phase A — structured query plan for `explain from … | …`.
+#
+# Produces the shape the DB-6 acceptance criteria specify: an ordered list of
+# operations, each tagged with its kind and, for filters, the column, the
+# comparator, the literal value, whether the column is indexed, and a
+# selectivity estimate.
+#
+# HONESTY NOTE ON SELECTIVITY
+# ---------------------------
+# There are no column histograms yet — that is DB-3 (#33). The criteria
+# explicitly permit a stub until DB-3 lands, so every estimate here is
+# `SELECTIVITY_STUB` and is reported with `"selectivity_source" => "stub"`.
+# Nothing downstream may treat these as measured: `plan_is_costed` returns
+# false while any estimate is a stub, so a future DB-7 cost-based reorder
+# cannot silently optimise against made-up numbers.
+
+# Included directly into the `KRL` module namespace (as Ast.jl / Parser.jl /
+# Evaluator.jl are) — not a submodule, so AST types resolve without imports.
+
+"""Placeholder selectivity used until DB-3 (#33) provides real histograms."""
+const SELECTIVITY_STUB = 0.5
+
+"""
+Columns carrying a secondary index today. Kept deliberately small and
+explicit: an over-claiming list would make `indexed` a fake signal, which is
+worse than reporting `false`. Extend as DB-3 lands real indexes.
+"""
+const INDEXED_COLUMNS = Set([
+    "colouring_count_3",
+    "colouring_count_5",
+    "presentation_hash",
+    "quandle_key",
+])
+
+_cmp_symbol(k::Symbol) = get(Dict(
+    :eq => "=", :neq => "!=", :lt => "<", :lte => "<=",
+    :gt => ">", :gte => ">=", :iso => "≅", :path => "~>", :in => "in",
+), k, string(k))
+
+_source_name(s::KRLSource) =
+    s isa KRLSourceKnots      ? "knots" :
+    s isa KRLSourceDiagrams   ? "diagrams" :
+    s isa KRLSourceInvariants ? "invariants" :
+    s isa KRLSourceNamed      ? s.name :
+    s isa KRLSourceSubquery   ? "<subquery>" : "<unknown>"
+
+# Pull (column, comparator, value) out of a comparison predicate when the
+# shape is the simple `column <cmp> literal` the plan can describe. Anything
+# else (nested boolean, function call, column-to-column) yields `nothing` and
+# is reported as an opaque predicate rather than being mis-described.
+_literal_value(e::KRLExpr) =
+    e isa KRLInt      ? e.n :
+    e isa KRLFloat    ? e.x :
+    e isa KRLString   ? e.s :
+    e isa KRLBool     ? e.b :
+    e isa KRLKnotName ? e.name : nothing
+
+function _simple_comparison(e::KRLExpr)
+    e isa KRLCompare || return nothing
+    e.left isa KRLVar || return nothing
+    val = _literal_value(e.right)
+    val === nothing && return nothing
+    (e.left.name, _cmp_symbol(e.op), val)
+end
+
+# Split a conjunction into its conjuncts so `filter a = 1 and b < 2` plans as
+# two filter operations, which is what makes a future reorder meaningful.
+function _conjuncts(e::KRLExpr)
+    e isa KRLAnd && return vcat(_conjuncts(e.left), _conjuncts(e.right))
+    [e]
+end
+
+function _filter_ops(pred::KRLExpr)
+    ops = Dict{String, Any}[]
+    for c in _conjuncts(pred)
+        simple = _simple_comparison(c)
+        if simple === nothing
+            push!(ops, Dict{String, Any}(
+                "op" => "filter",
+                "predicate" => "<expression>",
+                "indexed" => false,
+                "selectivity_estimate" => SELECTIVITY_STUB,
+                "selectivity_source" => "stub",
+            ))
+        else
+            col, cmp, val = simple
+            push!(ops, Dict{String, Any}(
+                "op" => "filter",
+                "column" => col,
+                "comparator" => cmp,
+                "value" => val,
+                "indexed" => col in INDEXED_COLUMNS,
+                "selectivity_estimate" => SELECTIVITY_STUB,
+                "selectivity_source" => "stub",
+            ))
+        end
+    end
+    ops
+end
+
+"""
+    explain_plan(q::KRLQuery) -> Vector{Dict{String, Any}}
+
+Ordered plan for `q`: a `scan` of the source followed by one entry per
+pipeline stage, with conjoined filters split into separate `filter`
+operations.
+
+The order returned is **execution order as written** — no reordering is
+performed. Cost-based reordering is DB-7 and requires real selectivity
+(DB-3), which is why every estimate here is flagged `"stub"`.
+"""
+function explain_plan(q::KRLQuery)::Vector{Dict{String, Any}}
+    plan = Dict{String, Any}[Dict{String, Any}(
+        "op" => "scan",
+        "table" => _source_name(q.source),
+    )]
+    for stage in q.stages
+        if stage isa KRLFilterStage
+            append!(plan, _filter_ops(stage.pred))
+        elseif stage isa KRLSortStage
+            push!(plan, Dict{String, Any}(
+                "op" => "sort",
+                "keys" => length(stage.items),
+            ))
+        elseif stage isa KRLTakeStage
+            push!(plan, Dict{String, Any}("op" => "take", "n" => stage.n))
+        elseif stage isa KRLSkipStage
+            push!(plan, Dict{String, Any}("op" => "skip", "n" => stage.n))
+        else
+            push!(plan, Dict{String, Any}("op" => _stage_op_name(stage)))
+        end
+    end
+    plan
+end
+
+_stage_op_name(s::KRLPipeStage) =
+    s isa KRLReturnStage    ? "return" :
+    s isa KRLGroupByStage   ? "group_by" :
+    lowercase(replace(string(nameof(typeof(s))), "KRL" => "", "Stage" => ""))
+
+"""
+    plan_is_costed(plan) -> Bool
+
+`true` only when every selectivity estimate in `plan` came from real
+statistics. While DB-3 (#33) is outstanding this is always `false` — the
+guard that stops DB-7 optimising against stub numbers.
+"""
+plan_is_costed(plan::Vector{Dict{String, Any}}) =
+    !any(get(op, "selectivity_source", "stub") == "stub" for op in plan)
diff --git a/server/krl/KRL.jl b/server/krl/KRL.jl
@@ -28,6 +28,7 @@ include("Lexer.jl")
 include("Ast.jl")
 include("Parser.jl")
 include("SqlFrontend.jl")
+include("ExplainPlan.jl")
 include("Evaluator.jl")
 
 end # module KRL
diff --git a/server/krl/Lexer.jl b/server/krl/Lexer.jl
@@ -62,6 +62,8 @@ const TokenKind = Symbol
 # ─────────────────────────────────────────────────────────────────────────────
 
 const KRL_KEYWORDS = Set([
+    # query-level modifiers
+    "explain",
     # pipeline stages
     "from", "filter", "sort", "take", "skip", "return",
     "group_by", "aggregate",
diff --git a/server/krl/Parser.jl b/server/krl/Parser.jl
@@ -189,6 +189,17 @@ function _parse_statement(ps::ParserState)::KRLStatement
         return _parse_let_stmt(ps)
     end
 
+    # explain from … | …   (DB-6 Phase A: plan, do not execute)
+    if _kw(ps, "explain")
+        _advance!(ps)
+        et = _peek(ps)
+        _kw(ps, "from") || throw(KRLParseError(
+            "expected `from` after `explain`, got $(et.kind)($(repr(et.value)))",
+            et.line, et.col))
+        q = _parse_query(ps)
+        return KRLExplainStmt(q, t.line, t.col)
+    end
+
     # from … | … (pipeline query)
     if _kw(ps, "from")
         q = _parse_query(ps)
@@ -197,7 +208,7 @@ function _parse_statement(ps::ParserState)::KRLStatement
 
     t = _peek(ps)
     throw(KRLParseError(
-        "expected statement (from/let/rule/axiom), got $(t.kind)($(repr(t.value)))",
+        "expected statement (from/explain/let/rule/axiom), got $(t.kind)($(repr(t.value)))",
         t.line, t.col))
 end
 
diff --git a/server/krl/test/explain_test.jl b/server/krl/test/explain_test.jl
@@ -0,0 +1,73 @@
+# SPDX-License-Identifier: MPL-2.0
+# Copyright (c) 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
+#
+# DB-6 Phase A — `explain from … | …` produces a structured plan.
+
+using Test
+include(joinpath(@__DIR__, "..", "KRL.jl"))
+using .KRL: parse_krl, explain_plan, plan_is_costed, SELECTIVITY_STUB,
+            KRLExplainStmt, KRLQueryStmt, KRLParseError
+
+@testset "DB-6: explain is a distinct statement, not a query" begin
+    prog = parse_krl("explain from knots | filter crossing < 8")
+    @test length(prog.statements) == 1
+    @test prog.statements[1] isa KRLExplainStmt
+    # A plain query must NOT become an explain — the two are distinguishable
+    # so no existing consumer can be handed a plan where it expected rows.
+    @test parse_krl("from knots | filter crossing < 8").statements[1] isa KRLQueryStmt
+end
+
+@testset "DB-6: explain requires a query after it" begin
+    @test_throws KRLParseError parse_krl("explain")
+    @test_throws KRLParseError parse_krl("explain take 5")
+end
+
+@testset "DB-6: plan shape matches the acceptance criteria" begin
+    prog = parse_krl(
+        "explain from knots | filter colouring_count_3 = 9 and crossing < 8 | take 5")
+    plan = explain_plan(prog.statements[1].query)
+
+    @test plan[1]["op"] == "scan"
+    @test plan[1]["table"] == "knots"
+
+    # A conjunction becomes SEPARATE filter operations — that is what makes a
+    # future cost-based reorder (DB-7) meaningful.
+    @test plan[2]["op"] == "filter"
+    @test plan[2]["column"] == "colouring_count_3"
+    @test plan[2]["comparator"] == "="
+    @test plan[2]["value"] == 9
+    @test plan[2]["indexed"] == true         # carries a secondary index
+
+    @test plan[3]["column"] == "crossing"
+    @test plan[3]["comparator"] == "<"
+    @test plan[3]["value"] == 8
+    @test plan[3]["indexed"] == false        # does not
+
+    @test plan[4]["op"] == "take"
+    @test plan[4]["n"] == 5
+end
+
+@testset "DB-6: selectivity is honestly labelled a stub until DB-3" begin
+    plan = explain_plan(
+        parse_krl("explain from knots | filter crossing < 8").statements[1].query)
+    filt = plan[2]
+    @test filt["selectivity_estimate"] == SELECTIVITY_STUB
+    @test filt["selectivity_source"] == "stub"
+    # The guard that stops DB-7 optimising against invented numbers. This must
+    # stay false until real histograms land (#33) — if it ever returns true
+    # while estimates are stubs, the cost model is lying.
+    @test plan_is_costed(plan) == false
+end
+
+@testset "DB-6: a predicate the plan cannot describe is not mis-described" begin
+    # Column-to-column comparison is not a simple `column <cmp> literal`, so
+    # it must be reported opaquely rather than given a fabricated column.
+    plan = explain_plan(
+        parse_krl("explain from knots | filter crossing < genus").statements[1].query)
+    @test plan[2]["op"] == "filter"
+    @test plan[2]["predicate"] == "<expression>"
+    @test !haskey(plan[2], "column")
+    @test plan[2]["indexed"] == false
+end
+
+println("krl-explain-tests-ok")
