Derived tables in FROM correlate to outer scope: always-defer execution into FromSource.LateralPlan, JoinDriver lateral branch applies ON predicates and null-fills for LEFT, fixes EF Core's Distinct().Count() shape.

RyanLamansky · RyanLamansky · commit b2ec9c7a0f67 · 2026-05-08T14:38:40.000-05:00
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -156,7 +156,11 @@ ORDER BY: a non-set-op SELECT keeps branch-internal ORDER BY (which can referenc
 ### JOINs
 `INNER JOIN ... ON`, bare `JOIN` (= INNER), `LEFT [OUTER] JOIN ... ON`, `CROSS JOIN`, `CROSS APPLY (...) [AS alias]`, `OUTER APPLY (...) [AS alias]`. Multi-table chains compose left-to-right. Self-joins via alias work. ON-predicate UNKNOWN excludes. Aliases parse with or without `AS`.
 
-APPLY is the lateral form: the right side is a derived-table SELECT re-executed per outer row, with the outer tuple's columns visible inside its WHERE / projection. CROSS APPLY drops outer rows whose plan yields zero rows (INNER-style); OUTER APPLY null-fills the right side (LEFT-style). No `ON` clause — correlation lives inside the inner WHERE. The lateral plan stays deferred (`FromSource.LateralPlan`) and re-executes via `Selection.Execute(currentTupleResolver)` per outer tuple in `JoinDriver`. EF Core 10 emits `CROSS APPLY` for `SelectMany(a => a.Books.Where(...))` over a collection navigation.
+APPLY is the lateral form: the right side is a derived-table SELECT re-executed per outer row, with the outer tuple's columns visible inside its WHERE / projection. CROSS APPLY drops outer rows whose plan yields zero rows (INNER-style); OUTER APPLY null-fills the right side (LEFT-style). No `ON` clause — correlation lives inside the inner WHERE. EF Core 10 emits `CROSS APPLY` for `SelectMany(a => a.Books.Where(...))` over a collection navigation.
+
+**Derived tables in FROM are always deferred** — `ParseSingleFromSource` stores every `(SELECT ...) AS alias` source on `FromSource.LateralPlan` and the executor re-runs it through `Selection.Execute(outerResolver)` per surrounding row. This matches SQL Server's "any FROM derived table can correlate" rule (not just APPLY), and is required for shapes like `(SELECT COUNT(*) FROM (SELECT DISTINCT col FROM t WHERE t.k = outer.k) AS sub)` that EF Core 10 emits for `Distinct().Count()` over correlated sets. Static parse-time correlation detection isn't sufficient because outer references in WHERE / ON predicates are resolved through `Run`, not `GetSqlType`. The always-defer cost is one inner-plan execution per outer Execute call regardless — the same one the eager path used to do at parse time. The chained outer-type-resolver (preferring `ParserContext.OuterTypeResolver` when set, falling back to the explicit chain parameter) is threaded into the inner Parse so projection / GROUP BY references against outer columns also resolve at parse time.
+
+`JoinDriver`'s lateral branch handles both APPLY (CrossApply / OuterApply, no ON predicate) and ordinary derived tables in INNER / LEFT / CROSS join slots — the latter apply their `ON` predicate after the inner row materializes, and LEFT joins null-fill on no-match the same way OUTER APPLY does. Leftmost-source lateral plans run from the surrounding `outerResolver` directly (no per-tuple feedback at level 0).
 
 ### CASE
 Searched (`CASE WHEN cond THEN ... [ELSE ...] END`) and simple (`CASE input WHEN val ...`). Branches evaluate in order; first true predicate wins. UNKNOWN excludes (matches WHERE); simple-form `CASE NULL WHEN NULL` falls through. Result type computed via `SqlType.Promote` across all THEN/ELSE, cached on first `GetSqlType`; `Run` coerces matched values to the common type. No-match-no-ELSE → typed NULL.
@@ -270,7 +274,6 @@ Type-metadata accessors (`GetDataTypeName` / `GetFieldType`) read from `Simulate
 - Locks / MVCC / isolation levels — the simulator has no isolation; uncommitted writes are immediately visible to all readers (single-Simulation, single-thread-at-a-time assumption). `BEGIN DISTRIBUTED TRANSACTION` and `BEGIN TRANSACTION ... WITH MARK '...'` aren't parsed. `XACT_ABORT` / `SET TRANSACTION ISOLATION LEVEL`.
 - `RIGHT JOIN` (rewrite as LEFT with sources swapped); `FULL OUTER JOIN`. Both raise `NotSupportedException` at parse.
 - Comma-separated FROM (legacy ANSI-89 join syntax).
-- Plain derived tables in FROM (without APPLY) don't see outer scope — they execute eagerly at parse time. Lateral access requires `CROSS APPLY` / `OUTER APPLY`. SQL Server actually allows derived-table-sees-outer in any FROM subquery; the gap shows up in compound shapes like `(SELECT … FROM (SELECT TOP(N) … WHERE outer.col = …) AS t)` where the inner correlation isn't expressed via APPLY.
 - `ANY` / `SOME` / `ALL` quantifiers.
 - `UNION` / `UNION ALL` inside a subquery body.
 - Row-constructor `IN ((1,2), (3,4))`.
diff --git a/SqlServerSimulator.Tests.EFCore/EFCoreSubquery.cs b/SqlServerSimulator.Tests.EFCore/EFCoreSubquery.cs
@@ -121,4 +121,35 @@ public void Where_ScalarComparisonAgainstSubquery_FiltersByMaxAmount()
             .ToArray();
         CollectionAssert.AreEqual(new[] { 2 }, customerIds);
     }
+
+    [TestMethod]
+    public void Projection_DistinctCorrelatedCount_EmitsCorrelatedDerivedTable()
+    {
+        // EF Core 10 emits `(SELECT COUNT(*) FROM (SELECT DISTINCT col FROM t
+        // WHERE t.k = outer.k) AS sub)` for `Distinct().Count()` over a
+        // correlated subset — the inner derived table references the outer
+        // scope via its WHERE. Before the always-defer derived-table fix
+        // this raised "Invalid column name" because plain derived tables
+        // didn't see outer scope.
+        using var context = SeededContext();
+        var rows = context.Customers
+            .OrderBy(c => c.Id)
+            .Select(c => new
+            {
+                c.Id,
+                DistinctAmounts = context.CustomerOrders
+                    .Where(o => o.CustomerId == c.Id)
+                    .Select(o => o.Amount)
+                    .Distinct()
+                    .Count(),
+            })
+            .ToArray();
+        Assert.HasCount(3, rows);
+        // c.Id=1 has orders [10, 20] → 2 distinct amounts.
+        // c.Id=2 has order [30] → 1.
+        // c.Id=3 has none → 0.
+        Assert.AreEqual(2, rows[0].DistinctAmounts);
+        Assert.AreEqual(1, rows[1].DistinctAmounts);
+        Assert.AreEqual(0, rows[2].DistinctAmounts);
+    }
 }
diff --git a/SqlServerSimulator.Tests/CorrelatedDerivedTableTests.cs b/SqlServerSimulator.Tests/CorrelatedDerivedTableTests.cs
@@ -0,0 +1,147 @@
+using System.Data.Common;
+using static Microsoft.VisualStudio.TestTools.UnitTesting.Assert;
+
+namespace SqlServerSimulator;
+
+/// <summary>
+/// Tests for derived tables in FROM that reference outer-scope columns —
+/// SQL Server's "any FROM derived table can correlate" rule, not just the
+/// CROSS APPLY / OUTER APPLY shape. The simulator threads the outer-type
+/// resolver into the inner Parse and defers execution per Selection
+/// instance so the inner plan re-runs with each call's outer resolver.
+/// </summary>
+[TestClass]
+public sealed class CorrelatedDerivedTableTests
+{
+    private static DbConnection SeededSales()
+    {
+        var connection = new Simulation().CreateOpenConnection();
+        _ = connection.CreateCommand(
+            "create table sales (region varchar(10), salesperson varchar(20), amount decimal(10,2))").ExecuteNonQuery();
+        _ = connection.CreateCommand(
+            "insert into sales values " +
+            "('east', 'alice',   100), " +
+            "('east', 'alice',   200), " +
+            "('east', 'bob',     150), " +
+            "('west', 'carol',   300), " +
+            "('west', 'carol',   500), " +
+            "('west', 'dan',      50)").ExecuteNonQuery();
+        return connection;
+    }
+
+    [TestMethod]
+    public void DistinctCorrelatedCount_TheEFShape_Works()
+    {
+        // The exact shape EF Core 10 emits for `db.X.Select(s => new {
+        // DistinctCount = db.Y.Where(...).Distinct().Count() })`. Before
+        // this fix, the simulator threw "Invalid column name 's.Region'"
+        // because the inner derived table couldn't see the outer scope.
+        using var connection = SeededSales();
+        using var reader = connection.CreateCommand(
+            "select s.region, (select count(*) from (select distinct s0.salesperson from sales as s0 where s0.region = s.region) as s1) from sales as s").ExecuteReader();
+        var counts = new Dictionary<string, int>();
+        while (reader.Read())
+            counts[reader.GetString(0)] = reader.GetInt32(1);
+        AreEqual(2, counts["east"]);  // alice, bob
+        AreEqual(2, counts["west"]);  // carol, dan
+    }
+
+    [TestMethod]
+    public void CorrelatedDerivedTableInWhereExists()
+    {
+        // `where exists (select 1 from (correlated derived) as sub where ...)`
+        using var connection = SeededSales();
+        using var reader = connection.CreateCommand(
+            "select distinct s.region from sales as s where exists (" +
+            "  select 1 from (select distinct x.salesperson from sales as x where x.region = s.region) as sub" +
+            "  where sub.salesperson = 'alice')").ExecuteReader();
+        var regions = new List<string>();
+        while (reader.Read())
+            regions.Add(reader.GetString(0));
+        // Only east has 'alice'; west doesn't.
+        HasCount(1, regions);
+        AreEqual("east", regions[0]);
+    }
+
+    [TestMethod]
+    public void NonCorrelatedDerivedTable_StillWorks()
+    {
+        // Regression — an uncorrelated derived table (no outer reference)
+        // should still produce its own row stream. Always-defer changes
+        // the runtime path but the result must be identical.
+        using var connection = SeededSales();
+        using var reader = connection.CreateCommand(
+            "select region, total from (select region, sum(amount) as total from sales group by region) as g order by region").ExecuteReader();
+        var pairs = new List<(string Region, decimal Total)>();
+        while (reader.Read())
+            pairs.Add((reader.GetString(0), reader.GetDecimal(1)));
+        HasCount(2, pairs);
+        AreEqual(("east", 450.00m), pairs[0]);
+        AreEqual(("west", 850.00m), pairs[1]);
+    }
+
+    [TestMethod]
+    public void NonCorrelatedDerivedTable_JoinedToHeapTable()
+    {
+        // Pre-existing JOIN-with-derived-table shape. After the always-defer
+        // change the JoinDriver lateral branch handles the ON predicate
+        // and INNER semantics for these too.
+        using var connection = SeededSales();
+        using var reader = connection.CreateCommand(
+            "select s.region, s.salesperson, t.total " +
+            "from sales as s inner join (select region, sum(amount) as total from sales group by region) as t " +
+            "  on s.region = t.region " +
+            "order by s.region, s.salesperson, s.amount").ExecuteReader();
+        var rows = new List<(string Region, string Person, decimal Total)>();
+        while (reader.Read())
+            rows.Add((reader.GetString(0), reader.GetString(1), reader.GetDecimal(2)));
+        HasCount(6, rows);
+        // Each detail row carries its region's broadcast total.
+        IsTrue(rows.Where(r => r.Region == "east").All(r => r.Total == 450.00m));
+        IsTrue(rows.Where(r => r.Region == "west").All(r => r.Total == 850.00m));
+    }
+
+    [TestMethod]
+    public void LeftJoin_DerivedTable_NullFillsNoMatch()
+    {
+        // LEFT JOIN to a derived table with an ON predicate that fails
+        // for some outer rows must null-fill, matching real SQL Server.
+        using var connection = SeededSales();
+        _ = connection.CreateCommand(
+            "create table regions (region varchar(10), label varchar(20))").ExecuteNonQuery();
+        _ = connection.CreateCommand(
+            "insert into regions values ('east', 'East'), ('north', 'North')").ExecuteNonQuery();
+        using var reader = connection.CreateCommand(
+            "select r.region, r.label, t.total " +
+            "from regions as r left join (select region, sum(amount) as total from sales group by region) as t " +
+            "  on r.region = t.region " +
+            "order by r.region").ExecuteReader();
+        var rows = new List<(string Region, string Label, decimal? Total)>();
+        while (reader.Read())
+            rows.Add((reader.GetString(0), reader.GetString(1), reader.IsDBNull(2) ? null : reader.GetDecimal(2)));
+        HasCount(2, rows);
+        AreEqual("east", rows[0].Region);
+        AreEqual(450.00m, rows[0].Total);
+        // 'north' has no matching sales rows → null-filled total.
+        AreEqual("north", rows[1].Region);
+        IsNull(rows[1].Total);
+    }
+
+    [TestMethod]
+    public void CorrelatedDerivedTable_AsScalarSubqueryFromSource()
+    {
+        // Scalar subquery whose own FROM is a correlated derived table —
+        // the EF distinct-count shape with TOP rather than COUNT.
+        using var connection = SeededSales();
+        using var reader = connection.CreateCommand(
+            "select s.region, (select top 1 sub.salesperson from (select distinct x.salesperson from sales as x where x.region = s.region) as sub order by sub.salesperson) " +
+            "from sales as s where s.salesperson = 'alice'").ExecuteReader();
+        // Both alice rows are in 'east'; the inner returns the first
+        // distinct salesperson alphabetically in 'east' = 'alice'.
+        var rows = new List<(string Region, string FirstSp)>();
+        while (reader.Read())
+            rows.Add((reader.GetString(0), reader.GetString(1)));
+        HasCount(2, rows);
+        IsTrue(rows.All(r => r.Region == "east" && r.FirstSp == "alice"));
+    }
+}
diff --git a/SqlServerSimulator/Parser/Selection.Execution.cs b/SqlServerSimulator/Parser/Selection.Execution.cs
@@ -912,7 +912,16 @@ private static SqlValue ResolveAcrossTuple(
 
         if (joins.Length == 0)
         {
-            foreach (var row in sources[0].Rows)
+            // Single FROM source. A correlated derived table here carries a
+            // LateralPlan whose execution depends on the enclosing scope's
+            // resolver (no per-row tuple to feed yet — we're at the outermost
+            // level of this Selection). Run it once with the outer resolver
+            // and stream its rows; eager (non-correlated) sources iterate
+            // their pre-evaluated row bytes the same way as before.
+            var rows = sources[0].LateralPlan is { } leftmostLateralPlan
+                ? leftmostLateralPlan.Execute(outerResolver).RowBytes
+                : sources[0].Rows;
+            foreach (var row in rows)
             {
                 tuple[0] = row;
                 yield return tuple;
@@ -923,6 +932,22 @@ private static SqlValue ResolveAcrossTuple(
         SqlValue Resolve(byte[]?[] currentTuple, MultiPartName name) =>
             ResolveAcrossTuple(sources, currentTuple, name, outerResolver, n => Resolve(currentTuple, n));
 
+        // Leftmost-source lateral plan in a multi-source FROM: same as the
+        // joins.Length==0 case — drive it from the outer resolver, then let
+        // the join driver chain through the remaining sources. The level==0
+        // branch in JoinDriver handles non-lateral leftmost sources; we
+        // pre-feed the lateral row stream via a temporary FromSource swap.
+        if (sources[0].LateralPlan is { } leadLateralPlan)
+        {
+            foreach (var row in leadLateralPlan.Execute(outerResolver).RowBytes)
+            {
+                tuple[0] = row;
+                foreach (var t in JoinDriver(sources, joins, tuple, Resolve, level: 1))
+                    yield return t;
+            }
+            yield break;
+        }
+
         foreach (var t in JoinDriver(sources, joins, tuple, Resolve, level: 0))
             yield return t;
     }
@@ -965,21 +990,27 @@ SqlValue Resolve(byte[]?[] currentTuple, MultiPartName name) =>
         var join = joins[level - 1];
         var matched = false;
 
-        // Lateral source (right side of CROSS APPLY / OUTER APPLY): the
-        // plan re-executes per outer tuple, and its result rows are this
-        // level's contribution to the join. No ON predicate — correlation
-        // lives inside the lateral plan's own WHERE clause.
+        // Lateral source: any source backed by a deferred plan. APPLY brings
+        // its own join kind (CrossApply / OuterApply, no ON predicate) and
+        // correlates via the inner WHERE; ordinary derived tables in INNER /
+        // LEFT / CROSS JOIN slots also flow here since
+        // <see cref="ParseSingleFromSource"/> always defers (correlation isn't
+        // statically detectable). Apply the ON predicate when the join has
+        // one, and null-fill for both LEFT and OUTER APPLY when nothing
+        // matched.
         if (sources[level].LateralPlan is { } lateralPlan)
         {
             foreach (var row in lateralPlan.Execute(name => resolve(tuple, name)).RowBytes)
             {
                 tuple[level] = row;
+                if (join.OnPredicate is not null && join.OnPredicate.Run(name => resolve(tuple, name)) != true)
+                    continue;
                 matched = true;
                 foreach (var t in JoinDriver(sources, joins, tuple, resolve, level + 1))
                     yield return t;
             }
             tuple[level] = null;
-            if (!matched && join.Kind == JoinKind.OuterApply)
+            if (!matched && join.Kind is JoinKind.OuterApply or JoinKind.Left)
             {
                 foreach (var t in JoinDriver(sources, joins, tuple, resolve, level + 1))
                     yield return t;
diff --git a/SqlServerSimulator/Parser/Selection.cs b/SqlServerSimulator/Parser/Selection.cs
