Skip to content

Commit 730d055

Browse files
committed
WWI-Full Phase 1 — partitioning + columnstore as silent skips, Latin1_General_CI_AS joins the recognized-collation whitelist. WideWorldImporters-Full (Microsoft's larger sibling sample to WWI-Standard; same 48 tables + 4.7M rows but with partitioning, columnstore indexes, and one natively-compiled procedure) now loads end-to-end with a single remaining Skipped element (the natively-compiled procedure, deferred to a follow-up bundle). The .vs/WideWorldImporters-Full.bacpac file is gitignored alongside the other two reference bacpacs.
**Loader dispatcher** (`Storage/Bacpac/ModelXmlReader.cs`): - `SqlPartitionFunction`, `SqlPartitionScheme`, `SqlColumnStoreIndex` join `SqlFilegroup` as phase-1 silent-skip entries. All four are storage-organization decorations with no semantic effect on row-store query results — `PartitionFunction`/`PartitionScheme` define filegroup-mapping boundaries that tables/indexes reference for physical placement, and `ColumnStoreIndex` is a read-optimization shape over the same row data (the simulator's linear-scan secondary indexes don't model column-major vs row-major storage). Surfacing any of them on `Skipped` would create category noise for features whose absence has zero observable effect on query results. - `IsHandledByAnotherPhase` accept-list extended with the three new types so the dispatcher's "last phase records to Skipped" guard doesn't double-record them on later passes. **Collation whitelist** (`Collation.cs`): - `Latin1_General_CI_AS` (the older 80-collation, sibling to the previously-recognized `Latin1_General_100_CI_AS`) added to `Collation.Recognized` with the standard `sys.fn_helpcollations` description. WWI-Full's `Warehouse.VehicleTemperatures` table declares two columns (`VehicleRegistration`, `FullSensorData`) with this collation — previously these surfaced as warnings with the `COLLATE` clause dropped; now they round-trip cleanly through `sys.columns.collation_name`. Same metadata-only contract as the existing entries: comparison/sort/LIKE still route through the default collation. **Diagnostic-probe finding** (worth recording for the next bundle's design phase): - WWI-Full and WWI-Standard carry essentially identical row data — the size difference (61MB vs 59MB compressed) is metadata, not workload. So WWI-Full is *not* the larger-workload memory/performance test we anticipated; it's a structural-coverage test. A genuine perf/memory test would need a different sample (or a synthetic one). - After this bundle, WWI-Full's `Skipped` list contains exactly one element: `Website.RecordColdRoomTemperatures`, a natively-compiled procedure. Its body decoration shape is `WITH NATIVE_COMPILATION = ON, SCHEMABINDING, EXECUTE AS OWNER AS BEGIN ATOMIC WITH (TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE = N'us_english') <body> END`. Unblocking it needs both (i) the `WITH NATIVE_COMPILATION = ON` clause in the procedure-header parser (parse-and-discard since the simulator doesn't natively-compile anything), and (ii) the `BEGIN ATOMIC WITH (...)` block which currently raises `NotSupportedException` at dispatch. **Tests**: - `SqlServerSimulator.Tests.Internal/Storage/BacpacLoaderTests.cs`: new `LoadWideWorldImportersFull` helper mirroring the WWI-Standard one, plus 3 new test methods — `Load_WWIFull_Loads_All_Tables` (48 tables land), `Load_WWIFull_Known_Gaps_Recorded_In_Skipped` (asserts the partition/columnstore categories don't appear on Skipped AND that exactly 1 SqlProcedure is on Skipped — locks the current census), `Load_WWIFull_Latin1_CI_AS_Columns_Warning_Free` (no `Latin1_General_CI_AS` warnings + the collation lands on `sys.columns.collation_name`). - `SqlServerSimulator.Tests/CollationMetadataTests.cs`: new public-API smoke test `Column_Collate_Latin1_General_CI_AS_RoundTrips`; existing `FnHelpCollations_ListsRecognized` count 2 → 3. **Doc updates**: - `docs/claude/bacpac-prerequisites.md`: new step 12 captures the WWI-Full Phase 1 bundle; new step 13 documents the natively-compiled-procedure body decoration as the sole remaining `Skipped` category; the WWI-discovered gap inventory in step 16 (renumbered from 14) strikes through the partition/columnstore additions. - `CLAUDE.md`: BACPAC pointer entry extended — WWI-Full now mentioned alongside AW2025 + WWI-Standard with the partition/columnstore silent-skip note.
1 parent febeae6 commit 730d055

6 files changed

Lines changed: 122 additions & 8 deletions

File tree

CLAUDE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -164,7 +164,7 @@ Per-feature deep-dives live under `docs/claude/`. Each entry below is a trigger:
164164
- **Touching `XmlSqlType` (singleton in `Storage/XmlType.cs`, SystemTypeId=241, IsLob=true), `XmlSchemaCollection` storage class + `Schema.XmlSchemaCollections` dict + `Database.AllocateXmlCollectionId` (seeds at 65536 per probe), `HeapColumn.XmlSchemaCollection` per-column binding, the `xml`-recognizing arms of `SqlType.ResolveSimpleKeyword` / `Precedence` / `SystemTypeId`, the `xml(name)` / `xml(CONTENT name)` / `xml(DOCUMENT name)` column-type parsing in `ParseOneColumnIntoLists` (via `PeekIsXmlSchemaArgument` / `ParseXmlSchemaCollectionArgument`), `Simulation.Xml.cs` partial (CREATE XML SCHEMA COLLECTION / DROP XML SCHEMA COLLECTION / CREATE [PRIMARY] XML INDEX / FOR PATH|VALUE|PROPERTY secondary form), the `Xml` contextual keyword + `Keyword.Primary` dispatch in `TryParseCreate`, `XmlIndex` / `XmlSecondaryIndexType` storage on `HeapTable.XmlIndexes`, the `XmlMethodCall` expression in `Parser/Expressions/XmlMethodCall.cs` (closed accept-list of `value` / `nodes` / `query` / `exist` / `modify`, parses cleanly, throws `NotSupportedException` at `Run`), or the `sys.xml_schema_collections` / `sys.xml_indexes` catalog views** → [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md) (the "`xml` data type + XML schema collections + XML methods + XML indexes" section carries the implementation detail).
165165
- **Touching `GeographySqlType` / `GeometrySqlType` (singletons in `Storage/SpatialType.cs`, both inherit `SpatialSqlType : SqlType(SqlTypeCategory.String)`, SystemTypeId=240, UserTypeId=130/129, IsLob=true), `SqlValue.FromGeography` / `FromGeometry` factories + the spatial branches in `FromString`, the `geography` / `geometry`-recognizing arms of `SqlType.ResolveSimpleKeyword` / `Precedence` / `SystemTypeId` / `UserTypeId`, the `SpatialMethodCall` expression in `Parser/Expressions/SpatialMethodCall.cs` (broad ~70-name accept-list of OGC + Microsoft-extension methods; parses cleanly + throws `NotSupportedException` at `Run` except `.ToString()` returns the stored WKT), the `SpatialStaticCall` expression in `Parser/Expressions/SpatialStaticCall.cs` (`geography::Parse(wkt)` / `geography::STGeomFromText(wkt, srid)` / `geometry::Point(x, y, srid)` work; other static methods throw), the `::` dispatch for `geography`/`geometry` in `Expression.cs` alongside `hierarchyid::`, the `.ToString()` polymorphic delegation in `HierarchyIdMethodCall.Run` (spatial receivers return WKT through the spatial path), the `SpatialIndex` storage class + `SpatialIndexKind` enum on `HeapTable.SpatialIndexes`, the `Simulation.Spatial.cs` partial (CREATE SPATIAL INDEX with USING / BOUNDING_BOX / GRIDS / CELLS_PER_OBJECT parsing + default tessellation_scheme), the `Spatial` contextual keyword routing in `TryParseCreate`, or the `sys.spatial_indexes` (23-col probe-confirmed) / `sys.spatial_index_tessellations` (16-col probe-confirmed) / `sys.spatial_reference_systems` (6-col, empty seed) catalog views** → [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md) (the "`geography` / `geometry` data types" section carries the implementation detail).
166166
- **Adding a new top-level statement parser or changing the dispatch loop's statement-separator rules**[`docs/claude/grammar.md`](docs/claude/grammar.md) + [`docs/claude/control-flow.md`](docs/claude/control-flow.md).
167-
- **Working on BACPAC import (`Simulation.FromBacpac(string, out BacpacLoadResult)` + Stream overload, internal until externally consumed) — the model.xml dispatcher in `SqlServerSimulator/Storage/Bacpac/ModelXmlReader.cs` (9 phases: DB options + schemas + UDDTs + **sequences + roles + table types + filegroups (skip)** → tables → constraints → FKs → (unused) → views → programmable objects → **deferred computed columns + indexes** → extended properties → BCP data), the `BacpacReader` + `BcpRowReader` + `HierarchyIdWireDecoder` data path, the `BacpacLoadResult.Skipped` + `ElementCounts` + `Warnings` + `TableColumnIsAlias` diagnostics carriers, the resilient-loader contract (per-element exceptions land on Skipped with `"Load failed: …"` prefix and don't abort the load; computed-column failures use `"Deferred: …"` prefix so the AW guard stays meaningful), the **BCP-filters-computed-columns contract** (`BacpacReader.LoadRowsFromBcp` strips columns with `HeapColumn.Computed != null` before passing to `BcpRowReader`/`RowEncoder` — DACFx-emitted BCP files exclude computed columns from the wire layout regardless of PERSISTED; the loader emits computed columns *without* the PERSISTED qualifier so the simulator recomputes on every read), the wire-format matrix (fixed-raw / 1-byte-prefix / 2-byte-prefix / 8-byte-prefix tiers — see [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md) "BCP wire format" section)** → [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md). Loader baseline ships (2026-05-15) with AdventureWorks2025 at **100% row coverage** (760,167 / 760,167; zero BCP-file failures); WideWorldImporters-Standard loads end-to-end with **48/48 tables + 26/26 sequences + 9/9 roles + 4/4 table types + 8/8 computed columns + 42/42 procedures + 94/94 indexes + 3/3 views + 1/1 scalar function + 4.7M rows**, zero remaining Skipped categories — every WWI element type loads (collation metadata stored as declaration via `Database.CollationName`, though comparison semantics still route through the default collation per `Collation.Default`). MAX types (varchar(MAX) / nvarchar(MAX) / varbinary(MAX)) + xml + CLR-UDT family all use the simple inline 8-byte-prefix wire shape (NOT TDS-PLP chunked form); plain `bit` uses 1-byte length prefix regardless of nullability (probe-confirmed via Production.Document.FolderFlag, the sole non-UDDT bit column in AW). Hierarchyid wire decoder covers AW's [0..79] positive-ordinal envelope via a 4-prefix order-preserving prefix code (`01` / `100` / `101` / `110`); negative ordinals + ordinals ≥ 80 + dotted sub-ordinals raise `NotSupportedException` for a follow-up bundle to extend. Geography rows load with `SpatialLocation = NULL` (WKB→WKT decoding deferred).
167+
- **Working on BACPAC import (`Simulation.FromBacpac(string, out BacpacLoadResult)` + Stream overload, internal until externally consumed) — the model.xml dispatcher in `SqlServerSimulator/Storage/Bacpac/ModelXmlReader.cs` (9 phases: DB options + schemas + UDDTs + **sequences + roles + table types + filegroups (skip)** → tables → constraints → FKs → (unused) → views → programmable objects → **deferred computed columns + indexes** → extended properties → BCP data), the `BacpacReader` + `BcpRowReader` + `HierarchyIdWireDecoder` data path, the `BacpacLoadResult.Skipped` + `ElementCounts` + `Warnings` + `TableColumnIsAlias` diagnostics carriers, the resilient-loader contract (per-element exceptions land on Skipped with `"Load failed: …"` prefix and don't abort the load; computed-column failures use `"Deferred: …"` prefix so the AW guard stays meaningful), the **BCP-filters-computed-columns contract** (`BacpacReader.LoadRowsFromBcp` strips columns with `HeapColumn.Computed != null` before passing to `BcpRowReader`/`RowEncoder` — DACFx-emitted BCP files exclude computed columns from the wire layout regardless of PERSISTED; the loader emits computed columns *without* the PERSISTED qualifier so the simulator recomputes on every read), the wire-format matrix (fixed-raw / 1-byte-prefix / 2-byte-prefix / 8-byte-prefix tiers — see [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md) "BCP wire format" section)** → [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md). Loader baseline ships (2026-05-15) with AdventureWorks2025 at **100% row coverage** (760,167 / 760,167; zero BCP-file failures); WideWorldImporters-Standard loads end-to-end with **48/48 tables + 26/26 sequences + 9/9 roles + 4/4 table types + 8/8 computed columns + 42/42 procedures + 94/94 indexes + 3/3 views + 1/1 scalar function + 4.7M rows**, zero remaining Skipped categories — every WWI element type loads (collation metadata stored as declaration via `Database.CollationName`, though comparison semantics still route through the default collation per `Collation.Default`). **WideWorldImporters-Full** (2026-05-16 probe) adds partitioning + columnstore + a natively-compiled procedure onto the same row volume; the loader survives with `SqlPartitionFunction` / `SqlPartitionScheme` / `SqlColumnStoreIndex` as silent skips (storage-organization decorations with no semantic effect on row-store queries, matching `SqlFilegroup`'s pattern). The only remaining Skipped category is one natively-compiled procedure (`Website.RecordColdRoomTemperatures`) whose `WITH NATIVE_COMPILATION = ON … BEGIN ATOMIC` body decoration isn't parsed. MAX types (varchar(MAX) / nvarchar(MAX) / varbinary(MAX)) + xml + CLR-UDT family all use the simple inline 8-byte-prefix wire shape (NOT TDS-PLP chunked form); plain `bit` uses 1-byte length prefix regardless of nullability (probe-confirmed via Production.Document.FolderFlag, the sole non-UDDT bit column in AW). Hierarchyid wire decoder covers AW's [0..79] positive-ordinal envelope via a 4-prefix order-preserving prefix code (`01` / `100` / `101` / `110`); negative ordinals + ordinals ≥ 80 + dotted sub-ordinals raise `NotSupportedException` for a follow-up bundle to extend. Geography rows load with `SpatialLocation = NULL` (WKB→WKT decoding deferred).
168168

169169
## Not modeled
170170

SqlServerSimulator.Tests.Internal/Storage/BacpacLoaderTests.cs

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,16 @@ private static Simulation LoadWideWorldImporters(out BacpacLoadResult diagnostic
4848
return Simulation.FromBacpac(path, out diagnostics);
4949
}
5050

51+
private static Simulation LoadWideWorldImportersFull(out BacpacLoadResult diagnostics)
52+
{
53+
var path = ResolveBacpacPath("WideWorldImporters-Full.bacpac");
54+
if (string.IsNullOrEmpty(path))
55+
{
56+
Inconclusive(".vs/WideWorldImporters-Full.bacpac not present in this workspace; skipping WWI-Full smoke test.");
57+
}
58+
return Simulation.FromBacpac(path, out diagnostics);
59+
}
60+
5161
[TestMethod]
5262
public void Load_AW_Creates_All_Five_Schemas()
5363
{
@@ -860,4 +870,81 @@ public void Load_WWI_Most_Tables_Loaded()
860870
// level).
861871
AreEqual(48, reader.GetInt32(0));
862872
}
873+
874+
[TestMethod]
875+
public void Load_WWIFull_Loads_All_Tables()
876+
{
877+
// WWI-Full has the same 48 tables as Standard but adds partitioning,
878+
// columnstore, and one native-compiled SP. The partition-aware /
879+
// columnstore-aware decoration elements (SqlPartitionFunction,
880+
// SqlPartitionScheme, SqlColumnStoreIndex) are skip-with-diagnostic;
881+
// the native-compiled SP failure is independently tracked.
882+
var simulation = LoadWideWorldImportersFull(out _);
883+
using var connection = (SimulatedDbConnection)simulation.CreateDbConnection();
884+
connection.Open();
885+
using var command = connection.CreateCommand();
886+
command.CommandText = "SELECT COUNT(*) FROM sys.tables;";
887+
using var reader = command.ExecuteReader();
888+
IsTrue(reader.Read());
889+
AreEqual(48, reader.GetInt32(0));
890+
}
891+
892+
[TestMethod]
893+
public void Load_WWIFull_Known_Gaps_Recorded_In_Skipped()
894+
{
895+
_ = LoadWideWorldImportersFull(out var diagnostics);
896+
var grouped = diagnostics.Skipped.GroupBy(s => s.ElementType)
897+
.ToDictionary(g => g.Key, g => g.Count());
898+
TestContext.WriteLine("=== WWI-Full Skipped (current) ===");
899+
foreach (var kv in grouped.OrderByDescending(k => k.Value))
900+
TestContext.WriteLine($"{kv.Value,5} {kv.Key}");
901+
902+
// Storage-layout / read-optimization decorations are silent skips,
903+
// not Skipped entries (matching SqlFilegroup's pattern). Surfacing
904+
// them on Skipped would create category noise for features whose
905+
// absence has zero semantic effect on query results.
906+
IsFalse(grouped.ContainsKey("SqlColumnStoreIndex"), "Columnstore indexes skip-with-diagnostic; no semantic effect on row-store query results.");
907+
IsFalse(grouped.ContainsKey("SqlPartitionFunction"), "Partition functions are filegroup-mapping metadata; skip-with-diagnostic.");
908+
IsFalse(grouped.ContainsKey("SqlPartitionScheme"), "Partition schemes are filegroup-mapping metadata; skip-with-diagnostic.");
909+
910+
// NATIVE_COMPILATION procedure body is the sole remaining loader
911+
// gap for WWI-Full (1 procedure: Website.RecordColdRoomTemperatures).
912+
// The WITH NATIVE_COMPILATION = ON ... BEGIN ATOMIC body decoration
913+
// isn't parsed; the procedure body falls onto Skipped with a
914+
// SimulatedSqlException parse failure.
915+
IsTrue(grouped.TryGetValue("SqlProcedure", out var procSkipped));
916+
AreEqual(1, procSkipped);
917+
}
918+
919+
[TestMethod]
920+
public void Load_WWIFull_Latin1_CI_AS_Columns_Warning_Free()
921+
{
922+
// WWI-Full's Warehouse.VehicleTemperatures table declares two columns
923+
// (VehicleRegistration, FullSensorData) with COLLATE Latin1_General_CI_AS.
924+
// Before the whitelist entry, this produced 2 Warnings + dropped clauses;
925+
// the whitelist entry recognizes the name and the COLLATE clauses land.
926+
var simulation = LoadWideWorldImportersFull(out var diagnostics);
927+
var collationWarnings = diagnostics.Warnings
928+
.Where(w => w.Contains("Latin1_General_CI_AS", StringComparison.Ordinal))
929+
.ToList();
930+
IsEmpty(collationWarnings);
931+
932+
// Confirm round-trip via sys.columns.
933+
using var connection = (SimulatedDbConnection)simulation.CreateDbConnection();
934+
connection.Open();
935+
using var command = connection.CreateCommand();
936+
command.CommandText = """
937+
SELECT c.collation_name
938+
FROM sys.columns c
939+
JOIN sys.tables t ON c.object_id = t.object_id
940+
JOIN sys.schemas s ON t.schema_id = s.schema_id
941+
WHERE s.name = 'Warehouse'
942+
AND t.name = 'VehicleTemperatures'
943+
AND c.name = 'VehicleRegistration';
944+
""";
945+
using var reader = command.ExecuteReader();
946+
IsTrue(reader.Read());
947+
AreEqual("Latin1_General_CI_AS", reader.GetString(0));
948+
}
949+
863950
}

SqlServerSimulator.Tests/CollationMetadataTests.cs

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,9 +124,18 @@ public void Column_Collate_UnrecognizedName_RaisesNotSupported()
124124

125125
[TestMethod]
126126
public void FnHelpCollations_ListsRecognized()
127-
=> AreEqual(2, new Simulation().ExecuteScalar(
127+
=> AreEqual(3, new Simulation().ExecuteScalar(
128128
"SELECT COUNT(*) FROM sys.fn_helpcollations()"));
129129

130+
[TestMethod]
131+
public void Column_Collate_Latin1_General_CI_AS_RoundTrips()
132+
{
133+
var sim = new Simulation();
134+
_ = sim.ExecuteNonQuery("CREATE TABLE t (b nvarchar(50) COLLATE Latin1_General_CI_AS)");
135+
AreEqual("Latin1_General_CI_AS", sim.ExecuteScalar(
136+
"SELECT collation_name FROM sys.columns WHERE name = 'b'"));
137+
}
138+
130139
[TestMethod]
131140
public void FnHelpCollations_ColumnsAreNameAndDescription()
132141
{

SqlServerSimulator/Collation.cs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ private protected Collation()
3434
{
3535
["SQL_Latin1_General_CP1_CI_AS"] = "Latin1-General, case-insensitive, accent-sensitive, kanatype-insensitive, width-insensitive for Unicode Data, SQL Server Sort Order 52 on Code Page 1252 for non-Unicode Data",
3636
["Latin1_General_100_CI_AS"] = "Latin1-General-100, case-insensitive, accent-sensitive, kanatype-insensitive, width-insensitive",
37+
["Latin1_General_CI_AS"] = "Latin1-General, case-insensitive, accent-sensitive, kanatype-insensitive, width-insensitive",
3738
}.ToFrozenDictionary(StringComparer.OrdinalIgnoreCase);
3839

3940
/// <summary>

SqlServerSimulator/Storage/Bacpac/ModelXmlReader.cs

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -129,10 +129,20 @@ private static void RunPhase(List<XElement> elements, DbConnection connection, B
129129
// so the per-SqlTable computed-column pass completes
130130
// before the first SqlIndex emission runs.
131131
("SqlIndex", 8) => Run(() => EmitIndex(element, name, connection, viewNames, result)),
132-
// Filegroups are storage-layout metadata; the simulator
133-
// has a single in-process heap so they're parse-and-skip.
132+
// Filegroups + partitioning + columnstore are storage-layout
133+
// concerns; the simulator has a single in-process heap so
134+
// all four are parse-and-skip. PartitionFunction /
135+
// PartitionScheme define filegroup-mapping boundaries that
136+
// tables / indexes reference for physical placement (no
137+
// semantic effect when filegroups themselves are skipped).
138+
// ColumnStoreIndex is a read-optimization shape over the
139+
// same row data; the simulator's linear-scan secondary
140+
// indexes don't model column-major vs row-major storage.
134141
// Phase 1 placement is fine — no dependencies either way.
135142
("SqlFilegroup", 1) => Run(static () => { }),
143+
("SqlPartitionFunction", 1) => Run(static () => { }),
144+
("SqlPartitionScheme", 1) => Run(static () => { }),
145+
("SqlColumnStoreIndex", 1) => Run(static () => { }),
136146
("SqlExtendedProperty", 9) => Run(() => EmitExtendedProperty(element, name, connection, viewNames, result)),
137147
// Permission statements emit after roles + everything-securable
138148
// — phase 7 is "everything is in place except extended
@@ -174,6 +184,7 @@ private static bool Run(Action action)
174184
private static bool IsHandledByAnotherPhase(string type) => type
175185
is "SqlDatabaseOptions" or "SqlSchema" or "SqlUserDefinedDataType"
176186
or "SqlSequence" or "SqlRole" or "SqlTableType" or "SqlFilegroup"
187+
or "SqlPartitionFunction" or "SqlPartitionScheme" or "SqlColumnStoreIndex"
177188
or "SqlTable"
178189
or "SqlPrimaryKeyConstraint" or "SqlUniqueConstraint"
179190
or "SqlCheckConstraint" or "SqlDefaultConstraint"

0 commit comments

Comments
 (0)