Skip to content

Commit b550a7c

Browse files
committed
BACPAC loader Phases F+G — extended properties + BCP row data. With this bundle, AdventureWorks loads end-to-end through a single Simulation.FromBacpac call: schema + constraints + indexes + views + functions + procedures + triggers + extended properties (Phase F), plus 718K of 760K rows of actual table data (Phase G). Remaining row gap is feature-gated by four wire-format decoders (hierarchyid / geography / xml / MAX types) that the loader cleanly defers to Skipped rather than aborting.
**Phase F — extended properties** in `EmitExtendedProperty`. DACFx encodes the host kind in the element name's leading bracketed segment (`[SqlColumn]` / `[SqlTableBase]` / `[SqlSchema]` / `[SqlDatabaseOptions]` / `[SqlDatabaseDdlTrigger]` / `[SqlFilegroup]`); the emitter resolves each to `sp_addextendedproperty`'s `@level0type` / `@level1type` / `@level2type` arg triplet (SCHEMA / TABLE | VIEW / COLUMN) and uses the existing `viewNames` HashSet to distinguish TABLE vs VIEW level1. Out-of-scope hosts (SqlFilegroup, SqlDatabaseDdlTrigger) plus 9 column/table-level properties whose host columns/tables didn't load (computed-col tables, vJobCandidate-family views) land on Skipped. New shared helpers `SplitBracketedSegments`, `Unbracket`, `TrySplit2Part`, `TrySplit3Part` factor out the bracketed-name parsing. SqlPermissionStatement intentionally Skipped — AW's 2 entries are Always-Encrypted permissions using DACFx integer enum codes (1107/1108) that need a separate enum→T-SQL mapping. **Phase G — BCP data loader**. New `BcpRowReader` decodes 18 base types directly into `SqlValue[]` rows; `BacpacReader.LoadRowsFromBcp` pokes each row into `HeapTable.Heap` via `RowEncoder.EncodeRow` + `Heap.Insert`, bypassing SQL parsing for the 760K-row hot path. Decoded types: int / bigint / smallint / tinyint / bit, datetime / smalldatetime / date, money / smallmoney, decimal/numeric (BigInteger-based mantissa decode for arbitrary precision), datetime2 / time / datetimeoffset, uniqueidentifier, varchar / nvarchar / nchar / char (bounded), varbinary / binary (bounded). Per-file try/catch surfaces decode failures on Skipped with the exception type + message rather than aborting the whole load. **Two real probe-confirmed wire-format quirks** (verified via hex-dump of AW's `Sales.SpecialOffer` and `HumanResources.Shift` BCP files on 2026-05-15, both corrections vs. the original prereqs-doc matrix): 1. **UDDT-aliased columns get a 1-byte length prefix even when NOT NULL**, regardless of underlying type. AW's `Sales.SalesOrderHeader.OnlineOrderFlag` (declared via `dbo.Flag` alias = `bit NOT NULL`) encodes as `01 01` (1-byte prefix = 1, then 1 byte value) rather than a single raw bit byte. Threaded through via a new `BacpacLoadResult.TableColumnIsAlias` side map populated during `EmitTable` (via the new `IsAliasTypedColumn` helper that checks `<References ExternalSource="BuiltIns" …>` vs. unmarked refs in the TypeSpecifier's Type relationship) and consumed in `BcpRowReader.TryReadRow`. The fix unblocked 33K rows in AW (685K → 718K loaded). 2. **money / smallmoney / time(N) / datetime2(N) / datetimeoffset(N) are fixed-raw NOT NULL, not always length-prefixed** as the original prereqs-doc matrix claimed. `Sales.SpecialOffer.DiscountPct` smallmoney NOT NULL = 4 raw bytes (no prefix) for value 0; `HumanResources.Shift.StartTime` time(7) NOT NULL = 5 raw bytes (no prefix). Reclassified from `ReadLengthPrefixed1Variable` to `ReadFixedRaw` with width = `TimeSqlType.timeBytes` (3/4/5 by precision) / `DateTime2SqlType.timeBytes + 3` / `DateTimeOffsetSqlType.timeBytes + 5`. NULL handling for these types uses the same `0xFF`-as-NULL 1-byte prefix as int/datetime/etc. **Loader bug fix in `TranslateSimpleColumn`**: `IsNullable` absent from DACFx means "inherit the column-type's default" (NULL for builtins per ANSI, the alias's stored IsNullable for UDDT-typed columns). Was previously defaulting to True → always emitting `NULL` in the CREATE TABLE column declaration → forcing UDDT-typed columns (where the alias defaults NOT NULL, e.g. `dbo.Flag`) to load as nullable. New `isNullableExplicit` flag tracks the presence/absence distinction; omitting the marker entirely when absent lets the simulator's resolver pick the right default through both paths. **Loader infrastructure expansion**: `BacpacLoadResult` gains the `TableColumnIsAlias: Dictionary<string, bool[]>` side map keyed by bracketed `[schema].[table]`; per-row-decode bookkeeping adds `ElementCounts["_DataFile"]` (per-shard count) and `ElementCounts["_DataRows"]` (cumulative). `BacpacReader.Load` now does a single pass over all archive entries — model.xml first (existing 7-phase dispatch), then the Data folder iteration for the row pass. New helper `ParseDataFolderName` splits `Data/<schema>.<table>/TableData-NNN-NNNNN.BCP` paths into `(schema, table)` tuples. `RunPhase`'s last-phase counter advances to 8 to cover the new extended-property dispatch. **Loader pushback stream**: new `PushbackStream` helper class inside `BcpRowReader` peeks the first byte of each row to detect EOF without throwing, then puts the byte back into the decoder pipeline. Required because `Stream.ReadByte` returns -1 at EOF but the column decoders all use `ReadExact` (which throws on premature EOF) — without the pushback the loader couldn't distinguish "end of file" from "torn row." **AW coverage**: 5/5 schemas, 71/71 tables, 6/6 UDDTs, 71/71 PKs, 1/1 UQ, 90/90 FKs, 89/89 CHECKs, 152/152 DEFAULTs, 89/95 indexes, 11/20 views, 8/10 procs, 10/11 functions, 10/10 DML triggers, 1/1 DDL trigger, 527/538 extended properties, and **718,369/760,167 rows**. Remaining deferrals (all on Skipped with type-named reasons): hierarchyid (HumanResources.Employee.OrganizationNode, Production.Document.DocumentNode), geography (Person.Address.SpatialLocation), xml (Person.Person.Demographics, HumanResources.JobCandidate.Resume), varbinary(MAX) / nvarchar(MAX) (Production.ProductPhoto's photo blobs, etc.).
1 parent 1809bce commit b550a7c

5 files changed

Lines changed: 760 additions & 16 deletions

File tree

SqlServerSimulator.Tests.Internal/Storage/BacpacLoaderTests.cs

Lines changed: 67 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -83,16 +83,18 @@ public void Load_AW_Element_Counts_Match_Probe()
8383
public void Load_AW_Unhandled_Elements_Recorded_In_Skipped()
8484
{
8585
_ = LoadAdventureWorks(out var diagnostics);
86-
// Phases A-C handled types are off Skipped; everything else still
87-
// appears there awaiting future bundles.
86+
// Phases A-F handled types are off Skipped; the few remaining bucket
87+
// types (SqlComputedColumn, SqlPermissionStatement, full-text, XML
88+
// schema collections / indexes, filegroup-hosted extended properties)
89+
// appear awaiting their own bundles.
8890
IsNotEmpty(diagnostics.Skipped);
8991
IsEmpty(diagnostics.Skipped.Where(s => s.ElementType == "SqlTable").ToList());
9092
IsEmpty(diagnostics.Skipped.Where(s => s.ElementType == "SqlPrimaryKeyConstraint").ToList());
9193
IsEmpty(diagnostics.Skipped.Where(s => s.ElementType == "SqlForeignKeyConstraint").ToList());
9294
IsEmpty(diagnostics.Skipped.Where(s => s.ElementType == "SqlCheckConstraint").ToList());
9395
IsEmpty(diagnostics.Skipped.Where(s => s.ElementType == "SqlDefaultConstraint").ToList());
94-
IsNotEmpty(diagnostics.Skipped.Where(s => s.ElementType == "SqlExtendedProperty").ToList());
9596
IsNotEmpty(diagnostics.Skipped.Where(s => s.ElementType == "SqlComputedColumn").ToList());
97+
IsNotEmpty(diagnostics.Skipped.Where(s => s.ElementType == "SqlPermissionStatement").ToList());
9698
}
9799

98100
[TestMethod]
@@ -253,6 +255,68 @@ private static int QueryCount(SimulatedDbConnection connection, string sql)
253255

254256
public TestContext TestContext { get; set; } = null!;
255257

258+
[TestMethod]
259+
public void Load_AW_Bcp_Data_Loads()
260+
{
261+
var simulation = LoadAdventureWorks(out var diagnostics);
262+
// AW carries 760,167 rows across 1103 BCP shards. Phase G's
263+
// first-cut decoder lands ~718K (94%); the remaining ~42K live in
264+
// tables with xml / hierarchyid / geography / varbinary(MAX)
265+
// columns whose wire-format decoders are deferred to a follow-up
266+
// bundle. Each unloaded shard surfaces on Skipped with the type
267+
// that's blocking it.
268+
var rowsLoaded = diagnostics.ElementCounts.GetValueOrDefault("_DataRows", 0);
269+
IsGreaterThanOrEqualTo(700_000, rowsLoaded, $"expected >= 700K rows, got {rowsLoaded}");
270+
271+
// Production.ProductCategory's 4 rows must land (no exotic types).
272+
using var connection = (SimulatedDbConnection)simulation.CreateDbConnection();
273+
connection.Open();
274+
AreEqual(4, QueryCount(connection, "SELECT COUNT(*) FROM Production.ProductCategory;"));
275+
276+
// Sales.SpecialOffer's 16 rows must land (exercises smallmoney +
277+
// nullable int + uniqueidentifier + alias-typed dbo.Flag).
278+
AreEqual(16, QueryCount(connection, "SELECT COUNT(*) FROM Sales.SpecialOffer;"));
279+
280+
// The Skipped data files all name their blocking type or feature.
281+
var dataFileFailures = diagnostics.Skipped.Where(s => s.ElementType == "_DataFile").ToList();
282+
foreach (var entry in dataFileFailures)
283+
{
284+
IsTrue(entry.Reason.Contains("hierarchyid", StringComparison.OrdinalIgnoreCase)
285+
|| entry.Reason.Contains("geography", StringComparison.OrdinalIgnoreCase)
286+
|| entry.Reason.Contains("xml", StringComparison.OrdinalIgnoreCase)
287+
|| entry.Reason.Contains("MAX", StringComparison.OrdinalIgnoreCase),
288+
$"unexpected BCP failure: {entry.ElementName}: {entry.Reason}");
289+
}
290+
}
291+
292+
[TestMethod]
293+
public void Load_AW_Extended_Properties_Land()
294+
{
295+
var simulation = LoadAdventureWorks(out var diagnostics);
296+
using var connection = (SimulatedDbConnection)simulation.CreateDbConnection();
297+
connection.Open();
298+
// AW has 538 SqlExtendedProperty elements: 461 column-level + 69
299+
// table-level + 5 schema + 1 DB + 1 filegroup + 1 DDL-trigger. The
300+
// loader handles SqlColumn / SqlTableBase / SqlSchema /
301+
// SqlDatabaseOptions hosts; SqlFilegroup + SqlDatabaseDdlTrigger
302+
// hosts are out of scope, and 9 column/table-level properties miss
303+
// because their host columns / tables didn't load (computed-col
304+
// tables, vJobCandidate-family views, etc.). 527 land in practice.
305+
var landed = QueryCount(connection, "SELECT COUNT(*) FROM sys.extended_properties;");
306+
IsGreaterThanOrEqualTo(525, landed);
307+
308+
// The 1 DB-level MS_Description should be present.
309+
using var command = connection.CreateCommand();
310+
command.CommandText = "SELECT CAST(value AS nvarchar(MAX)) FROM sys.extended_properties WHERE class = 0 AND name = 'MS_Description';";
311+
using var reader = command.ExecuteReader();
312+
IsTrue(reader.Read());
313+
AreEqual("AdventureWorks 2025 Sample OLTP Database", reader.GetString(0));
314+
315+
// Filegroup + DDL-trigger hosts land on Skipped with the expected reason.
316+
var skipped = diagnostics.Skipped.Where(s => s.ElementType == "SqlExtendedProperty").ToList();
317+
IsNotEmpty(skipped.Where(s => s.Reason.Contains("SqlFilegroup", StringComparison.Ordinal)).ToList());
318+
}
319+
256320
[TestMethod]
257321
public void Load_AW_Cascade_FK_Has_Correct_Action()
258322
{

SqlServerSimulator/Storage/Bacpac/BacpacLoadResult.cs

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,16 @@ internal sealed class BacpacLoadResult
2929

3030
/// <summary>Element counts seen during the walk, keyed by element type.</summary>
3131
internal readonly Dictionary<string, int> ElementCounts = new(StringComparer.Ordinal);
32+
33+
/// <summary>
34+
/// Per-table per-column "declared via UDDT alias" flags, keyed by
35+
/// <c>[schema].[table]</c>. UDDT-typed columns get a 1-byte length prefix
36+
/// in BCP wire format even when NOT NULL, regardless of the underlying
37+
/// type's natural encoding — the BCP decoder needs this distinction to
38+
/// align column boundaries correctly. Populated during model.xml
39+
/// emission, consumed during the data-load pass.
40+
/// </summary>
41+
internal readonly Dictionary<string, bool[]> TableColumnIsAlias = new(StringComparer.OrdinalIgnoreCase);
3242
}
3343

3444
/// <summary>

SqlServerSimulator/Storage/Bacpac/BacpacReader.cs

Lines changed: 85 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -39,18 +39,95 @@ public static void Load(Stream stream, Simulation simulation, BacpacLoadResult r
3939
using (var modelStream = modelEntry.Open())
4040
ModelXmlReader.Apply(modelStream, simulation, result);
4141

42-
// Data load deferred to a follow-up phase. The Data/ entries are
43-
// enumerated here so the caller sees the inventory in
44-
// result.ElementCounts under a synthetic "_DataFile" bucket; the
45-
// actual row-decoder pass lands when BcpRowReader is implemented.
42+
// Data load pass — iterate every Data/<schema>.<table>/TableData-NNN-NNNNN.BCP
43+
// entry, resolve the destination HeapTable, and stream-decode rows
44+
// into the simulator's storage via Heap.Insert (bypasses SQL parsing
45+
// so the AW 760K-row payload loads in tolerable time).
46+
var database = simulation.Databases[Simulation.DefaultDatabaseName];
4647
foreach (var entry in archive.Entries)
4748
{
48-
if (entry.FullName.StartsWith("Data/", StringComparison.Ordinal)
49-
&& entry.FullName.EndsWith(".BCP", StringComparison.Ordinal))
49+
if (!entry.FullName.StartsWith("Data/", StringComparison.Ordinal)
50+
|| !entry.FullName.EndsWith(".BCP", StringComparison.Ordinal))
5051
{
51-
_ = result.ElementCounts.TryGetValue("_DataFile", out var current);
52-
result.ElementCounts["_DataFile"] = current + 1;
52+
continue;
53+
}
54+
55+
_ = result.ElementCounts.TryGetValue("_DataFile", out var current);
56+
result.ElementCounts["_DataFile"] = current + 1;
57+
58+
var (schemaName, tableName) = ParseDataFolderName(entry.FullName);
59+
if (!database.Schemas.TryGetValue(schemaName, out var schema)
60+
|| !schema.HeapTables.TryGetValue(tableName, out var table))
61+
{
62+
result.Skipped.Add(new BacpacSkipped("_DataFile", entry.FullName,
63+
$"Destination table {schemaName}.{tableName} not found — likely failed to load during DDL phase."));
64+
continue;
65+
}
66+
67+
try
68+
{
69+
using var bcpStream = entry.Open();
70+
LoadRowsFromBcp(bcpStream, table, schemaName, tableName, result);
71+
}
72+
catch (Exception ex) when (ex is InvalidDataException or NotSupportedException
73+
or EndOfStreamException or ArgumentOutOfRangeException
74+
or ArgumentException or OverflowException or FormatException
75+
or IndexOutOfRangeException)
76+
{
77+
// Best-effort: a single malformed row aborts that file but
78+
// doesn't kill the whole load. Each per-file failure lands on
79+
// Skipped with the exception type + message so the caller has
80+
// a precise inventory.
81+
result.Skipped.Add(new BacpacSkipped("_DataFile", entry.FullName,
82+
$"BCP decode failed ({ex.GetType().Name}): {ex.Message}"));
5383
}
5484
}
5585
}
86+
87+
/// <summary>
88+
/// Walks the per-row BCP stream and pokes each row directly into the
89+
/// destination heap. Bypasses the simulator's INSERT machinery — no
90+
/// IDENTITY allocation (bacpac data carries the original identity
91+
/// values), no FK enforcement, no triggers (data is already consistent
92+
/// from the source DB).
93+
/// </summary>
94+
private static void LoadRowsFromBcp(Stream bcpStream, HeapTable table, string schemaName, string tableName, BacpacLoadResult result)
95+
{
96+
// Look up the per-column alias map populated during model.xml table
97+
// emission. Key uses bracketed [schema].[table] form to match the
98+
// DACFx name shape; default to empty if missing.
99+
var qualifiedKey = $"[{schemaName}].[{tableName}]";
100+
var columnIsAlias = result.TableColumnIsAlias.TryGetValue(qualifiedKey, out var flags)
101+
? flags
102+
: new bool[table.Columns.Length];
103+
104+
var rowCount = 0;
105+
while (true)
106+
{
107+
var values = BcpRowReader.TryReadRow(bcpStream, table.Columns, columnIsAlias);
108+
if (values is null)
109+
break;
110+
var rowBytes = RowEncoder.EncodeRow(table.Columns, values, table.Heap);
111+
_ = table.Heap.Insert(rowBytes);
112+
rowCount++;
113+
}
114+
115+
_ = result.ElementCounts.TryGetValue("_DataRows", out var rowsSoFar);
116+
result.ElementCounts["_DataRows"] = rowsSoFar + rowCount;
117+
}
118+
119+
/// <summary>
120+
/// Splits <c>Data/&lt;schema&gt;.&lt;table&gt;/TableData-NNN-NNNNN.BCP</c>
121+
/// into <c>(schema, table)</c>. The folder name is always two bracketed
122+
/// segments separated by a single dot — sub-schema dots aren't a concern
123+
/// here since SQL Server schemas can't contain dots in their names.
124+
/// </summary>
125+
private static (string Schema, string Table) ParseDataFolderName(string entryFullName)
126+
{
127+
var firstSlash = entryFullName.IndexOf('/', StringComparison.Ordinal);
128+
var secondSlash = entryFullName.IndexOf('/', firstSlash + 1);
129+
var folder = entryFullName[(firstSlash + 1)..secondSlash];
130+
var dot = folder.IndexOf('.', StringComparison.Ordinal);
131+
return (folder[..dot], folder[(dot + 1)..]);
132+
}
56133
}

0 commit comments

Comments
 (0)