Commit b550a7c
committed
BACPAC loader Phases F+G — extended properties + BCP row data. With this bundle, AdventureWorks loads end-to-end through a single
**Phase F — extended properties** in `EmitExtendedProperty`. DACFx encodes the host kind in the element name's leading bracketed segment (`[SqlColumn]` / `[SqlTableBase]` / `[SqlSchema]` / `[SqlDatabaseOptions]` / `[SqlDatabaseDdlTrigger]` / `[SqlFilegroup]`); the emitter resolves each to `sp_addextendedproperty`'s `@level0type` / `@level1type` / `@level2type` arg triplet (SCHEMA / TABLE | VIEW / COLUMN) and uses the existing `viewNames` HashSet to distinguish TABLE vs VIEW level1. Out-of-scope hosts (SqlFilegroup, SqlDatabaseDdlTrigger) plus 9 column/table-level properties whose host columns/tables didn't load (computed-col tables, vJobCandidate-family views) land on Skipped. New shared helpers `SplitBracketedSegments`, `Unbracket`, `TrySplit2Part`, `TrySplit3Part` factor out the bracketed-name parsing. SqlPermissionStatement intentionally Skipped — AW's 2 entries are Always-Encrypted permissions using DACFx integer enum codes (1107/1108) that need a separate enum→T-SQL mapping.
**Phase G — BCP data loader**. New `BcpRowReader` decodes 18 base types directly into `SqlValue[]` rows; `BacpacReader.LoadRowsFromBcp` pokes each row into `HeapTable.Heap` via `RowEncoder.EncodeRow` + `Heap.Insert`, bypassing SQL parsing for the 760K-row hot path. Decoded types: int / bigint / smallint / tinyint / bit, datetime / smalldatetime / date, money / smallmoney, decimal/numeric (BigInteger-based mantissa decode for arbitrary precision), datetime2 / time / datetimeoffset, uniqueidentifier, varchar / nvarchar / nchar / char (bounded), varbinary / binary (bounded). Per-file try/catch surfaces decode failures on Skipped with the exception type + message rather than aborting the whole load.
**Two real probe-confirmed wire-format quirks** (verified via hex-dump of AW's `Sales.SpecialOffer` and `HumanResources.Shift` BCP files on 2026-05-15, both corrections vs. the original prereqs-doc matrix):
1. **UDDT-aliased columns get a 1-byte length prefix even when NOT NULL**, regardless of underlying type. AW's `Sales.SalesOrderHeader.OnlineOrderFlag` (declared via `dbo.Flag` alias = `bit NOT NULL`) encodes as `01 01` (1-byte prefix = 1, then 1 byte value) rather than a single raw bit byte. Threaded through via a new `BacpacLoadResult.TableColumnIsAlias` side map populated during `EmitTable` (via the new `IsAliasTypedColumn` helper that checks `<References ExternalSource="BuiltIns" …>` vs. unmarked refs in the TypeSpecifier's Type relationship) and consumed in `BcpRowReader.TryReadRow`. The fix unblocked 33K rows in AW (685K → 718K loaded).
2. **money / smallmoney / time(N) / datetime2(N) / datetimeoffset(N) are fixed-raw NOT NULL, not always length-prefixed** as the original prereqs-doc matrix claimed. `Sales.SpecialOffer.DiscountPct` smallmoney NOT NULL = 4 raw bytes (no prefix) for value 0; `HumanResources.Shift.StartTime` time(7) NOT NULL = 5 raw bytes (no prefix). Reclassified from `ReadLengthPrefixed1Variable` to `ReadFixedRaw` with width = `TimeSqlType.timeBytes` (3/4/5 by precision) / `DateTime2SqlType.timeBytes + 3` / `DateTimeOffsetSqlType.timeBytes + 5`. NULL handling for these types uses the same `0xFF`-as-NULL 1-byte prefix as int/datetime/etc.
**Loader bug fix in `TranslateSimpleColumn`**: `IsNullable` absent from DACFx means "inherit the column-type's default" (NULL for builtins per ANSI, the alias's stored IsNullable for UDDT-typed columns). Was previously defaulting to True → always emitting `NULL` in the CREATE TABLE column declaration → forcing UDDT-typed columns (where the alias defaults NOT NULL, e.g. `dbo.Flag`) to load as nullable. New `isNullableExplicit` flag tracks the presence/absence distinction; omitting the marker entirely when absent lets the simulator's resolver pick the right default through both paths.
**Loader infrastructure expansion**: `BacpacLoadResult` gains the `TableColumnIsAlias: Dictionary<string, bool[]>` side map keyed by bracketed `[schema].[table]`; per-row-decode bookkeeping adds `ElementCounts["_DataFile"]` (per-shard count) and `ElementCounts["_DataRows"]` (cumulative). `BacpacReader.Load` now does a single pass over all archive entries — model.xml first (existing 7-phase dispatch), then the Data folder iteration for the row pass. New helper `ParseDataFolderName` splits `Data/<schema>.<table>/TableData-NNN-NNNNN.BCP` paths into `(schema, table)` tuples. `RunPhase`'s last-phase counter advances to 8 to cover the new extended-property dispatch.
**Loader pushback stream**: new `PushbackStream` helper class inside `BcpRowReader` peeks the first byte of each row to detect EOF without throwing, then puts the byte back into the decoder pipeline. Required because `Stream.ReadByte` returns -1 at EOF but the column decoders all use `ReadExact` (which throws on premature EOF) — without the pushback the loader couldn't distinguish "end of file" from "torn row."
**AW coverage**: 5/5 schemas, 71/71 tables, 6/6 UDDTs, 71/71 PKs, 1/1 UQ, 90/90 FKs, 89/89 CHECKs, 152/152 DEFAULTs, 89/95 indexes, 11/20 views, 8/10 procs, 10/11 functions, 10/10 DML triggers, 1/1 DDL trigger, 527/538 extended properties, and **718,369/760,167 rows**. Remaining deferrals (all on Skipped with type-named reasons): hierarchyid (HumanResources.Employee.OrganizationNode, Production.Document.DocumentNode), geography (Person.Address.SpatialLocation), xml (Person.Person.Demographics, HumanResources.JobCandidate.Resume), varbinary(MAX) / nvarchar(MAX) (Production.ProductPhoto's photo blobs, etc.).Simulation.FromBacpac call: schema + constraints + indexes + views + functions + procedures + triggers + extended properties (Phase F), plus 718K of 760K rows of actual table data (Phase G). Remaining row gap is feature-gated by four wire-format decoders (hierarchyid / geography / xml / MAX types) that the loader cleanly defers to Skipped rather than aborting.1 parent 1809bce commit b550a7c
5 files changed
Lines changed: 760 additions & 16 deletions
File tree
- SqlServerSimulator.Tests.Internal/Storage
- SqlServerSimulator/Storage/Bacpac
Lines changed: 67 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
86 | | - | |
87 | | - | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
88 | 90 | | |
89 | 91 | | |
90 | 92 | | |
91 | 93 | | |
92 | 94 | | |
93 | 95 | | |
94 | | - | |
95 | 96 | | |
| 97 | + | |
96 | 98 | | |
97 | 99 | | |
98 | 100 | | |
| |||
253 | 255 | | |
254 | 256 | | |
255 | 257 | | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
256 | 320 | | |
257 | 321 | | |
258 | 322 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
32 | 42 | | |
33 | 43 | | |
34 | 44 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
46 | 47 | | |
47 | 48 | | |
48 | | - | |
49 | | - | |
| 49 | + | |
| 50 | + | |
50 | 51 | | |
51 | | - | |
52 | | - | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
53 | 83 | | |
54 | 84 | | |
55 | 85 | | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
56 | 133 | | |
0 commit comments