|
| 1 | +# BACPAC import — prerequisite feature checklist |
| 2 | + |
| 3 | +Working document for the eventual `Simulation.FromBacpac` (or `FromBacPac` — naming TBD) entry point. The plan is **emit T-SQL CREATE statements from `model.xml`, feed them through the existing parser**, then load BCP data files in `DataPhaseTables` order. The loader is a translator, not a second object-construction pipeline; the more T-SQL the parser already accepts, the smaller the loader. |
| 4 | + |
| 5 | +Reference sample: `.vs/AdventureWorks2025.bacpac` (Microsoft AdventureWorks2025, 71 tables, 760,167 total rows, 17 MB compressed). Element counts and AW-usage tallies below are probe-confirmed from that file's `model.xml` + `Origin.xml` on 2026-05-14. |
| 6 | + |
| 7 | +## Model.xml — the simulator already handles |
| 8 | + |
| 9 | +These Element types map 1:1 to features the parser already eats; the loader synthesizes the appropriate `CREATE …` text and the existing code paths do the work. No new simulator features needed. |
| 10 | + |
| 11 | +| Element | AW count | Maps to | |
| 12 | +|---|---|---| |
| 13 | +| `SqlSchema` | 5 | `CREATE SCHEMA` | |
| 14 | +| `SqlTable` / `SqlSimpleColumn` / `SqlComputedColumn` / `SqlTypeSpecifier` | 71 / 481 / 302 | `CREATE TABLE` with columns + computed | |
| 15 | +| `SqlPrimaryKeyConstraint` / `SqlUniqueConstraint` | 71 / 1 | inline + table-level PK/UQ | |
| 16 | +| `SqlForeignKeyConstraint` (incl. `OnDeleteAction`=CASCADE) | 90 (2 cascade) | `CONSTRAINT … FOREIGN KEY … REFERENCES … ON DELETE CASCADE` | |
| 17 | +| `SqlCheckConstraint` (raw T-SQL in `CheckExpressionScript`) | 89 | `CONSTRAINT … CHECK (…)` | |
| 18 | +| `SqlDefaultConstraint` (raw T-SQL in `DefaultExpressionScript`) | 152 | `CONSTRAINT … DEFAULT (…)` | |
| 19 | +| `SqlIndex` / `SqlIndexedColumnSpecification` | 95 | `CREATE [UNIQUE] [CLUSTERED] INDEX` | |
| 20 | +| `SqlView` (raw SELECT in `QueryScript`) | 20 | `CREATE VIEW … [WITH SCHEMABINDING] AS …` (SCHEMABINDING parses + discards) | |
| 21 | +| `SqlProcedure` (raw body in `BodyScript`, header in `SysCommentsObjectAnnotation.HeaderContents`) | 10 | `CREATE PROCEDURE` | |
| 22 | +| `SqlScalarFunction` / `SqlMultiStatementTableValuedFunction` / `SqlScriptFunctionImplementation` | 10 / 1 / 11 | `CREATE FUNCTION` | |
| 23 | +| `SqlDmlTrigger` (`SqlTriggerType` 2=AFTER, 3=INSTEAD OF; `IsInsert/Update/DeleteTrigger`) | 10 | `CREATE TRIGGER` | |
| 24 | +| `SqlSubroutineParameter` (`IsOutput`, type via `TypeSpecifier`) | 41 | function/proc parameters | |
| 25 | +| `SqlInlineConstraintAnnotation` | 1 | constraint inline-vs-table-level marker | |
| 26 | +| `OnlinePropertyAnnotation Name="[LastValue]"` | (per identity column) | identity high-water resume | |
| 27 | +| `SysCommentsObjectAnnotation` (`HeaderContents`, `FooterContents`) | 52 | header reconstruction for proc/func/view/trigger | |
| 28 | + |
| 29 | +DDL emission strategy: walk the model in dependency-correct order (schemas → tables → table constraints/indexes → views → functions → procedures → triggers), use each `<Element>`'s properties to assemble the `CREATE` header, concatenate `BodyScript` / `QueryScript` / `CheckExpressionScript` / etc. as-is (they're already valid T-SQL), feed the result through `SimulatedDbCommand.ExecuteNonQuery`. The `HeaderContents` annotation gives a probe-confirmed canonical form to copy when in doubt. |
| 30 | + |
| 31 | +## Model.xml — prerequisite features (blocking AW load) |
| 32 | + |
| 33 | +Sorted approximately by surface-area / effort, smallest first. Each is a candidate for its own bundle. |
| 34 | + |
| 35 | +### [ ] Database options parse-and-discard expansion (small) |
| 36 | +`SqlDatabaseOptions` carries 18+ properties; today's `SET <option>` accept-list covers most session-scope toggles but not database-scope ones. Need parse-and-discard for: `RecoveryMode`, `IsTornPageProtectionOn`, `TargetRecoveryTimePeriod`, `QueryStoreDesiredState`, `QueryStoreIntervalLength`, `TemporalHistoryRetentionEnabled`, `IsAcceleratedDatabaseRecoveryOn`, `IsOptimizedLockingOn`, `IsCursorDefaultScopeGlobal`, `IsFullTextEnabled`. **Load-bearing**: `IsReadCommittedSnapshot` → set `Database.AllowReadCommittedSnapshot` (already modeled, just needs the wire-up from the model). `Collation` from the model → respect, or hard-error if it diverges from the simulator's `SQL_Latin1_General_CP1_CI_AS` default. |
| 37 | + |
| 38 | +The natural shape is a database-scope `ALTER DATABASE … SET (…)` statement emitter from the loader; most options become no-ops in the parser. AW uses defaults for everything except `IsReadCommittedSnapshot=True`, `IsAcceleratedDatabaseRecoveryOn=True`, `IsOptimizedLockingOn=True`, `TargetRecoveryTimePeriod=60`, `QueryStoreDesiredState=2`. |
| 39 | + |
| 40 | +### [ ] UDDTs / alias types (`CREATE TYPE … FROM …`) (small-medium) |
| 41 | +6 in AW: `[dbo].[AccountNumber]` (nvarchar(15)), `[dbo].[Flag]` (bit), `[dbo].[Name]` (nvarchar(50)), `[dbo].[NameStyle]` (bit), `[dbo].[OrderNumber]` (nvarchar(25)), `[dbo].[Phone]` (nvarchar(25)). All simple alias-over-builtin with optional NOT NULL. Columns reference them by full name (`[dbo].[Name]`) instead of `[nvarchar]` in their `TypeSpecifier`'s `Type` relationship. |
| 42 | + |
| 43 | +Distinct from the already-shipped `CREATE TYPE … AS TABLE` (TVP namespace). Two options: |
| 44 | +1. **Real feature**: add an `AliasTypes` dict to `Schema`, route `TypeSpecifier`'s `Type` relationship through it during DDL emission, support `CREATE TYPE name FROM <underlying>` parser. `sys.types` already needs to surface these. |
| 45 | +2. **Loader-only inline expansion**: walk the model first, build an alias-resolution map, substitute the underlying type at DDL emission time. Smaller scope, but the simulator can't load alias-typed `DECLARE @x dbo.Name` or `CREATE TYPE … FROM` written by application code. |
| 46 | + |
| 47 | +**Recommended (1)** — even though (2) is smaller, alias types show up in real EF Core apps (data-annotation `[Column(TypeName = "...")]` over a UDDT), and the `sys.types` surface needs them anyway for catalog correctness. |
| 48 | + |
| 49 | +### [ ] Extended properties (`sp_addextendedproperty` + `sys.extended_properties`) (medium) |
| 50 | +538 in AW — they're how SQL Server attaches descriptions/metadata to schemas/tables/columns/etc. Surface needed: |
| 51 | +- `sp_addextendedproperty @name='MS_Description', @value='…', @level0type='SCHEMA', @level0name='dbo', @level1type='TABLE', @level1name='ErrorLog', @level2type='COLUMN', @level2name='ErrorMessage'` (the canonical EXEC form the loader would emit) |
| 52 | +- `sys.extended_properties` catalog view (read-back) |
| 53 | +- `fn_listextendedproperty` table-valued system function (often used by ORMs and Schema Compare tools) |
| 54 | +- Storage: probably a `Dictionary<(int class, int major_id, int minor_id), Dictionary<string, SqlValue>>` on `Database`. |
| 55 | + |
| 56 | +Pure metadata — no semantic effect on queries. Probably one bundle on its own. |
| 57 | + |
| 58 | +### [ ] `hierarchyid` data type (medium) |
| 59 | +2 columns in AW (`[HumanResources].[Employee].[OrganizationNode]` + procedure parameter). hierarchyid is variable-length binary with a documented encoding for paths like `/1/2/3/`. Surface: |
| 60 | +- Storage type + literal parser (`hierarchyid::Parse('/1/2/3/')`, `hierarchyid::GetRoot()`) |
| 61 | +- Methods: `.GetAncestor(n)`, `.GetDescendant(child1, child2)`, `.GetLevel()`, `.IsDescendantOf(other)`, `.GetReparentedValue()`, `.ToString()`, `.Read()`/`.Write()`. AW procs use `.GetDescendant` + `.GetAncestor` per probe. |
| 62 | +- BCP wire format: variable-length binary, length-prefixed. |
| 63 | + |
| 64 | +Sizable but self-contained. The path encoding is well-documented (variable-bit ordinal encoding). |
| 65 | + |
| 66 | +### [ ] DDL trigger (`CREATE TRIGGER … ON DATABASE`) (small if scoped to parse-and-discard) |
| 67 | +1 in AW: `[ddlDatabaseTriggerLog]` — fires on `DDL_DATABASE_LEVEL_EVENTS`, writes to `dbo.DatabaseLog`. Surface: `CREATE TRIGGER … ON DATABASE … FOR <event_type_group> AS …` parser + storage + dispatch. Could legitimately be parse-and-store-but-never-fire for the baseline — the trigger only fires on DDL events the simulator may not even dispatch to a trigger loop in the first place. Worth a probe to confirm AW apps actually depend on its side effects. |
| 68 | + |
| 69 | +### [ ] Permission statements (`GRANT` / `REVOKE` / `DENY`) (medium — needs principal model) |
| 70 | +2 in AW. Real surface needs: |
| 71 | +- `CREATE USER` / `CREATE ROLE` / `ALTER ROLE … ADD MEMBER` (or accept-as-no-op for the principals AW references — `public` and the schema authorizers) |
| 72 | +- `GRANT <perm> ON <object> TO <principal>` / `REVOKE` / `DENY` |
| 73 | +- `sys.database_principals`, `sys.database_permissions`, `sys.database_role_members` |
| 74 | + |
| 75 | +For the loader's "baseline AW load" goal, parse-and-discard is probably enough — the simulator has no permission enforcement, so GRANT/REVOKE are no-ops semantically. The catalog views surface as empty/synthesized. Real feature work deferred. |
| 76 | + |
| 77 | +### [ ] Full-text catalog + index (large — likely skip-with-diagnostic) |
| 78 | +1 catalog (`[AW2025FullTextCatalog]`) + 3 indexes in AW. Full surface: `CREATE FULLTEXT CATALOG` / `CREATE FULLTEXT INDEX … ON tbl(col LANGUAGE 1033) KEY INDEX <pk> ON <catalog>`, `CONTAINS()` / `FREETEXT()` predicates, `CONTAINSTABLE` / `FREETEXTTABLE` rowset functions, `sys.fulltext_catalogs` / `sys.fulltext_indexes`. |
| 79 | + |
| 80 | +The query-time predicates (`CONTAINS`, `FREETEXT`) are the hard part — they need a tokenizer/stemmer/inverted-index/relevance-rank pipeline. Recommend **skip-with-diagnostic** for the loader; AW data still loads, full-text-using queries fail at parse with `NotSupportedException("Full-text search is not modeled")`. Real feature deferred indefinitely unless an application needs it. |
| 81 | + |
| 82 | +### [ ] `xml` data type + XML schema collections + XML methods + XML indexes (very large) |
| 83 | +9 column uses in AW (`Production.Document.DocumentSummary`, `Person.Person.AdditionalContactInfo`, `HumanResources.JobCandidate.Resume`, etc.), 6 `SqlXmlSchemaCollection` (with embedded XSD schemas in `SchemaExpression`), 8 `SqlXmlIndex` (`PrimaryXmlIndexUsage` 3, secondary index types). Surface: |
| 84 | +- Storage type + `xml(SchemaCollection)` parametrization |
| 85 | +- `CREATE XML SCHEMA COLLECTION` (with XSD payload) |
| 86 | +- XML methods: `.value('xpath', 'sqltype')`, `.nodes('xpath')`, `.query('xpath')`, `.exist('xpath')`, `.modify('xml dml')` |
| 87 | +- Implicit/explicit cast between `xml` and `[n]varchar` |
| 88 | +- XML primary + secondary indexes (PATH / VALUE / PROPERTY) |
| 89 | +- `FOR XML` query-output clause (separate but related) |
| 90 | +- `sys.xml_schema_collections`, `sys.xml_indexes` |
| 91 | + |
| 92 | +Genuinely large — XPath + XML DML are independent sub-languages. Recommend **skip-with-diagnostic** in the loader for the baseline (load xml columns as `nvarchar(MAX)` containing the raw XML — preserves application read-back via `.ToString()`, breaks XPath methods). Real feature could be one or several bundles down the road. |
| 93 | + |
| 94 | +### [ ] `geography` / `geometry` data types (large — likely skip-with-diagnostic) |
| 95 | +1 column in AW (`Person.Address.SpatialLocation`). Spatial types have their own large surface (WKT/WKB parsing, OGC methods, spatial indexes). Recommend **skip-with-diagnostic**; load as `varbinary(MAX)` or `nvarchar(MAX)` in degraded mode, application queries that call `.STDistance` etc. fail at parse. |
| 96 | + |
| 97 | +## BCP wire format |
| 98 | + |
| 99 | +The `Data/<schema>.<table>/TableData-NNN-NNNNN.BCP` files are the per-table data payload. Probed against `Production.ProductCategory` (4 rows, 192 bytes, schema `int IDENTITY NOT NULL, nvarchar(50) NOT NULL, uniqueidentifier NOT NULL, datetime NOT NULL` — verified row 1 = `(1, 'Bikes', <guid>, 2019-04-30 00:00:00)`): |
| 100 | + |
| 101 | +| Type family | Wire layout | Notes | |
| 102 | +|---|---|---| |
| 103 | +| Fixed-width numeric (`int`, `bigint`, `smallint`, `tinyint`, `bit`) | raw bytes, little-endian, no prefix | `int` = 4 bytes LE | |
| 104 | +| Fixed-width temporal (`datetime`, `smalldatetime`, `date`) | raw bytes, no prefix | `datetime` = 4-byte int32 days + 4-byte uint32 ticks-of-day (1/300 sec) | |
| 105 | +| Variable-length text/binary (`nvarchar`, `varchar`, `varbinary`) | 2-byte LE byte-length prefix + bytes | nvarchar = UTF-16 LE; `0xFFFF` likely = NULL (needs probe confirm) | |
| 106 | +| Length-prefixed fixed (`uniqueidentifier`, `decimal`/`numeric`, `money`, `smallmoney`, `datetime2`, `datetimeoffset`, `time`) | 1-byte length-prefix (= type width) + bytes | guid = `0x10` + 16 bytes; `0x00` likely = NULL | |
| 107 | +| MAX types (`varchar(MAX)`, `nvarchar(MAX)`, `varbinary(MAX)`, `text`, `ntext`, `image`) | length prefix likely 8-byte for full size + bytes (or chunked) | needs probe — not in ProductCategory | |
| 108 | +| `hierarchyid` | variable-length binary, length-prefixed | covered by hierarchyid feature work | |
| 109 | +| `xml` | variable-length text/binary, length-prefixed | covered by xml feature work; if xml is loaded as nvarchar(MAX) in degraded mode, falls through to the MAX-types row | |
| 110 | +| `sql_variant` | special envelope (type byte + value) | not yet investigated, AW may or may not use | |
| 111 | + |
| 112 | +Encoding-edge probes needed (carve out tiny custom BACPACs locally via `SqlPackage`): |
| 113 | +- NULL sentinel for each prefix class (1-byte / 2-byte / fixed-no-prefix) |
| 114 | +- `decimal(p, s)` precision/scale layout (probably sign byte + LE mantissa) |
| 115 | +- `datetime2(N)` and `datetimeoffset(N)` precision dependence |
| 116 | +- MAX-type encoding when row >> 8KB |
| 117 | +- `varbinary(N)` for `rowversion` columns (auto-generated server-side — does BACPAC export them or skip?) |
| 118 | +- `IDENTITY` reseed: confirm `LastValue` annotation matches the actual max-allocated rather than max-inserted |
| 119 | + |
| 120 | +## Order of operations toward AW baseline |
| 121 | + |
| 122 | +Rough sequence — work each bundle to completion, update this checklist, then revisit BACPAC scoping once the prerequisites land: |
| 123 | + |
| 124 | +1. **Database options expansion** + **UDDTs** (smallest two, no XML / spatial blocker dependencies) |
| 125 | +2. **Extended properties** (mid-size, self-contained) |
| 126 | +3. **`hierarchyid`** (large but self-contained, unblocks `HumanResources.Employee`) |
| 127 | +4. **DDL trigger + permission statements** as parse-and-store-but-no-enforce (smallest scope; both end up as catalog-view-visible no-ops) |
| 128 | +5. **Loader baseline implementation**, with `xml` / `geography` / full-text **loaded in degraded mode** (xml/geography → nvarchar(MAX), full-text indexes → parse-and-discard). Diagnostics report which features were degraded. |
| 129 | +6. **Real xml + spatial + full-text** as separate post-baseline initiatives, each promoted from degraded-mode-via-diagnostic to first-class as bundles complete. |
| 130 | + |
| 131 | +Loader code layout (target, when baseline lands): |
| 132 | +- `SqlServerSimulator/Storage/Bacpac/BacpacReader.cs` — OPC zip walker, dispatches to model + data readers |
| 133 | +- `SqlServerSimulator/Storage/Bacpac/ModelXmlReader.cs` — `model.xml` → DDL emitter |
| 134 | +- `SqlServerSimulator/Storage/Bacpac/BcpRowReader.cs` — `*.BCP` → row decoder |
| 135 | +- `SqlServerSimulator/Storage/Bacpac/BacpacLoadResult.cs` — diagnostics carrier (Skipped + Degraded lists) |
| 136 | +- Public surface: `internal static Simulation Simulation.FromBacpac(string path, out BacpacLoadResult diagnostics)` + Stream overload, kept internal until baseline AW load works end-to-end |
| 137 | + |
| 138 | +## Status |
| 139 | + |
| 140 | +Pre-implementation. Scoping done 2026-05-14. Implementation paused until prerequisite features land; resume by reopening the session named `bacpac`. |
0 commit comments