Promote bulk gap-bridging + cleanup + workspace_swap hardening to main#38
Merged
Conversation
* Add PyYAML dependency for upcoming bulk-parameter.yml support
Increment 1 of Phase 2 (bulk gap-bridging). Plumbing only — no behavior
change. Subsequent increments will read bulk-parameter.yml via PyYAML to
drive substitutions and VariableLibrary value-set activation in the bulk
deploy path.
Adds PyYAML to:
- requirements-dev.txt (so editor and local pytest can resolve the import)
- .github/workflows/reusable-deploy-bulk.yml pip install (so the bulk
workflow has it at deploy time without relying on it coming in
transitively via fabric-cicd, which the bulk path does not install)
* Add bulk-parameter.yml schema and config loader (no wiring yet)
Increment 2 of Phase 2 (bulk gap-bridging). Adds the bulk-deploy config file
and a parser for it; main() does not yet call the loader, so bulk deploy
behavior is unchanged.
New file: data/fabric/bulk-parameter.yml
- Schema independent from fabric-cicd's parameter.yml
- Lists the 5 active substitution rules and the variable_library
active_value_set placeholder
scripts/deploy_bulk.py:
- Add SubstitutionRule and BulkConfig frozen dataclasses
- Add load_bulk_config(): YAML -> typed BulkConfig with explicit validation
on each malformed-input case (raises ValueError with line-of-fault info)
- Add BULK_PARAMETER_FILENAME constant; include it in EXCLUDED_FILES so the
config file is never sent as part of the bulk request payload (the
structural rule already excludes it; the named-files entry is
belt-and-suspenders + self-documenting)
tests/test_deploy_bulk.py:
- 15 new tests covering load_bulk_config happy paths and every documented
error case, plus a smoke test against the real shipped YAML file
Verified: 122 tests pass (was 107).
* Add pure substitution helpers for bulk deploy (no wiring yet)
Increment 3 of Phase 2 (bulk gap-bridging). Adds the building blocks
main() will use; nothing is called yet, so bulk deploy behavior is
unchanged.
scripts/deploy_bulk.py:
- DEPENDENCY_TYPES constant (Lakehouse, Ontology) — items others reference
via $items.<Type>.<Name>.$id, which therefore must be deployed first
- SUBSTITUTABLE_EXTENSIONS — text extensions safe to find/replace inside
- New helpers (all pure, all unit-tested):
item_type_of, item_display_name_of — parse part paths
partition_dependencies — split parts into (deps, rest)
extract_item_ids — bulk response -> id lookup
resolve_dynamic_value — $workspace.$id, $items.<T>.<N>.$id
apply_substitutions — walk parts, decode/replace/encode
resolve_active_value_set — $environment placeholder
find_variable_library_id — locate the deployed VL by item type
Naming intentionally avoids "Phase 1 / Phase 2" terminology in bulk because
not all bulk deploys end up doing two POSTs (single POST is fine when the
config has no $items.* references). Talks about "dependencies" and "the
rest" instead. deploy_fabric_cicd.py keeps its PHASE1_TYPES naming —
different file, different deploy implementation, no consistency loss.
tests/test_deploy_bulk.py:
- 42 new tests covering each helper's happy path, edge cases, and error
modes. Includes parametrized cases for path parsing, identity-check on
apply_substitutions for unchanged parts, and fail-loud assertions for
unresolved placeholders / multiple VariableLibraries.
Verified: 164 tests pass (was 122).
* Wire bulk deploy into two-deploy flow with substitutions
Increment 4 of Phase 2 (bulk gap-bridging). First behavioral change in
this branch: bulk deploys now apply substitutions from bulk-parameter.yml
so deployed items reference the target workspace's IDs instead of dev's.
scripts/deploy_bulk.py:
- Extract POST + sync/async + LRO + result-fetch out of main() into a
reusable post_bulk() helper that returns the result body. Lets the
caller chain two deploys (extract IDs from one, substitute into the
next). Calls check_per_item_status internally to keep fail-loud
behavior on per-item errors.
- Rewrite main() with a decision: if bulk-parameter.yml has no
$items.<Type>.<Name>.$id references, do a single POST (with
$workspace.$id-only substitutions if any). Otherwise run a two-deploy
flow: POST dependencies, extract their IDs, substitute the rest,
POST the rest.
- Read ENVIRONMENT env var (used by Increment 5; harmless to read now).
- Module docstring: drop 'parameter.yml find_replace' from Known Gaps,
document the new deploy flow and the ENVIRONMENT input.
.github/workflows/reusable-deploy-bulk.yml:
- Pass the workflow's environment input through to the script as the
ENVIRONMENT env var. Increment 5 uses it for value-set activation.
Verified: 164 tests still pass. Substitution helpers (Increment 3) are
now used; orchestration in main() follows the file's existing convention
of being validated end-to-end via deploy workflows, not unit-tested.
* Activate VariableLibrary value set after bulk deploy
Increment 5 of Phase 2 (bulk gap-bridging). Closes the second observed
gap from Phase 2's problem statement: bulk deploys now select the active
value set on the deployed VariableLibrary, matching what fabric-cicd
does automatically.
scripts/deploy_bulk.py:
- New helper activate_variable_library_value_set(): issues
PATCH /v1/workspaces/{ws}/variableLibraries/{id} with
{"properties": {"activeValueSetName": <name>}}. Endpoint is GA
(no ?beta=true), SPN supported. Reference:
https://learn.microsoft.com/en-us/rest/api/fabric/variablelibrary/items/update-variable-library
- Refactor main() so single-deploy and two-deploy branches both produce
an item_id_map, and the value-set activation runs once after either.
- Activation is gated on bulk-parameter.yml's
variable_library.active_value_set being non-null. If config requests
activation but no VariableLibrary was deployed, emit a workflow
warning rather than failing — the deploy itself succeeded.
tests/test_deploy_bulk.py:
- 2 new tests for the activation helper: happy path verifies URL/body/
headers; failure path verifies sys.exit with status code in message.
Verified: 166 tests pass (was 164).
Promote Phase 2 (bulk gap-bridging) to test
* Rename reusable-deploy-supported.yml to reusable-deploy-fabric-cicd.yml
Symmetric naming with reusable-deploy-bulk.yml. Both reusable workflows
are now named after their underlying deploy mechanism rather than what
they deploy ('supported items' was vague — both paths deploy supported
items).
Renames:
- File: reusable-deploy-supported.yml -> reusable-deploy-fabric-cicd.yml
- Reusable workflow name: 'Reusable: Deploy Supported Items' ->
'Reusable: Deploy via fabric-cicd'
- Reusable job name: 'Deploy supported items (...)' ->
'Deploy via fabric-cicd (...)'
- Orchestrator job ID: deploy-supported -> deploy-fabric-cicd
(in deploy-test.yml and deploy-prod.yml)
- Orchestrator job name: 'Deploy supported items' -> 'Deploy via fabric-cicd'
Other touches:
- reusable-deploy-bulk.yml header comments updated to reference the new
filename (two stale references)
- scripts/deploy_fabric_cicd.py module docstring updated
Not changed in this commit (covered by the docs pass on this same branch):
- 6 references in fabric-hybrid-cicd-guide.md
- 2 narrative 'deploy supported items' phrasings that describe the
sandwich-pattern concept, not workflow names — those stay
Caller wiring verified: deploy-test.yml and deploy-prod.yml point at the
new filename. ETL workflow_run triggers reference orchestrator names
(unchanged), so ETL chains stay intact.
Tests: 166 pass.
* Hardening: token mask, Retry-After clamping, nested VL config, log clarity
Four small post-Phase-2 cleanups applied together because they all touch
deploy_bulk.py and overlap.
1. Drop the ::add-mask:: line in acquire_token. The line itself emitted
the bearer token to stdout before GitHub's mask filter could redact
it, defeating the purpose. The token is never logged elsewhere, so
no mask was needed in the first place. Test inverted to assert the
token does NOT appear in stdout (regression guard).
2. Add POLL_CEILING_SECONDS = 600 and a _parse_retry_after helper that
handles None, unparseable strings, and clamps to
[POLL_FLOOR_SECONDS, POLL_CEILING_SECONDS]. Both poll_lro and
interpret_post_response now use it. Prevents a pathological
Retry-After value (or unparseable garbage) from either crashing the
script or sleeping past the global polling timeout.
3. New VariableLibraryConfig nested dataclass replaces the flat
BulkConfig.variable_library_active_value_set field. The dataclass
shape now mirrors the YAML structure, and future VariableLibrary
settings can be added without changing the parent shape.
4. main()'s log output now distinguishes three cases:
- bulk-parameter.yml missing
- present but no rules
- present with N rules
Cleaner mental model when debugging 'why isn't substitution
happening?'.
Tests: 173 pass (was 166; +7 for _parse_retry_after happy/clamp/fallback
paths and the new interpret_post_response defensive behavior).
* Add TypedDicts for Fabric API response shapes across scripts/
Pure annotation pass. No runtime behavior change; tests pass unchanged.
Adds TypedDict classes to document the shapes the scripts send and
receive from the Fabric REST API. They give Pylance/mypy the
information needed to catch field-name typos and to autocomplete on
response objects. Not enforced at runtime.
scripts/deploy_bulk.py — 4 new TypedDicts:
- DefinitionPart (total): one definitionParts[] element
- ImportItemDetail (partial): one importItemDefinitionsDetails[] entry
- BulkResponseBody (partial): sync 200 / LRO /result body
- LROStatusBody (partial): /v1/operations/{id} poll body
Function signatures updated: partition_dependencies, extract_item_ids,
apply_substitutions, build_definition_parts, check_per_item_status,
interpret_post_response, post_bulk, plus the body local in poll_lro.
scripts/run_fabric_etl.py — 2 new TypedDicts:
- FabricItem (partial): one List Items value[] element
- JobStatusBody (partial): Jobs API status response
Function signatures updated: find_item_id_by_name,
interpret_poll_response, plus the items local in main().
scripts/workspace_swap.py — 1 new TypedDict:
- ItemTypeRegistryEntry (total): one ITEM_TYPES element
Replaces the union-soup annotation
`list[dict[str, str | list[str] | bool | Callable[[str], bool] | None]]`
with a single named type. Highest readability win in this pass.
scripts/deploy_fabric_cicd.py — no changes (delegates to fabric-cicd
library types; no dict shapes worth typing locally).
`headers: dict` annotations left alone in all files — they're loose
CaseInsensitive mappings from requests that don't benefit from
TypedDict.
Verified:
- 173 tests pass (no regressions)
- Zero Pylance errors across all six edited files
* Docs: bulk implementation guide + bulk narrative cleanup
Adds the Bulk CI/CD Implementation Guide and updates surrounding docs to
honestly describe what the bulk path does, what it bridges, and what it
doesn't. Closes the documentation gap from Phase 2 (the bulk gap-bridging
work) and the workflow rename earlier on this branch.
New file:
- fabric-bulk-cicd-guide.md (~370 lines) — implementation guide for the
bulk deploy path. Mirrors the structure of fabric-hybrid-cicd-guide.md
(architecture, repo structure, deployment flow, workflows,
configuration strategy, prerequisites, initial deploy, gotchas), with
bulk-specific sections for the two-deploy decision, extension
patterns, and limitations not bridged. Self-contained — duplicates
workspace setup / SPN / GitHub environments rather than linking, so it
reads end-to-end without bouncing between docs.
Updated docs:
- README.md — new row in the Documentation table for the bulk guide.
- fabric-cicd-release-options.md — bulk row in the comparison table now
says 'None at the API level' with a note pointer; new note block
explains substitution and value-set activation are caller-implemented
workarounds, not API capabilities, and links to the new bulk guide;
recommendation paragraph reworded to reflect that bulk is still in
Preview and that bridging is caller responsibility.
- fabric-hybrid-cicd-guide.md — workflow rename (6 occurrences of the
old reusable-deploy-supported.yml filename), repo structure tree
updated with all currently-existing workflows / scripts / data items,
new note pointing to the bulk path with link to the bulk guide.
- .github/workflows/reusable-deploy-bulk.yml — Known Gaps comment block
expanded honestly: distinguishes API gaps from caller-bridged gaps,
notes orphan cleanup is not implemented because the API doesn't
support delete.
- scripts/deploy_bulk.py — module docstring expanded with the same
framing (API gaps vs caller bridging) and the Known Gaps list now
distinguishes 'no full parameter.yml feature coverage' (specific) from
'no orphan cleanup' (broader).
Verified:
- 173 tests still pass
- Zero stale references to reusable-deploy-supported.yml or the old
deploy-supported job ID anywhere in the repo
- All cross-links between docs resolve to real anchors
* Swap to feature workspace + UTF-8 stdout fix for workspace_swap scripts/workspace_swap.py: - Force UTF-8 reconfigure on sys.stdout/sys.stderr at module load. Without this, the Unicode box-drawing characters used in section headers and the summary banner crash on Windows consoles that default to cp1252. Surfaced when running --dry-run end-to-end for the first time. data/fabric (workspace_swap output for branch test-workspace-swap-changes): - New per-branch value set: Patterns_Variables.VariableLibrary/valueSets/ test-workspace-swap-changes.json - settings.json valueSetsOrder updated - Patterns_Semantic_Model.SemanticModel: Direct Lake URL repointed - Import_Patterns_Data.Notebook: META block lakehouse dependency repointed * Update * Harden workspace_swap: mandatory .env, YES confirmation, self-heal recovery Surfaced by a real-world incident where stale .env values produced a swap pointed at the wrong workspace and the script's one-way find/replace couldn't recover from the resulting bad state. scripts/workspace_swap.py changes: 1. resolve_feature_ids() now requires .env. Removes the value-set-priority path and the interactive-prompt fallback. Both were silently overriding what the developer typed in .env. Missing .env, missing keys, or blank values \u2192 sys.exit() with a clear error pointing at .env.sample. 2. New _confirm_swap_to_feature() helper. Prints a 'Planned swap' summary showing dev \u2192 feature for both workspace and lakehouse, then requires the user to type literal 'YES' (case-sensitive, exact match). Anything else \u2192 abort cleanly. Skipped on --dry-run since the dry-run is itself the verification step. 3. New _read_previous_feature_ids() helper + recovery pass in _run_swap_to_feature(). Reads the previously-applied feature IDs from the per-branch value-set file BEFORE the value set is rewritten in step 3, then runs a second find/replace pass in step 4 to rewrite any stale feature IDs that remain in repointable files. Skipped when the previous IDs match the target (no recovery needed) or when there's no previous record (first swap). 4. --check-ready and --swap-to-dev are unaffected. swap-to-dev still reads from the value-set file as the source of feature IDs. tests/test_workspace_swap.py changes: - Removed the value-set-priority and prompt-fallback tests - Added 7 strict 'exits cleanly' tests for resolve_feature_ids - New TestConfirmSwapToFeature class: 7 tests covering YES / lowercase / blank / EOF / whitespace / planned-summary-display - New TestReadPreviousFeatureIds class: 4 tests covering the helper - Wrapped existing _run_swap_to_feature test calls in YES input mock - Added test_dry_run_skips_confirmation - Added test_recovers_from_stale_feature_ids_in_files (regression guard for today's incident) - Added test_no_recovery_pass_when_stale_matches_target Verified: 185 tests pass (was 173). * Fix corrupted feature IDs in SemanticModel + Notebook (manual recovery) The first run of workspace_swap on this branch (commit 9aebf8f) was driven by stale .env values, applying the wrong workspace ID (613e22cd-308a-4106-9745-58bd5164568a) and the wrong lakehouse ID (1b80046f-d76e-4748-acfc-575f3fbd23f4) into: - Patterns_Semantic_Model.SemanticModel/definition/expressions.tmdl - Import_Patterns_Data.Notebook/notebook-content.py The Fabric UI subsequently corrected the value-set file (commit 7f7ac85), which severed the breadcrumb the new self-heal recovery pass uses to discover stale IDs. So the recovery pass can't help with this specific state \u2014 the fix is manual. Replaces: 613e22cd-308a-4106-9745-58bd5164568a \u2192 1484d4b0-4c88-4347-9cce-2d2bd03848b0 1b80046f-d76e-4748-acfc-575f3fbd23f4 \u2192 914bb153-d687-4ea9-9870-d61434b2884f Also normalizes the trailing newline in the value-set file (added by the script's JSON serializer; cosmetic). After this commit, all .env GUIDs match what's in the tracked Fabric files. Going forward, the script's confirmation prompt + self-heal recovery will catch this class of incident automatically. * Ghost * Swap back to dev for PR readiness Reverts the test-workspace-swap-changes branch to dev IDs ahead of opening the PR to dev. The check-pr-ready CI workflow enforces this — feature IDs in tracked Fabric files would block the merge. Reverted by scripts/workspace_swap.py --swap-to-dev: - Patterns_Semantic_Model.SemanticModel/definition/expressions.tmdl (Direct Lake URL: feature → dev) - Import_Patterns_Data.Notebook/notebook-content.py (META block: feature → dev) - Patterns_Variables.VariableLibrary/valueSets/test-workspace-swap-changes.json (deleted — per-branch value set no longer needed) - Patterns_Variables.VariableLibrary/settings.json (test-workspace-swap-changes removed from valueSetsOrder)
* Swap to feature workspace + chat-driven confirmation in prompt Test of the updated swap-to-feature.prompt.md flow that moves the YES confirmation into the chat UI instead of a live terminal prompt. scripts/workspace_swap.py output for branch swap-file-smoke-test: - New per-branch value set: Patterns_Variables.VariableLibrary/valueSets/ swap-file-smoke-test.json - settings.json valueSetsOrder updated - Patterns_Semantic_Model.SemanticModel: Direct Lake URL repointed to feature workspace + lakehouse - Import_Patterns_Data.Notebook: META block lakehouse dependency repointed to feature workspace + lakehouse .github/prompts/swap-to-feature.prompt.md updated to instruct the agent to ask for confirmation in the chat UI (vscode_askQuestions) and then pipe YES into the script (echo "YES" | python ...) so the terminal never blocks on the script's interactive Confirm: prompt. * Swap back to dev for PR readiness Reverts the swap-file-smoke-test branch to dev IDs ahead of opening the PR to dev. The check-pr-ready CI workflow enforces this — feature IDs in tracked Fabric files would block the merge. Reverted by scripts/workspace_swap.py --swap-to-dev: - Patterns_Semantic_Model.SemanticModel/definition/expressions.tmdl (Direct Lake URL: feature → dev) - Import_Patterns_Data.Notebook/notebook-content.py (META block: feature → dev) - Patterns_Variables.VariableLibrary/valueSets/swap-file-smoke-test.json (deleted — per-branch value set no longer needed) - Patterns_Variables.VariableLibrary/settings.json (swap-file-smoke-test removed from valueSetsOrder)
… pass (#36) PR #34 made .env mandatory for swap-to-feature, removed the interactive prompt fallback, removed the value-set-priority shortcut, and added a case-sensitive YES confirmation. PR #35 moved that confirmation into the chat UI for slash-command invocations. fabric-development-process.md still described the old behavior in three spots. Updates: - Step 4 (swap-to-feature): drop the 'or prompts if .env is missing' claim; add the YES confirmation step; add the recovery pass step. - Local .env Setup section: replace 'falls back to interactive prompt' + 'value set is source of truth' with the new model (.env always authoritative for swap-to-feature; value set used by swap-to-dev and the recovery pass; YES confirmation gate). - Copilot Chat section: note that /swap-to-feature moves the YES confirmation into the chat UI. Tests: 185 pass (no script changes).
Promote bulk gap-bridging + cleanup + workspace_swap hardening to test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes the cluster of work that's accumulated since the last test → main promotion.
DEPLOY_METHOD=bulkis the active deploy path; the prod-environment Bulk API workflow will fire on this merge.Commits being promoted
Risk
Validated end-to-end on the test environment (Bulk API deploy + ETL chain succeeded after PR #37 merged to test). The prod Bulk API workflow uses the same reusable + script.
What deploys
Deploy to Prod (Bulk API)triggers on this merge.ETL - Prodchains afterward via workflow_run.