Merge pull request #38 from michaeldeongreen/test

Promote bulk gap-bridging + cleanup + workspace_swap hardening to main

Author: michaeldeongreen

.github/prompts/swap-to-feature.prompt.md

@@ -2,6 +2,15 @@
 description: Swap the repo's IDs to your feature workspace and create the value set for the current branch.
 mode: agent
 ---
-Run `python scripts/workspace_swap.py` from the repo root and report the output to me. The script will rewrite tracked Fabric files (semantic model, notebooks) so they point at your feature workspace instead of dev, create a feature value set, and update settings.json. It reads the feature workspace and lakehouse GUIDs from `.env` at the repo root — if `.env` is missing or empty, it will fall back to an interactive prompt.
+The script `scripts/workspace_swap.py` rewrites tracked Fabric files (semantic model, notebooks) so they point at your feature workspace instead of dev, creates a feature value set, and updates settings.json. It reads `FEATURE_WORKSPACE_ID` and `FEATURE_LAKEHOUSE_ID` from `.env` at the repo root and asks for a `YES` confirmation before applying. When invoked through this prompt, that confirmation must happen here in chat — never in the terminal — to avoid live-prompt handling problems.
 
-After the script finishes, summarize what changed and remind me to commit and push the changes, then sync the workspace from the Fabric UI.
+Follow this exact sequence:
+
+1. Read `.env` at the repo root and read the dev workspace + lakehouse IDs from `data/fabric/Patterns_Variables.VariableLibrary/variables.json`.
+2. Show me the planned swap in chat (current branch, dev → feature workspace ID, dev → feature lakehouse ID).
+3. Use the chat UI to ask me a single question with two options: `YES` and `NO` (uppercase, exact). Do not proceed until I answer.
+4. If I answer `YES`, run `echo "YES" | python scripts/workspace_swap.py`. The piped `YES` satisfies the script's confirmation gate non-interactively, so the terminal never blocks. Report the full output.
+5. If I answer `NO`, do nothing further and confirm in chat that the swap was not applied.
+6. After a successful run only, summarize what changed and remind me to commit and push the changes, then sync the workspace from the Fabric UI.
+
+Do not run the script directly without piping `YES` (it will block on the terminal prompt). Do not skip the chat confirmation — the in-script confirmation is bypassed by the pipe and the chat dialog is the gate.

.github/workflows/deploy-prod.yml

@@ -21,13 +21,13 @@ permissions:
   contents: read
 
 jobs:
-  deploy-supported:
-    name: Deploy supported items
+  deploy-fabric-cicd:
+    name: Deploy via fabric-cicd
     # Gated by the DEPLOY_METHOD repository variable. Runs when unset or set to
     # 'fabric-cicd'. Set DEPLOY_METHOD='bulk' to route deployments through
     # deploy-prod-bulk.yml instead. Any other value disables both workflows.
     if: vars.DEPLOY_METHOD == '' || vars.DEPLOY_METHOD == 'fabric-cicd'
-    uses: ./.github/workflows/reusable-deploy-supported.yml
+    uses: ./.github/workflows/reusable-deploy-fabric-cicd.yml
     with:
       environment: Prod
       item_type_in_scope: '["Lakehouse", "Ontology", "VariableLibrary", "Notebook", "SemanticModel", "Report", "DataAgent"]'

.github/workflows/deploy-test.yml

@@ -18,13 +18,13 @@ permissions:
   contents: read
 
 jobs:
-  deploy-supported:
-    name: Deploy supported items
+  deploy-fabric-cicd:
+    name: Deploy via fabric-cicd
     # Gated by the DEPLOY_METHOD repository variable. Runs when unset or set to
     # 'fabric-cicd'. Set DEPLOY_METHOD='bulk' to route deployments through
     # deploy-test-bulk.yml instead. Any other value disables both workflows.
     if: vars.DEPLOY_METHOD == '' || vars.DEPLOY_METHOD == 'fabric-cicd'
-    uses: ./.github/workflows/reusable-deploy-supported.yml
+    uses: ./.github/workflows/reusable-deploy-fabric-cicd.yml
     with:
       environment: Test
       item_type_in_scope: '["Lakehouse", "Ontology", "VariableLibrary", "Notebook", "SemanticModel", "Report", "DataAgent"]'

.github/workflows/reusable-deploy-bulk.yml

@@ -1,6 +1,6 @@
 # Reusable workflow: Deploy supported Fabric items via the Bulk Import Item Definitions API (Preview).
 #
-# Alternative deployment path to reusable-deploy-supported.yml. Uses the Fabric
+# Alternative deployment path to reusable-deploy-fabric-cicd.yml. Uses the Fabric
 # REST API's bulk import endpoint instead of the fabric-cicd Python library.
 # Selected at orchestrator level via the DEPLOY_METHOD repository variable.
 #
@@ -14,10 +14,19 @@
 #   - Every item type in the request payload must support service principals
 #     (the bulk API requires SPN support for ALL items in the request, not just some)
 #
-# Known gaps vs. reusable-deploy-supported.yml (fabric-cicd):
-#   - No parameter.yml find_replace / key_value_replace substitution
-#   - No orphan cleanup (Bulk Import API only supports Create/Update, not Delete)
-#   - No item_type_in_scope filter (deploys everything in repository_directory)
+# Known gaps vs. reusable-deploy-fabric-cicd.yml (fabric-cicd):
+#   The Bulk Import API itself has no parameterization, no value-set
+#   activation, and no delete support. This repo bridges the first two in
+#   scripts/deploy_bulk.py + data/fabric/bulk-parameter.yml. The remaining
+#   gaps are intentionally not implemented:
+#   - No full parameter.yml feature coverage. bulk-parameter.yml supports
+#     find_replace + $items + $workspace + $environment placeholders only.
+#     fabric-cicd's key_value_replace, spark_pool, semantic_model_binding
+#     are not implemented here.
+#   - No orphan cleanup. The Bulk Import API only supports Create/Update,
+#     not Delete; deletes would need a separate per-item DELETE call loop.
+#   - No item_type_in_scope filter. Deploys everything under
+#     repository_directory.
 #
 # API references:
 #   - Bulk import: https://learn.microsoft.com/en-us/rest/api/fabric/core/items/bulk-import-item-definitions(beta)
@@ -60,7 +69,7 @@ jobs:
           python-version: "3.12"
 
       - name: Install dependencies
-        run: pip install requests
+        run: pip install requests PyYAML
 
       - name: Bulk import item definitions
         env:
@@ -69,4 +78,5 @@ jobs:
           AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}
           FABRIC_WORKSPACE_ID: ${{ secrets.FABRIC_WORKSPACE_ID }}
           REPOSITORY_DIRECTORY: ${{ inputs.repository_directory }}
+          ENVIRONMENT: ${{ inputs.environment }}
         run: python scripts/deploy_bulk.py

…/workflows/reusable-deploy-supported.yml …orkflows/reusable-deploy-fabric-cicd.yml.github/workflows/reusable-deploy-supported.yml renamed to .github/workflows/reusable-deploy-fabric-cicd.yml .github/workflows/reusable-deploy-supported.yml renamed to .github/workflows/reusable-deploy-fabric-cicd.yml

@@ -13,7 +13,7 @@
 #   - Fabric Admin must enable "Service principals can use Fabric APIs"
 #   - parameter.yml in repository_directory for environment-specific replacements
 
-name: "Reusable: Deploy Supported Items"
+name: "Reusable: Deploy via fabric-cicd"
 
 on:
   workflow_call:
@@ -38,7 +38,7 @@ permissions:
 
 jobs:
   deploy:
-    name: Deploy supported items (${{ inputs.environment }})
+    name: Deploy via fabric-cicd (${{ inputs.environment }})
     runs-on: ubuntu-latest
     timeout-minutes: 30
     environment: ${{ inputs.environment }}

README.md

@@ -57,7 +57,8 @@ Branch protection (PR required, source-branch restrictions, status checks) is en
 | Document | Description |
 |---|---|
 | [CI/CD Release Options](fabric-cicd-release-options.md) | Evaluates all CI/CD release options for Fabric (Deployment Pipelines, Git-based, Build-based, Hybrid) and recommends the Hybrid approach. Includes a [comparison of fabric-cicd vs the new Bulk Import / Export APIs](fabric-cicd-release-options.md#tooling-within-option-3-fabric-cicd-vs-bulk-apis) (Preview) within Option 3. **Start here** if you're deciding on a strategy. |
-| [Hybrid CI/CD Implementation Guide](fabric-hybrid-cicd-guide.md) | Deep dive into the implementation: workflow structure, configuration strategy, prerequisites, setup steps, and gotchas. |
+| [Hybrid CI/CD Implementation Guide](fabric-hybrid-cicd-guide.md) | Deep dive into the recommended fabric-cicd implementation: workflow structure, configuration strategy, prerequisites, setup steps, and gotchas. |
+| [Bulk CI/CD Implementation Guide](fabric-bulk-cicd-guide.md) | Implementation guide for the alternative Bulk Import API (Preview) deploy path. Covers the gap-bridging workarounds (substitution, value-set activation), the two-deploy decision, extension patterns, and limitations not bridged. |
 | [Development Process](fabric-development-process.md) | How developers work day-to-day: branch-out workflow, the workspace swap script, and PR readiness check. |
 | [CI/CD Governance Considerations](fabric-cicd-governance-considerations.md) | Considerations on identities, RBAC, branch protection, and approval gates for the CI/CD pipeline. Includes pointers to adjacent controls owned outside the pipeline (security/compliance topics). |
 

data/fabric/bulk-parameter.yml

@@ -0,0 +1,33 @@
+# Bulk-deploy substitution rules and post-deploy configuration.
+#
+# This file is read by scripts/deploy_bulk.py to bridge the gaps between the
+# Bulk Import API and what fabric-cicd handles automatically:
+#   - find/replace substitutions (no parameter.yml equivalent in bulk API)
+#   - VariableLibrary active value-set selection (no bulk API equivalent)
+#
+# Format is intentionally distinct from fabric-cicd's parameter.yml — bulk
+# does not (and should not) implement parameter.yml's full DSL. The two
+# config files coexist in this directory: parameter.yml for fabric-cicd,
+# bulk-parameter.yml for bulk. Both are excluded from the bulk request
+# payload by deploy_bulk.py.
+#
+# Dynamic placeholders (resolved at deploy time):
+#   $workspace.$id                    target workspace ID
+#   $items.<Type>.<Name>.$id          ID of an item deployed in phase 1
+#   $environment                      value of the ENVIRONMENT env var (Test/Prod)
+
+substitutions:
+  # Replace dev lakehouse ID in items that reference it.
+  - find: "c185283c-9dd9-4e40-a17c-aa6303e3a2e9"
+    replace_with: "$items.Lakehouse.PatternsLakehouse.$id"
+    item_types: [VariableLibrary, SemanticModel, Notebook]
+
+  # Replace dev workspace ID in items that reference it.
+  - find: "d7270f11-feba-4990-baa6-d45e47f23737"
+    replace_with: "$workspace.$id"
+    item_types: [SemanticModel, Notebook]
+
+variable_library:
+  # Active value-set name to set on the deployed VariableLibrary.
+  # Resolved at deploy time. Set to null to skip activation.
+  active_value_set: "$environment"