Skip to content

Commit cb39a3a

Browse files
michaeldeongreenmichaeldeongreen
authored
Simplify CI/CD: remove skeleton workflows, add initial deployment guide (#4)
* feat: bootstrap feature-ontology workspace bindings - Create value set for michaeldeongreen-feature-ontology - Repoint semantic model and notebooks to feature workspace/lakehouse - Update settings.json valueSetsOrder * Fabric UI Ghost commit. * feat: print variable library IDs in patients notebook for validation * Created Ontology and Data Agent. * feat: extensible item type registry for branch env management Refactor branch_env.py to use a registry-driven architecture so new Fabric item types (Ontology, DataAgent, etc.) can be added with a single config entry instead of per-type functions. Changes: - Replace 4 per-type functions with generic repoint_items() and validate_no_ids() that iterate the ITEM_TYPES registry - Add Ontology and DataAgent to registry (validation-only, no rewrite needed — they use portable logicalIds, not actual dev IDs) - Add --validate CLI mode for CI use, eliminating shell/Python duplication in the workflow - Simplify validate-branch-env.yml to a single Python call - Add run-tests.yml workflow for pytest on PRs to any branch - Add 25 unit tests covering all registry-driven functions - Add Ontology find_replace entry to parameter.yml for CI/CD - Document Item Type Reference table in development process guide * Enable Ontology and DataAgent deployment via fabric-cicd * Restore Ontology to Phase 1 — fabric-cicd requires it before DataAgent * Reset dev IDs and update documentation for Ontology/DataAgent deployment * Ghost check in. * Phantom commit in Fabric UI. * Remove skeleton workflows, simplify deploy to single job, add initial deployment guide with Ontology workaround
1 parent a732cae commit cb39a3a

37 files changed

Lines changed: 1321 additions & 493 deletions

‎.github/copilot-instructions.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Project Guidelines
2+
3+
## Overview
4+
5+
Reference implementation for CI/CD in Microsoft Fabric. Demonstrates version control, deployment, and environment-specific configuration for Fabric workspace items using GitHub Actions and [fabric-cicd](https://microsoft.github.io/fabric-cicd).
6+
7+
## Project Layout
8+
9+
```
10+
scripts/ Python CLI scripts (stdlib-only, Python 3.10+)
11+
tests/ pytest unit tests
12+
data/fabric/ Fabric item definitions (git-synced from Fabric workspaces)
13+
.github/workflows/ GitHub Actions CI/CD pipelines
14+
.github/instructions/ Path-specific Copilot instructions (Python, Actions)
15+
```
16+
17+
## Build and Test
18+
19+
```bash
20+
pip install -r requirements-dev.txt
21+
python -m pytest tests/ -v
22+
```
23+
24+
No build step — scripts are standalone CLI tools, not an installable package. `pyproject.toml` configures pytest only (`pythonpath = ["scripts"]`).
25+
26+
## Key Scripts
27+
28+
- `scripts/branch_env.py` — Manages feature branch workspace bindings (bootstrap, reset, validate). Uses an **item type registry** pattern — new Fabric item types are added as config entries, not new functions.
29+
- `scripts/branch_env.py --validate` — CI check for PR readiness (dev IDs present, no stray value sets).
30+
31+
## Fabric Item Types
32+
33+
Items fall into two categories based on how they reference environment resources:
34+
35+
- **Actual IDs** (SemanticModel, Notebook): Embed real workspace/lakehouse GUIDs. Must be rewritten by `branch_env.py` for feature branches and reverted before PR.
36+
- **Logical IDs** (Ontology, DataAgent): Reference items via `.platform` `logicalId`, resolved by Fabric at runtime. Portable across Branch Out workspaces — no rewriting needed.
37+
38+
## CI/CD Configuration
39+
40+
- `data/fabric/parameter.yml` — Deploy-time `find_replace` rules for fabric-cicd. Maps dev IDs to dynamic placeholders (`$workspace.$id`, `$items.Lakehouse.*.$id`).
41+
- `data/fabric/Patterns_Variables.VariableLibrary/` — Runtime configuration via value sets (Test, Prod, feature branches).
42+
43+
## Conventions
44+
45+
- Path-specific instructions in `.github/instructions/` govern Python and Actions code style — follow them.
46+
- Never commit notebook META blocks containing internal metadata (security).
47+
- Validate GUIDs from external sources before using them.
48+
- Pin GitHub Actions to commit SHAs, not version tags.
49+
- All file I/O must specify `encoding="utf-8"`.
50+
51+
## Workflows
52+
53+
| Workflow | Trigger | Purpose |
54+
|----------|---------|---------|
55+
| `validate-branch-env.yml` | PR to `dev` | Blocks merge if dev IDs not restored |
56+
| `run-tests.yml` | PR (any branch) | Runs pytest when scripts/tests change |
57+
58+
## Documentation
59+
60+
See `fabric-development-process.md` for the Branch Out workflow, item type reference table, and step-by-step bootstrap/reset guides. See `fabric-hybrid-cicd-guide.md` for the deployment architecture.

‎.github/workflows/deploy-prod.yml‎

Lines changed: 7 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,14 @@
11
# Orchestrator workflow for deploying Fabric items to the Prod workspace.
22
#
3-
# Same sandwich pattern as deploy-test.yml but targeting the prod environment.
3+
# Deploys all supported items via fabric-cicd. The reusable workflow handles
4+
# phased deployment internally (Lakehouse + Ontology first, then everything
5+
# else) to satisfy fabric-cicd dependency resolution.
6+
#
47
# The `prod` GitHub Environment should have protection rules configured
58
# (e.g., required reviewers, branch policy restricting to main only).
69
#
7-
# Jobs run sequentially via `needs:`. The ETL workflow (etl-prod.yml) triggers
8-
# automatically via workflow_run once all 3 jobs complete successfully.
10+
# The ETL workflow (etl-prod.yml) triggers automatically via workflow_run
11+
# once deployment completes successfully.
912

1013
name: Deploy to Prod
1114

@@ -23,21 +26,5 @@ jobs:
2326
uses: ./.github/workflows/reusable-deploy-supported.yml
2427
with:
2528
environment: Prod
26-
item_type_in_scope: '["Lakehouse", "VariableLibrary", "Notebook", "SemanticModel", "Report"]'
27-
secrets: inherit
28-
29-
promote-unsupported:
30-
name: Promote unsupported items
31-
needs: deploy-supported
32-
uses: ./.github/workflows/reusable-deploy-unsupported.yml
33-
with:
34-
environment: Prod
35-
secrets: inherit
36-
37-
deploy-supported-dependent:
38-
name: Deploy dependent supported items
39-
needs: promote-unsupported
40-
uses: ./.github/workflows/reusable-deploy-supported-dependent-on-unsupported.yml
41-
with:
42-
environment: Prod
29+
item_type_in_scope: '["Lakehouse", "Ontology", "VariableLibrary", "Notebook", "SemanticModel", "Report", "DataAgent"]'
4330
secrets: inherit

‎.github/workflows/deploy-test.yml‎

Lines changed: 6 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,11 @@
11
# Orchestrator workflow for deploying Fabric items to the Test workspace.
22
#
3-
# Implements the "sandwich" pattern from the hybrid CI/CD recommendation:
4-
# Job 1: Deploy supported items via fabric-cicd
5-
# Job 2: Promote unsupported items (e.g., Ontologies) via Deployment Pipelines (skeleton)
6-
# Job 3: Deploy supported items that depend on unsupported items (skeleton)
3+
# Deploys all supported items via fabric-cicd. The reusable workflow handles
4+
# phased deployment internally (Lakehouse + Ontology first, then everything
5+
# else) to satisfy fabric-cicd dependency resolution.
76
#
8-
# Jobs run sequentially via `needs:`. The ETL workflow (etl-test.yml) triggers
9-
# automatically via workflow_run once all 3 jobs complete successfully.
7+
# The ETL workflow (etl-test.yml) triggers automatically via workflow_run
8+
# once deployment completes successfully.
109

1110
name: Deploy to Test
1211

@@ -24,21 +23,5 @@ jobs:
2423
uses: ./.github/workflows/reusable-deploy-supported.yml
2524
with:
2625
environment: Test
27-
item_type_in_scope: '["Lakehouse", "VariableLibrary", "Notebook", "SemanticModel", "Report"]'
28-
secrets: inherit
29-
30-
promote-unsupported:
31-
name: Promote unsupported items
32-
needs: deploy-supported
33-
uses: ./.github/workflows/reusable-deploy-unsupported.yml
34-
with:
35-
environment: Test
36-
secrets: inherit
37-
38-
deploy-supported-dependent:
39-
name: Deploy dependent supported items
40-
needs: promote-unsupported
41-
uses: ./.github/workflows/reusable-deploy-supported-dependent-on-unsupported.yml
42-
with:
43-
environment: Test
26+
item_type_in_scope: '["Lakehouse", "Ontology", "VariableLibrary", "Notebook", "SemanticModel", "Report", "DataAgent"]'
4427
secrets: inherit

‎.github/workflows/etl-test.yml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,7 @@
22
#
33
# Uses workflow_run to decouple deploy and ETL as separate workflow concerns.
44
# The `if` condition ensures ETL only runs when the deploy workflow succeeded —
5-
# if any sandwich job fails, the deploy workflow is marked as failed and this
6-
# workflow skips execution.
5+
# if the deploy workflow fails, this workflow skips execution.
76
#
87
# ETL_ITEM_ID is no longer needed — the notebook is resolved by name at runtime.
98

‎.github/workflows/reusable-deploy-supported-dependent-on-unsupported.yml‎

Lines changed: 0 additions & 49 deletions
This file was deleted.

‎.github/workflows/reusable-deploy-supported.yml‎

Lines changed: 24 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
# to publish all supported item definitions from the Git repo to the target
55
# Fabric workspace, then removes any orphaned items no longer in the repo.
66
#
7-
# Called by: deploy-test.yml, deploy-prod.yml (Job 1 of the sandwich pattern)
7+
# Called by: deploy-test.yml, deploy-prod.yml
88
#
99
# Prerequisites:
1010
# - GitHub Environment secrets: AZURE_TENANT_ID, AZURE_CLIENT_ID,
@@ -52,7 +52,7 @@ jobs:
5252
python-version: "3.12"
5353

5454
- name: Install dependencies
55-
run: pip install fabric-cicd azure-identity
55+
run: pip install "fabric-cicd>=1.0.0,<2.0.0" azure-identity
5656

5757
# fabric-cicd reads parameter.yml from the repository_directory root
5858
# and applies environment-specific find_replace / key_value_replace rules
@@ -90,27 +90,35 @@ jobs:
9090
workspace_id = os.environ['FABRIC_WORKSPACE_ID']
9191
environment = os.environ['ENVIRONMENT']
9292
93-
# Phase 1: Deploy Lakehouse first so it exists in the target workspace.
94-
# fabric-cicd's \$items dynamic variables resolve by querying the live
95-
# workspace during parameterization. On first deployment to an empty
96-
# workspace, the Lakehouse must exist before Variable Library and
97-
# Semantic Model can resolve \$items.Lakehouse.PatternsLakehouse.\$id.
98-
# On subsequent deployments, the Lakehouse already exists and this
99-
# phase simply updates it (idempotent).
100-
print('Phase 1: Deploying Lakehouse...')
101-
lakehouse_ws = FabricWorkspace(
93+
# Deployment uses a 2-phase approach to satisfy item dependencies.
94+
# fabric-cicd caches workspace state once at the start of each
95+
# publish_all_items() call, so items deployed within the same call
96+
# are not visible to later items' logicalId or \$items resolution.
97+
#
98+
# Phase 1: Lakehouse + Ontology
99+
# - Lakehouse must exist so parameter.yml \$items.Lakehouse rules resolve.
100+
# - Ontology must exist so DataAgent's logicalId reference resolves.
101+
# Phase 2: All remaining items (DataAgent, Notebook, SemanticModel, etc.).
102+
#
103+
# On subsequent deployments all items already exist and phases are
104+
# idempotent — they simply update in place.
105+
106+
phase1_types = ['Lakehouse', 'Ontology']
107+
108+
print('Phase 1: Deploying Lakehouse + Ontology...')
109+
phase1_ws = FabricWorkspace(
102110
repository_directory=repo_dir,
103111
workspace_id=workspace_id,
104112
environment=environment,
105113
token_credential=credential,
106-
item_type_in_scope=['Lakehouse'],
114+
item_type_in_scope=phase1_types,
107115
)
108-
publish_all_items(lakehouse_ws)
116+
publish_all_items(phase1_ws)
109117
110118
# Phase 2: Deploy all remaining item types.
111-
# The Lakehouse now exists, so \$items references resolve correctly.
112-
remaining = [t for t in (item_type_in_scope or []) if t != 'Lakehouse'] or None
113-
print('Phase 2: Deploying ' + str(remaining or 'all remaining') + '...')
119+
phase1_set = set(phase1_types)
120+
remaining = [t for t in (item_type_in_scope or []) if t not in phase1_set] or None
121+
print('Phase 2: Deploying remaining items: ' + str(remaining or 'all') + '...')
114122
workspace = FabricWorkspace(
115123
repository_directory=repo_dir,
116124
workspace_id=workspace_id,

‎.github/workflows/reusable-deploy-unsupported.yml‎

Lines changed: 0 additions & 43 deletions
This file was deleted.

‎.github/workflows/run-tests.yml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
name: Run Tests
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- "scripts/**"
7+
- "tests/**"
8+
9+
permissions:
10+
contents: read
11+
12+
jobs:
13+
test:
14+
name: Run unit tests
15+
runs-on: ubuntu-latest
16+
timeout-minutes: 5
17+
18+
steps:
19+
- name: Checkout repository
20+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
21+
22+
- name: Set up Python
23+
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405
24+
with:
25+
python-version: "3.12"
26+
27+
- name: Install test dependencies
28+
run: pip install -r requirements-dev.txt
29+
30+
- name: Run tests
31+
run: pytest tests/ -v

0 commit comments

Comments
 (0)