diff --git a/.github/workflows/check-pr-ready.yml b/.github/workflows/check-pr-ready.yml index 482816b..655d854 100644 --- a/.github/workflows/check-pr-ready.yml +++ b/.github/workflows/check-pr-ready.yml @@ -3,7 +3,6 @@ name: Check PR Ready on: pull_request: branches: [dev] - paths: ["data/fabric/**"] permissions: contents: read diff --git a/.github/workflows/enforce-promotion-path.yml b/.github/workflows/enforce-promotion-path.yml new file mode 100644 index 0000000..79b2769 --- /dev/null +++ b/.github/workflows/enforce-promotion-path.yml @@ -0,0 +1,38 @@ +name: Enforce Promotion Path + +on: + pull_request: + branches: [test, main] + +permissions: + contents: read + +jobs: + enforce-promotion-path: + name: Enforce promotion path + runs-on: ubuntu-latest + timeout-minutes: 1 + steps: + - name: Verify PR source branch + run: | + BASE="${{ github.base_ref }}" + HEAD="${{ github.head_ref }}" + echo "PR: $HEAD -> $BASE" + case "$BASE" in + test) + if [[ "$HEAD" != "dev" ]]; then + echo "::error::PRs into 'test' must originate from 'dev'. Got '$HEAD'." + exit 1 + fi + ;; + main) + if [[ "$HEAD" != "test" ]]; then + echo "::error::PRs into 'main' must originate from 'test'. Got '$HEAD'." + exit 1 + fi + ;; + *) + echo "Branch '$BASE' has no promotion-path restriction." + ;; + esac + echo "Source branch '$HEAD' is valid for target '$BASE'." diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml index 8f5b13a..4d088ec 100644 --- a/.github/workflows/run-tests.yml +++ b/.github/workflows/run-tests.yml @@ -2,9 +2,6 @@ name: Run Tests on: pull_request: - paths: - - "scripts/**" - - "tests/**" permissions: contents: read diff --git a/README.md b/README.md index ac975aa..c927965 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,29 @@ # Microsoft Fabric SDLC Patterns -A reference implementation for CI/CD in Microsoft Fabric, demonstrating how to version-control, deploy, and manage Fabric workspace items across dev, test, and production environments using GitHub Actions and the [fabric-cicd](https://microsoft.github.io/fabric-cicd) Python library. +A reference implementation and solution accelerator for the developer workflow and CI/CD pipeline in Microsoft Fabric, demonstrating how to choose a release strategy, implement deployments, work day-to-day on feature branches, and govern the pipeline for dev, test, and production using GitHub Actions and the [fabric-cicd](https://microsoft.github.io/fabric-cicd) Python library. Both the developer workflow and the deployment pipeline are fully implemented end-to-end so the repo runs as a complete reference, not isolated examples. + +*Based on field experience with Microsoft Fabric customers and partners. Opinions expressed here are my own and do not represent Microsoft's official guidance.* --- ## Who is this for? -Teams and engineers who need to establish a reliable software development lifecycle (SDLC) for Microsoft Fabric — including automated deployments, environment-specific configuration, and Git-based version control for Fabric items. +Engineers and platform teams responsible for getting Microsoft Fabric workloads from a developer's laptop to production safely and repeatably — covering the developer workflow, the deployment pipeline itself, and the governance layered on top. + +Architects and decision-makers evaluating Fabric will also find the [CI/CD Release Options](fabric-cicd-release-options.md) and [Governance Considerations](fabric-cicd-governance-considerations.md) useful for understanding the operating model before committing. --- ## Architecture ``` +Feature branch (feature/*) + │ + │ PR → dev branch + ▼ Git repo (dev branch) │ - │ PR merge → test branch + │ PR merge → test branch (source must be dev) ▼ ┌──────────────────────────────────────────────┐ │ deploy-test.yml │ @@ -25,7 +33,7 @@ Git repo (dev branch) │ └─ Fabric REST API: run notebook │ └──────────────────────────────────────────────┘ │ - │ PR merge → main branch + │ PR merge → main branch (source must be test) ▼ ┌──────────────────────────────────────────────┐ │ deploy-prod.yml │ @@ -36,6 +44,8 @@ Git repo (dev branch) └──────────────────────────────────────────────┘ ``` +Branch protection (PR required, source-branch restrictions, status checks) is enforced by GitHub branch rulesets and the [enforce-promotion-path.yml](.github/workflows/enforce-promotion-path.yml) workflow — see the [Governance Considerations](fabric-cicd-governance-considerations.md). + ![Hybrid Recommendation Flow](assets/hybrid-recommendation-flow.svg) --- @@ -47,6 +57,7 @@ Git repo (dev branch) | [CI/CD Release Options](fabric-cicd-release-options.md) | Evaluates all CI/CD release options for Fabric (Deployment Pipelines, Git-based, Build-based, Hybrid) and recommends the Hybrid approach. **Start here** if you're deciding on a strategy. | | [Hybrid CI/CD Implementation Guide](fabric-hybrid-cicd-guide.md) | Deep dive into the implementation: workflow structure, configuration strategy, prerequisites, setup steps, and gotchas. | | [Development Process](fabric-development-process.md) | How developers work day-to-day: branch-out workflow, the workspace swap script, and PR readiness check. | +| [CI/CD Governance Considerations](fabric-cicd-governance-considerations.md) | Considerations on identities, RBAC, branch protection, and approval gates for the CI/CD pipeline. Includes pointers to adjacent controls owned outside the pipeline (security/compliance topics). | --- @@ -89,17 +100,17 @@ When designing your development and CI/CD processes, identify which items in you 2. **Three Fabric Workspaces** — Dev (Git-connected), Test, and Prod 3. **Service Principal** — With Contributor role on Test and Prod workspaces 4. **GitHub Environments** — `Test` and `Prod` with environment-scoped secrets -5. **Fabric Admin Setting** — "Service principals can use Fabric APIs" enabled +5. **Fabric Admin Setting** — Service principal access to Fabric APIs enabled in the Fabric Admin portal under Developer settings (see [developer tenant settings](https://learn.microsoft.com/en-us/fabric/admin/service-admin-portal-developer)) ### Setup 1. Create a Service Principal and add it as Contributor on Test and Prod workspaces -2. Create GitHub Environments (`Test`, `Prod`) with secrets: `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `FABRIC_WORKSPACE_ID` +2. Create GitHub Environments (`Test`, `Prod`) with secrets: `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `FABRIC_WORKSPACE_ID` *(this demo uses a client secret for simplicity; for production, evaluate [GitHub OIDC federation](fabric-cicd-governance-considerations.md#identity-model--pick-the-right-identity-for-the-job) to remove the stored secret)* 3. Connect the Dev workspace to the `dev` branch via Fabric Git integration (folder: `data/fabric/`) 4. Create `dev`, `test`, and `main` branches 5. Develop on `dev`, merge to `test` (triggers Test deploy), merge to `main` (triggers Prod deploy) -For detailed setup instructions, see the [Implementation Guide](fabric-hybrid-cicd-guide.md#prerequisites--setup). +For detailed setup instructions, see the [Implementation Guide](fabric-hybrid-cicd-guide.md#prerequisites--setup). For branch-protection rulesets, deploy-time approvals, and the source-branch promotion path enforced in this repo, see the [Governance Considerations](fabric-cicd-governance-considerations.md). --- diff --git a/assets/development-swap-to-dev-flow.svg b/assets/development-swap-to-dev-flow.svg index 966ca77..44dbabb 100644 --- a/assets/development-swap-to-dev-flow.svg +++ b/assets/development-swap-to-dev-flow.svg @@ -32,8 +32,7 @@ GitHub Actions - validate-branch- - env.yml + check-pr-ready.yml diff --git a/fabric-cicd-governance-considerations.md b/fabric-cicd-governance-considerations.md new file mode 100644 index 0000000..898bba9 --- /dev/null +++ b/fabric-cicd-governance-considerations.md @@ -0,0 +1,75 @@ +# Fabric CI/CD Governance Considerations + +> Reference implementation: https://github.com/michaeldeongreen/microsoft-fabric-sdlc-patterns + +Considerations for partners and customers designing governance around CI/CD pipelines for Microsoft Fabric. The topics below are scoped to the CI/CD pipeline itself — workspace setup, identities used by the pipeline, branch protection, deployment gating, and audit. Adjacent controls that sit outside the CI/CD pipeline (network controls, data classification, encryption, runtime data-access policies) are listed at the end under "Adjacent controls owned outside the CI/CD pipeline" — they are part of any mature production Fabric deployment but owned by your platform / security team rather than the CI/CD pipeline owner. + +This guide layers governance controls on top of the implementation pattern in the [Hybrid CI/CD Implementation Guide](fabric-hybrid-cicd-guide.md). Read that doc first if you don't already have a working pipeline. + +## Identity model — pick the right identity for the job + +Three distinct identities show up in a Fabric solution. Confusing them is a common source of over-permissioned deployments. + +| Identity | Purpose | Created / managed by | Authentication | +|---|---|---|---| +| **CI/CD service principal** | Deploys Fabric items (notebooks, semantic models, pipelines) from GitHub Actions via `fabric-cicd` | App owner registers it in Microsoft Entra; workspace admin grants it Contributor on the target workspace | Currently a client secret in GitHub Environment secrets; consider GitHub OIDC federation | +| **Fabric Workspace Identity** | The workspace authenticating *outbound* to firewall-protected ADLS Gen2, OneLake shortcuts, etc. — not used by CI/CD | Workspace admin creates it in workspace settings; Fabric auto-manages credentials | Managed by Fabric — no secret to handle ([Workspace identity](https://learn.microsoft.com/en-us/fabric/security/workspace-identity)) | +| **Runtime workload UAMI** | Container Apps / Functions / AKS calling Fabric REST APIs at runtime (not deploying items) | Platform admin creates a user-assigned managed identity per workload, per environment | Secretless — managed by Azure ([managed identity overview](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview)) | + +> **OIDC for the CI/CD SP — worth evaluating.** Microsoft's [Well-Architected guidance](https://learn.microsoft.com/en-us/azure/well-architected/security/identity-access#protect-nonidentity-based-secrets) is "avoid using secrets when you can." With [GitHub OIDC](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-azure), the workflow exchanges a short-lived GitHub-issued token for an Azure access token — no client secret stored in GitHub. The Entra trust policy can bind to specific repo + branch + environment claims, so a workflow from a feature branch cannot assume the Prod SP. This eliminates secret rotation and the leaked-secret failure mode. The demo currently uses client secrets for simplicity; OIDC is the direction to head for production. + +## Security across workspaces + +Environment isolation, RBAC, and identity boundaries. + +| Concern area | Guidance | +|---|---| +| Environment isolation | | +| Tenant settings (Fabric admin) | | +| Access control (RBAC) | | +| CI/CD identity (deploys artifacts) | | +| Runtime workload identity (Container Apps / Functions / AKS calling Fabric) | | + +## Manual approvals and revision processes + +GitHub-native gates: code-review approval before merge (PR / branch protection), deploy-time approval before the workflow runs (Environment protection), and an audit trail across both. + +| Concern area | Guidance | +|---|---| +| PR checks for revisions (gate before merge) | | +| Deploy-time approval (gate before the workflow runs) | | +| Audit trail and revision history | | +| Rollback strategy (think about it before you need it) | | + +## Governance and compliance + +Mapping the controls above to the three governance asks: audit, separation of duties, and enterprise standards. + +| Concern area | Guidance | +|---|---| +| Audit requirements | | +| Separation of duties | | +| Enterprise governance standards | | + +## Adjacent controls owned outside the CI/CD pipeline + +These controls go beyond the CI/CD pipeline but are part of any mature Fabric production deployment. The right mix depends on your risk profile, regulatory posture, and licensing — some controls (e.g., Microsoft Purview, Customer-Managed Keys, Private Links) require specific Fabric / Microsoft 365 / Entra ID SKUs and may not be available in every tenant. They're listed separately because they sit with your platform / security / information-protection teams rather than the CI/CD pipeline owner. Surface them early in the conversation so the right specialists can scope and own them before you go to Prod. + +| Area | What to raise with the owning team | +|---|---| +| Network controls | Tenant-level access can be hardened with [Microsoft Entra Conditional Access for Fabric](https://learn.microsoft.com/en-us/fabric/security/security-conditional-access) (MFA, IP, device compliance) and [Private Links](https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview) to block public access. For firewalled ADLS Gen2, see [Trusted Workspace Access](https://learn.microsoft.com/en-us/fabric/security/security-trusted-workspace-access). | +| Encryption & data residency | OneLake [encrypts data at rest by default](https://learn.microsoft.com/en-us/fabric/security/security-fundamentals#data-at-rest). For regulated workloads, evaluate [workspace customer-managed keys](https://learn.microsoft.com/en-us/fabric/security/workspace-customer-managed-keys) and [multi-geo capacities](https://learn.microsoft.com/en-us/fabric/admin/service-admin-premium-multi-geo). Customer Lockbox is also listed in the [Fabric security capabilities](https://learn.microsoft.com/en-us/fabric/security/security-overview#capabilities) — confirm scope with your security team. | +| Workspace-level audit | Fabric workspace activity (item access, sharing changes, role assignments) is captured in the M365 unified audit log, surfaced via the Microsoft Purview portal. Retention and API access depend on your M365 / Audit licensing tier ([track user activities](https://learn.microsoft.com/en-us/fabric/admin/track-user-activities)). Engage your compliance / information-protection team to confirm what's enabled in your tenant. | +| Data classification & DLP | [Sensitivity labels](https://learn.microsoft.com/en-us/fabric/governance/information-protection), [DLP policies](https://learn.microsoft.com/en-us/power-bi/enterprise/service-security-dlp-policies-for-power-bi-overview), and [Insider Risk Management indicators for Fabric](https://learn.microsoft.com/en-us/purview/insider-risk-management-settings-policy-indicators#microsoft-fabric-indicators) are configured in [Microsoft Purview](https://learn.microsoft.com/en-us/fabric/governance/microsoft-purview-fabric) by your information-protection team. One CI/CD-relevant gotcha worth flagging: sensitivity labels are stripped on Git export ([reference](https://learn.microsoft.com/en-us/fabric/admin/git-integration-admin-settings#users-can-export-workspace-items-with-applied-sensitivity-labels-to-git-repositories)). | +| Fine-grained data access | Workspace roles are not the whole story — also configure [OneLake security](https://learn.microsoft.com/en-us/fabric/onelake/security/get-started-onelake-security) (table/folder grants) and [row-level / column-level / object-level security](https://learn.microsoft.com/en-us/fabric/security/permission-model#compute-permissions) on warehouses and semantic models. These are owned by data engineers, not the CI/CD pipeline. Note that deploying an item definition can overwrite security rules embedded in that item. | +| Repo & supply-chain hardening | Standard hygiene for any repo that holds deployment credentials and scripts: [GitHub Secret Scanning + push protection](https://docs.github.com/en/code-security/secret-scanning/introduction/about-secret-scanning), [CodeQL code scanning](https://docs.github.com/en/code-security/code-scanning/introduction-to-code-scanning/about-code-scanning), [Dependabot](https://docs.github.com/en/code-security/dependabot/dependabot-alerts/about-dependabot-alerts), and [Dependency Review on PRs](https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-dependency-review). Configure at the org or repo level. | + +## Related Microsoft references + +- [Fabric security overview](https://learn.microsoft.com/en-us/fabric/security/security-overview) +- [Fabric permission model](https://learn.microsoft.com/en-us/fabric/security/permission-model) +- [Fabric CI/CD overview](https://learn.microsoft.com/en-us/fabric/cicd/cicd-overview) and [choose the best CI/CD workflow](https://learn.microsoft.com/en-us/fabric/cicd/manage-deployment) (this repo implements Option 2: Git-based deployments using a build environment) +- [Azure Well-Architected: Identity and access management](https://learn.microsoft.com/en-us/azure/well-architected/security/identity-access) +- [Managed identity best practices (UAMI vs SAMI)](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/managed-identity-best-practice-recommendations) +- [GitHub OIDC in Azure](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-azure) +- [fabric-cicd library](https://microsoft.github.io/fabric-cicd) diff --git a/fabric-cicd-release-options.md b/fabric-cicd-release-options.md index 826af4d..84015c3 100644 --- a/fabric-cicd-release-options.md +++ b/fabric-cicd-release-options.md @@ -189,6 +189,7 @@ With this option, all deployments originate from the **Git repository**. Each st - **No deployment rules** — Configuration changes between stages must be handled via Fabric APIs post-deployment or through parameterization in the item definitions, since there are no Deployment Pipeline rules. - **PR discipline required** — The promotion process depends entirely on PR quality. Incomplete or poorly reviewed PRs directly affect the target stage workspace. - **Cherry-pick complexity** — Moving a subset of changes between stages often requires release branches with cherry-picked commits rather than a simple full merge. +- **Reliance on Fabric Git integration in upper environments** — Every stage's workspace (including Test and Prod) syncs via Fabric Git integration. Some customers and partners are uncomfortable putting that mechanism on the production deployment path. Known behaviors include "ghost commits" — semantically-insignificant changes the AS engine, line-break normalization, or service-side serialization re-introduce when committing — and the Source Control panel showing uncommitted-change drift the user didn't make. See [sync and commit limitations](https://learn.microsoft.com/en-us/fabric/cicd/git-integration/git-integration-process#sync-and-commit-limitations). ### Option 3 – Git-based with Build Environments diff --git a/fabric-hybrid-cicd-guide.md b/fabric-hybrid-cicd-guide.md index 0b77b15..6c73391 100644 --- a/fabric-hybrid-cicd-guide.md +++ b/fabric-hybrid-cicd-guide.md @@ -1,4 +1,4 @@ -# Microsoft Fabric SDLC Patterns — CI/CD Implementation +# Hybrid CI/CD Implementation Guide This repository implements the **Hybrid CI/CD recommendation** for Microsoft Fabric using **fabric-cicd**. It demonstrates how to deploy Fabric workspace items (Notebooks, Lakehouses, Variable Libraries, Semantic Models, Reports, Ontologies, Data Agents) across environments using GitHub Actions. @@ -75,7 +75,9 @@ microsoft-fabric-sdlc-patterns/ │ ├── etl-prod.yml # Triggers after deploy-prod succeeds │ ├── reusable-deploy-supported.yml # Template: fabric-cicd deployment │ ├── reusable-fabric-etl.yml # Template: run Notebook via Fabric REST API -│ └── check-pr-ready.yml # PR check: blocks feature IDs from merging to dev +│ ├── check-pr-ready.yml # PR check: blocks feature IDs from merging to dev +│ ├── run-tests.yml # PR check: runs pytest when scripts/tests change +│ └── enforce-promotion-path.yml # PR check: enforces dev→test→main source-branch promotion ├── data/ │ └── fabric/ # Fabric item definitions (repository_directory) │ ├── parameter.yml # fabric-cicd deploy-time parameterization @@ -202,7 +204,7 @@ az ad sp create-for-rbac --name "SPN-Microsoft-Fabric-SDLC-Patterns" \ - Add the SPN as **Contributor** on both Test and Prod workspaces (Workspace → Manage access → Add people or groups) - Contributor is the minimum required role per the [Fabric Create Item API](https://learn.microsoft.com/en-us/rest/api/fabric/core/items/create-item) documentation -> **Important:** A Fabric Admin must enable **"Service principals can use Fabric APIs"** in the Fabric Admin portal → Tenant settings → Developer settings. +> **Important:** A Fabric Admin must enable service principal access to Fabric APIs in the Fabric Admin portal under Developer settings, scoped to a security group containing only your CI/CD SPs. See [developer tenant settings](https://learn.microsoft.com/en-us/fabric/admin/service-admin-portal-developer) and the [Governance Considerations](fabric-cicd-governance-considerations.md) for details. ### 4. GitHub Environments diff --git a/scripts/workspace_swap.py b/scripts/workspace_swap.py index 2dc0f3f..dc6eb25 100644 --- a/scripts/workspace_swap.py +++ b/scripts/workspace_swap.py @@ -52,8 +52,8 @@ ALLOWED_VALUE_SETS = {"Test", "Prod"} # ── Item type registry ───────────────────────────────────────────────────── -# Each entry declares how a Fabric item type participates in branch-env -# management. Generic functions iterate this list instead of hard-coding +# Each entry declares how a Fabric item type participates in workspace +# swapping. Generic functions iterate this list instead of hard-coding # per-type logic, so adding a new item type is a single dict addition. # # name – display name for logging @@ -61,7 +61,7 @@ # needs_rewrite – True → dev↔feature ID replacement during bootstrap/reset # id_keys – which dev IDs to validate ("workspace", "lakehouse", or both) # content_filter – optional predicate on file text; None means process all -ITEM_TYPES: list[dict[str, str | list[str] | bool | list[str] | Callable[[str], bool] | None]] = [ +ITEM_TYPES: list[dict[str, str | list[str] | bool | Callable[[str], bool] | None]] = [ { "name": "SemanticModel", "file_patterns": ["*.SemanticModel/definition/expressions.tmdl"],