DCM Part 4: DCM Projects for Tasks quickstart guide#3210
DCM Part 4: DCM Projects for Tasks quickstart guide#3210sfc-gh-ridasafdar merged 13 commits intoSnowflake-Labs:masterfrom
Conversation
Covers task graphs, finalizer notifications, DMF quality gates, serverless alerts — all managed via DCM Plan & Deploy. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-70a3f9c825cad13635da0c3b27ee90c60f04b9aa
Generated by GitHub Actions |
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-aa440fcf7cf5bdc57f075f02fd46d04865e99883
Generated by GitHub Actions |
- Add OVERLAP_POLICY mention on root task and in feature tour - Add EXECUTE TASK USING CONFIG tip in post-deploy section - TASK_9 row explains retries inherited from root task - Switch finalizer description from HTML to plain-text JSON - Replace custom UDMF + Python UDF with 3 system DMFs - Remove unused alert_schedule from manifest snippet - Add OVERLAP_POLICY and USING CONFIG to Related Resources .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-1515be4ebbe4e254441526381a5ed9936297d695
Generated by GitHub Actions |
…on table - Add EXECUTE MANAGED ALERT grant to pre-deploy snippet - Add email verification section (step 8) - Fix INFORMATION_SCHEMA to fully-qualified in alert snippet - Add callout explaining dual email config (manifest vs alert) - Replace single inbox paragraph with two-email notification table - Add EXECUTE ALERT mention for on-demand testing - Insert select_project.png, plan_results.png, tasks_dag.png assets .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-782c43e930721df20d4a40bd1eeb1e99db8d2f56
Generated by GitHub Actions |
sfc-gh-jsommerfeld
left a comment
There was a problem hiding this comment.
nice job! I see you pimped my original DEMO DAG even more :)
please remove the PrPr callouts. procedure, target-state and alert are coming to PuPr with 10.15 this week
| <!-- ------------------------ --> | ||
| ## Post-Deploy: Stream, Alert, and a Manual Run | ||
|
|
||
| Streams and alerts are not yet supported as DCM `DEFINE` statements. Those pieces live in `scripts/02_post_deploy.sql`. (The DMF attachments themselves are already in DCM — see `expectations.sql` above.) |
There was a problem hiding this comment.
→ "Streams are not yet supported..."
There was a problem hiding this comment.
Will they be supported by 10.15 as well?
- Add Charlie Hammond and Jan Sommerfeld as authors - Add platform category - Rewrite DAG intro (explain "acyclic" in-line) - Soften language: "becomes much easier to maintain as a DCM Project" - Remove LIMITEDACCESS from doc links, drop Private Preview callout - Expand STARTED | SUSPENDED explanation as DCM-only DEFINE TASK extension - Rename DCM project to DCM_TASKS_PROJECT_DEV (avoid clash) - Remove "early-access" from DEFINE PROCEDURE description - Move failed-task alert into DCM project as DEFINE ALERT with STARTED - Update Post-Deploy section title and description (streams only) - Update conclusion bullet to reflect DCM-managed alert - Replace manual DROP cleanup with EXECUTE DCM PROJECT PURGE - Add What's Next CTA to advanced DCM docs .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-cbfbd829cbaceaa033d93e3e7a861817cf2e66b5
Generated by GitHub Actions |
|
@sfc-gh-jsommerfeld I pushed a new version, but let's wait for TARGET STATE to be public before publishing this. In the meantime, tell me if you have any feedback on the language |
.... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-7f0902c320a868e3ce3f644c5dc4e635ade8e977
Generated by GitHub Actions |
|
|
||
| In the [Get Started with Snowflake DCM Projects](https://www.snowflake.com/en/developers/guides/get-started-snowflake-dcm-projects/), [Build Data Pipelines with Snowflake DCM Projects](https://www.snowflake.com/en/developers/guides/build-data-pipelines-with-snowflake-dcm-projects/), and [DCM Projects for Dynamic Tables](https://www.snowflake.com/en/developers/guides/dcm-projects-for-dynamic-tables/) guides, you learned how DCM Projects manage Snowflake infrastructure declaratively — from a single project up to dynamic-table pipelines that evolve without full recomputation. | ||
|
|
||
| In this guide, you'll focus on **Tasks and task graphs** — the orchestration layer that drives scheduled and event-driven work across your pipeline. |
There was a problem hiding this comment.
Tasks and Task graphs
| You'll define a fifteen-task demo graph that shows off every interesting Task feature in one place: | ||
|
|
||
| - **Root task with retries, overlap policy, and graph-level config** pushed into children via `SYSTEM$GET_TASK_GRAPH_CONFIG` | ||
| - **Finalizer task** that emails a plain-text JSON summary of every graph run |
There was a problem hiding this comment.
even when it failed mid-way
| - **Serverless task** and **multi-predecessor task** patterns | ||
| - **Stream-conditional** and **return-value-conditional** child tasks | ||
| - **Failing task with retries** + dependent that gets skipped | ||
| - **DCM-managed target state** — a task deployed as `SUSPENDED` that never runs |
There was a problem hiding this comment.
I'd prefer to focus on the positive here 😉 target-state allows to deploy Tasks STARTED.
| If you don't already have one handy for running DCM itself: | ||
|
|
||
| ```sql | ||
| CREATE WAREHOUSE IF NOT EXISTS dcm_wh |
There was a problem hiding this comment.
I guess we can skip this one, if the Tasks run on the WH_4 below?
- DEFINE ALERT stays in sources/definitions/alerts.sql (without STARTED) - Alert deploys suspended; 02_post_deploy.sql resumes it once - Clarify alert vs finalizer as complementary (per Jan's review) - Update conclusion bullet to say 'task in the database' (broader scope) .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-1bbe75cf31aeb8ab06fc5bca881210e5684add4a
Generated by GitHub Actions |
.... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-f3d38b5826d0d62aa99deff1f20664244b709dbc
Generated by GitHub Actions |
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-43a0fd8a3f31d23bfc9db6e2a9e51da6559c5dc5
Generated by GitHub Actions |
… are now PuPr) Addresses Jan's review comment: 'please remove. they are both promoted to PuPr this week'. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-2f02d6b26bd0fb159c1f23552f462b51f8ab4980
Generated by GitHub Actions |
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-c23a94db68b36c86b5d4c21f2ae685742cedf034
Generated by GitHub Actions |
…rget-state) These three edits were lost during an earlier stash-pop cycle: - "Tasks and task graphs" -> "Tasks and Task graphs" (line 17) - Finalizer bullet: "... even when it failed mid-way" - Positive target-state bullet framing .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-0519d02f3e27ec40a0f03e7c89b67bb907c66f25
Generated by GitHub Actions |
🔗 Staging Preview ReadyQuickstart: 👉 Preview URL: https://publish-p57963-e462098.adobeaemcloud.com/en/developers/guides/dcm-projects-for-tasks-6e3631800b1399117ce5389ad5ac89d8465a3aa1
Generated by GitHub Actions |
Covers task graphs, finalizer notifications, DMF quality gates, serverless alerts — all managed via DCM Plan & Deploy.
.... Generated with Cortex Code