YET-1269: /manage-mac skill — CLI-first MaC YAML authoring by wojtekdeka · Pull Request #88 · monte-carlo-data/mc-agent-toolkit

wojtekdeka · 2026-05-20T08:49:55Z

Summary

Adds the /manage-mac skill for Monitors-as-Code YAML authoring — create, edit, validate, and import MaC YAML files from any MCP-compatible editor (Claude Code, Cursor, etc.).

CLI-first, 3-tier approach:

Tier	Tool	Used for
1 — CLI	`montecarlo` binary	Validate (`compile`), apply, import (`convert-to-mac`, `export`)
2 — MCP	Monte Carlo MCP server	Author YAML shapes via `dry_run=True`, resolve table metadata
3 — Manual	No external tools	Validate field names/enums/types as last resort

The skill checks CLI availability at startup and prompts the user to install/configure if missing, then gracefully falls back through tiers.

Four workflows:

Create — gather context, resolve table metadata via MCP, author YAML blocks via dry_run=True, compile + apply
Edit — add/modify/remove monitors; name-based identity (no UUID); deprecated field migration with confirmation
Validate — montecarlo monitors compile first; manual schema validation as fallback
Import — montecarlo monitors export / convert-to-mac first; get_monitors MCP as fallback

Deprecated type handling: skill actively declines field_health, dimension_tracking, field_quality, comparison, freshness, volume and suggests the correct modern alternative.

Schema validation gates in monitoring-advisor and tune-monitor:

Both skills now validate generated YAML against the published schema (clidocs.getmontecarlo.com/mac/schema.json) before presenting it to the user
tune-monitor also detects MaC-managed monitors and offers to hand off to /manage-mac to keep the YAML file as the source of truth

Follow-up: YET-1310 — publish a clean schema without deprecated fields.

Test plan

CLI available: create a metric monitor end-to-end — compile passes, apply command shown
CLI unavailable: validate falls back to manual tier, schema fetched via curl
Request deprecated type (freshness) — skill declines and suggests metric
Edit existing file — name preserved, no UUID lookup, diff shown
Import with convert-to-mac CLI path
Co-located dbt schema.yml — extra top-level keys not flagged
tune-monitor: generates YAML → schema validation runs before presenting to user
tune-monitor: MaC-managed monitor detected → offers to hand off to /manage-mac

🤖 Generated with Claude Code

- Add schemas/mac-schema.json (generated from monolith-django) - Add /manage-mac skill for schema-aware MaC YAML authoring (create, edit, validate) - Add plugin symlinks in all 5 editor plugins - Inject schema validation gate into monitoring-advisor data-monitor-creation.md - Inject schema validation rules into tune-monitor SKILL.md - Update README.md and skills/README.md to list manage-mac Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

linear · 2026-05-20T08:49:59Z

YET-1269

…ope disclaimer - Regenerate mac-schema.json: deprecated field aliases (resource, comparisons, notify_rule_run_failure, domain, domain_uuids) marked with deprecated: true, root additionalProperties removed for dbt YAML co-location compatibility - Add schema scope disclaimer to manage-mac, monitoring-advisor, and tune-monitor skills: schema validates structural correctness only, not cross-field semantics Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ated field migration guidance - Rebundle mac-schema.json: fields with deprecated aliases no longer in required, canonical fields carry description pointing to old name for bidirectional agent lookup - manage-mac: add deprecated field migration workflow — agent proactively lists all deprecated fields found and offers to migrate after explicit user confirmation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…specific authoring guidance - Add import workflow using get_monitors(config_format="yaml") - Fix validate check #5: allow extra top-level keys for dbt co-location - Add inline validation.alert_condition predicate tree example (schema gives no structure) - Add metric.data_source structure note (no flat table field) - Add auto threshold callout: operator AUTO/AUTO_HIGH/AUTO_LOW, alert_conditions required - Add sensitivity lowercase enum callout - Regenerate mac-schema.json with description hoisting and notifications: key - Add MaC deployment fork to monitoring-advisor data-monitor-creation.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Scope sensitivity field to metric monitors only (not valid on custom_sql) - Add audiences vs notifications: NaC distinction to Phase 0 - Add description to metric required fields list in Phase 0 - Expand validation template to include all required fields (data_source, schedule) - Note alert_condition singular vs alert_conditions plural on validation - Fix validate example: full_table_id → description (field that actually exists); move sensitivity typo example to metric type; add custom_sql sensitivity as wrong-type example - Edit workflow: Add instruction directs to Create workflow field-gathering steps; Remove instruction handles empty list by removing the type key Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- P0: Add explicit notes that query_performance and table monitor type do not have a flat table: field (use selection/asset_selection instead) - P1: Scope AUTO/AUTO_HIGH/AUTO_LOW operator callout to metric monitors only; note custom_sql and field_quality use GT/LT/GTE/LTE/EQ/NEQ - P1: Wire deprecated field detection into Edit workflow Step 2 - P2: Update ambiguous-intent fallback question to surface Import workflow - P2: Make Import Step 3 namespace prompt conditional (mirrors Create Step 3) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…cs, add import triggers - Expand ambiguous-intent fallback to list all 4 workflows explicitly (Create / Edit / Validate / Import) - Document binary predicate right-side value node requirement for starts_with, regex_match, etc. - Add Import example triggers to when_to_use frontmatter Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ontmatter Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…iptions, add tune-monitor awareness - Remove bundled schemas/mac-schema.json; all skills now fetch from https://docs.getmontecarlo.com/mac/schema.json via WebFetch — no manual sync required - Update manage-mac SKILL.md: Phase 0 uses WebFetch, frontmatter description reflects all 4 workflows (create/edit/validate/import), graceful degradation updated - Update tune-monitor Phase 5: schema validation uses published URL; new step warns that API changes to MaC-managed monitors will be overwritten on next apply and offers /manage-mac edit workflow as the correct path - Update data-monitor-creation.md schema reference to published URL - Update all 5 plugin CHANGELOGs: consistent /manage-mac description (create/edit/validate/import), remove stale "bundled locally" line Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…t/validate/import Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

santiagoaguiar · 2026-05-20T13:35:11Z

+
+The top-level structure under `montecarlo:` contains these monitor type keys:
+
+- `field_health`, `dimension_tracking`, `json_schema`, `metric`, `metric_comparison`


field_health/dimension_tracking/field_quality/comparison are deprecated - maybe we need to properly annotate this on the json schema to generate this skill.

fresheness/volume are discouraged and I would remove them from the skill schema if possible as well.

The rewritten skill actively declines all deprecated types and suggests the nearest valid replacement. Annotating them in the schema itself is a good follow-up — filed separately.

santiagoaguiar · 2026-05-20T13:36:27Z

+confuse them.
+
+`validation` monitors have an `alert_condition` field (singular — not `alert_conditions`)
+whose internal structure is a predicate tree not described further by the schema. The minimal


The predicates definitions can be fetched with the get predicates MCP tool.

Done — skill calls get_validation_predicates in Create Step 2 before authoring a validation monitor, and stops if the result is empty.

santiagoaguiar · 2026-05-20T13:37:06Z

+While reading the file, also check for deprecated field names (marked `deprecated: true` in
+the schema). If found, follow the deprecated field handling procedure in the Validate section
+before applying changes.
+


Maybe we can have a separate published file without deprecated fields... to avoid confusing the LLM with constructs we won't longer accept.

santiagoaguiar · 2026-05-20T13:37:55Z

+description: Create, edit, validate, and import Monitors-as-Code YAML files. Fetches the published MaC JSON Schema to ensure correctness across all 14 monitor types.
+when_to_use: |
+  Invoke when the user has a MaC YAML file they want to create, edit, or validate, or when they
+  want to export live monitors into a MaC YAML file.
+  Example triggers: "create a monitors YAML for this table", "add a metric monitor to my MaC file",
+  "validate my monitors.yaml before I apply it", "what's wrong with my MaC file",
+  "export my existing monitors to YAML", "get my monitors into a file so I can commit them",
+  "import my live monitors to YAML", "get a MaC file from my existing monitors".
+  Do NOT invoke when the user wants to discover what to monitor or generate monitors from scratch
+  via table exploration — use monitoring-advisor for that.


For monitor creation, a ton of knowledge is part of the data-monitor-creation skill. Can we leverage that somehow?

Done — Create Step 2 references ../monitoring-advisor/references/data-monitor-creation.md for table/field/UUID resolution steps.

- Remove deprecated/discouraged monitor types from supported type list: field_health, dimension_tracking, field_quality, comparison, freshness, volume - Keep single "do not author" note; removed all scattered mentions - Strengthen get_validation_predicates usage: always call the tool, list is reference only - Remove field_quality and volume from AUTO operator callout — verify via schema instead - Create Step 1: reference data-monitor-creation.md for table/field/domain/warehouse validation instead of duplicating that knowledge Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Create and Edit workflows now call create_or_update_*_monitor(dry_run=True) to get canonical YAML from the backend rather than hand-authoring from the JSON schema. The schema (WebFetch) is now only used in the Validate workflow for local validation without a server round-trip. Key changes: - Remove mandatory Phase 0 schema fetch; schema is now Validate-only - Add MCP tools table to Prerequisites; MCP server required for Create/Edit/Import - Create: tool-first via dry_run=True, with table/column validation from data-monitor-creation.md steps 1-3 - Edit: dry_run=True for both deployed (UUID via get_monitors) and undeployed monitors - Validate: unchanged — WebFetch schema, graceful degradation, type-specific reminders - Import: unchanged — get_monitors with config_format=yaml - Fallback to schema-based authoring for types without a dedicated MCP tool (json_schema, query_performance, bulk_monitor) - Version bump 1.0.0 → 2.0.0 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Create Step 1: prompt for metric type/expression on `metric` monitors - Validate: document `query_performance` alert_conditions structure (require `threshold` + `metric`; additionalProperties: false) - Validate: document `bulk_monitor.monitor_type` enum values (`bulk_metric`, `bulk_pii`); `metric` is not valid Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

santiagoaguiar · 2026-05-21T17:44:52Z

+For monitor types without a dedicated MCP tool (`json_schema`, `query_performance`,
+`bulk_monitor`), fall back to schema-based authoring: fetch the schema via Bash and derive all
+fields from it. Never guess field names.
+
+```bash
+curl -s https://clidocs.getmontecarlo.com/mac/schema.json
+```


Make sense, we should eventually add tools for these as well 👍 .

santiagoaguiar · 2026-05-21T17:45:45Z

+   `comparison`, `freshness`, or `volume`. If the user explicitly requests one of these,
+   decline: inform them it is no longer supported, and suggest the closest valid alternative
+   (e.g. `freshness` or `volume` → `metric` monitor tracking recency or row count;
+   `field_health` → `validation` monitor; `comparison` → `metric_comparison`).


Suggested change

`field_health` → `validation` monitor; `comparison` → `metric_comparison`).

`field_quality` → `validation` monitor; `comparison` → `metric_comparison`; `field_health` → `metric`).

Done, applied!

santiagoaguiar · 2026-05-21T17:46:47Z

+After receiving the dry_run output, check it for deprecated field names (e.g. `resource`,
+`notify_rule_run_failure`, `domain_uuids`, `comparisons`). If any are present, replace them with
+their canonical equivalents before assembling the file.


Should never happen, export should use 'modern' names.

Good point, removed — backend already returns modern names.

santiagoaguiar · 2026-05-21T17:50:01Z

+- If the monitor has been deployed and has a UUID: if the UUID is already known (e.g. present
+  in the file or supplied by the user), use it directly. Otherwise resolve it via
+  `get_monitors(full_table_id=..., config_format="yaml")`. Then call
+  `create_or_update_*_monitor(dry_run=True, monitor_uuid=<uuid>, ...)` with the updated
+  parameters. Use the returned YAML block to replace the existing monitor entry.
+- If the monitor has not yet been deployed (no UUID): call
+  `create_or_update_*_monitor(dry_run=True, ...)` without a UUID, using the current parameters
+  plus the requested changes. Replace the existing block with the returned YAML.


Given here we are strictly in the YAML realm, we shouldn't need to resolve the UUID, we only need to make sure that for edit we preserve the name attribute and use the same namespace, otherwise it will be a different monitor. You can still use create_or_update with dry_run=True to get the shape of the YAML. Pass the current monitor name as the name argument to create_or_update_*_monitor.

Switched to name-based identity throughout — no UUID lookup needed.

santiagoaguiar · 2026-05-21T17:52:30Z

+Some fields are YAML-level only and do not require a `dry_run` MCP call: `audiences`,
+`is_paused`, `labels`, `tags`, `priority`, `data_quality_dimension`, `domains`. For these,
+add or modify the field directly in the YAML without calling the MCP tool. Note: `is_paused`
+is accepted by the backend at apply time but is not present in the JSON Schema — if a schema
+scan flags it as unknown, ignore that warning.


I think these should be present already in create_or_update_*_monitor. Maybe instead of mentioning specific properties here provide a more general statement without actual examples to reduce drift.

Done, replaced with a general statement.

santiagoaguiar · 2026-05-21T17:53:41Z

+### Step 3: Apply and show the diff
+
+Show only what changed — before/after for modified blocks, or the new block for additions. Then
+write the updated file using the Edit tool.
+
+Ensure the `# yaml-language-server: $schema=https://clidocs.getmontecarlo.com/mac/schema.json`
+header is the first line. Add it if the existing file lacks it.
+
+If removing the last monitor of the last type, the file should contain only the
+yaml-language-server header and `montecarlo: {}`.


Should we add the required CLI call here?

Yes, added compile + apply to Edit Step 3.

santiagoaguiar · 2026-05-21T17:54:18Z

+## Validate workflow
+
+### Step 1: Fetch the schema


Should we instead add support to the CLI to run a local validation, then here instruct to use CLI?

Yes! Rewrote Validate to use montecarlo monitors compile as Step 1 — manual fallback only when CLI is unavailable.

santiagoaguiar · 2026-05-21T17:55:28Z

+## Import workflow
+
+### Step 1: Identify the source


The CLI has support to convert from UI to MaC and viceversa, and to export monitors to YAML. This procedure might still be more flexible but worth considering a reference to CLI tools.

Added CLI-first path to Import — export and convert-to-mac as Tier 1, get_monitors MCP as fallback.

santiagoaguiar · 2026-05-21T17:56:30Z

+## Prerequisites
+
+The Monte Carlo MCP server is required for **Create**, **Edit**, and **Import** workflows. No MCP
+tools are needed for **Validate** (file + schema only).
+


Should we suggest installing MC CLI for these flows? Is fundamental for apply and others.

Added a Prerequisites section + CLI check at workflow start with install/configure steps and a 3-tier system throughout the skill.

- Fix deprecated type alternatives mapping (field_health → metric, field_quality → validation per Santi's suggestion) - Remove post-dry_run deprecated field check (backend returns modern names) - Edit workflow: use name-based identity, drop UUID resolution - YAML-level fields: replace specific list with general statement - Add CLI-first tooling tier system (Tier 1: CLI, Tier 2: MCP, Tier 3: Manual) - CLI check at workflow start with install/configure instructions - Add Prerequisites section with docs links for CLI and MCP server - Validate workflow: montecarlo monitors compile as primary path - Edit workflow: compile + apply commands in Step 3 - Import workflow: convert-to-mac / export as CLI-first path - Rename tier 3 from "Schema" to "Manual" (customer-facing language) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Namespace enforcement: prompt before showing apply command (S33) - Type routing: map common phrases to monitor types (S39) - Deprecated migration decline path: proceed with edit without migrating (S53) - Multi-monitor: complete all dry_run calls before assembling file (S35) - CLI setup: resume workflow after install/configure (S4) - Monitor removal: explicitly preserve siblings in type list (S48) - YAML-level fields: add common examples (is_paused, labels, priority, etc.) (S56-58) - Validate Tier 3 label: explicit in step heading (S63) - Validate deprecated listing: all occurrences not just unique names (S79) - Entry point: in-body redirect to monitoring-advisor for discovery requests (S93) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

wojtekdeka and others added 3 commits May 20, 2026 10:27

chore: bump version to 1.12.0

bcd0bf5

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix: manage-mac name prefix and yaml-language-server schema separator

e3f23b1

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

wojtekdeka marked this pull request as draft May 20, 2026 09:00

wojtekdeka changed the title ~~feat: add /manage-mac skill and JSON Schema validation gates (YET-1269)~~ YET-1269: add /manage-mac skill and JSON Schema validation gates May 20, 2026

wojtekdeka changed the title ~~YET-1269: add /manage-mac skill and JSON Schema validation gates~~ YET-1269: add /manage-mac skill and MaC JSON Schema for AI agents May 20, 2026

wojtekdeka marked this pull request as ready for review May 20, 2026 10:02

wojtekdeka and others added 8 commits May 20, 2026 12:17

fix(manage-mac): add import/export example triggers to when_to_use fr…

ab1514f

…ontmatter Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs: update manage-mac descriptions in READMEs to reflect create/edi…

1096e2c

…t/validate/import Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

santiagoaguiar reviewed May 20, 2026

View reviewed changes

wojtekdeka and others added 3 commits May 20, 2026 16:13

santiagoaguiar approved these changes May 21, 2026

View reviewed changes

santiagoaguiar reviewed May 21, 2026

View reviewed changes

wojtekdeka and others added 3 commits May 22, 2026 08:50

Address the skill feedback.

48d8bbc

wojtekdeka changed the title ~~YET-1269: add /manage-mac skill and MaC JSON Schema for AI agents~~ YET-1269: /manage-mac skill — CLI-first MaC YAML authoring May 22, 2026

docs: update CHANGELOG to reflect CLI-first approach for manage-mac

9d8d257

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

wojtekdeka merged commit 1fcbcbe into main May 22, 2026
5 checks passed


		The top-level structure under `montecarlo:` contains these monitor type keys:

		- `field_health`, `dimension_tracking`, `json_schema`, `metric`, `metric_comparison`

	`field_health` → `validation` monitor; `comparison` → `metric_comparison`).
	`field_quality` → `validation` monitor; `comparison` → `metric_comparison`; `field_health` → `metric`).

Conversation

wojtekdeka commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

linear Bot commented May 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wojtekdeka commented May 20, 2026 •

edited

Loading