feat: add CLI for monitoring vending jobs and events

- Implemented a new CLI tool in `monitor.py` for managing and monitoring vending jobs and events.
- Added commands for listing jobs, peeking dead-letter queue, showing job stats, and watching for new messages.
- Integrated remote mode to interact with the vending API and local mode for direct Azure Storage access.

feat: create jobs handler package

- Introduced a new handler package for jobs with routes for `/jobs/stats`, `/jobs/list`, and `/jobs/dlq`.
- Added controller logic for handling job-related operations including peeking queues and purging the dead-letter queue.

feat: implement job models and responses

- Defined Pydantic models for job requests and responses in `models.py`.
- Created structured responses for job listing, stats, and job lookup.

refactor: update workflow engine usage

- Refactored the workflow engine to remove the deprecated `run_provisioning_workflow` function.
- Updated all references in tests and other modules to use the new `WorkflowEngine` directly.

test: update tests for new workflow engine

- Modified tests to accommodate changes in the workflow engine and ensure proper functionality.
- Ensured that all tests reflect the new structure and logic of the job handling and workflow execution.

Co-authored-by: Copilot <copilot@github.com>

Author: nielsweistra

CONTRIBUTING.md

@@ -119,7 +119,7 @@ handlers/          (FastAPI routers — driving adapters)
        ↓
 retry/dispatcher   (strategy: none / queue / dead_letter)
        ↓
-workflow.py::run_provisioning_workflow()
+WorkflowEngine(settings).run()
   ├── Gate steps   (pre-flight checks, executed first, abort on failure)
   └── Workflow steps (provisioning, topologically ordered by depends_on)
             ↓
@@ -144,7 +144,7 @@ extensions/         (auto-discovered plugins: webhook, API notify, ServiceNow)
 | `handlers/` | FastAPI routers (webhook, worker, preflight, replay, mock). |
 | `retry/` | Retry strategy: inline, Azure Storage Queue, or dead-letter. |
 | `config.py` | All settings via `Pydantic BaseSettings` + `get_settings()` singleton. |
-| `workflow.py` | Orchestrator: built-in steps 1–6 + `run_provisioning_workflow()`. Re-exports domain / registry symbols for backward compatibility. |
+| `workflow/engine.py` | Orchestrator: `WorkflowEngine` class — `run()` executes built-in steps 1–6. Re-exports domain / registry symbols for backward compatibility. |
 
 ---
 

docs/api.md

@@ -298,6 +298,183 @@ Triggers the provisioning workflow directly without a real Event Grid event. Onl
 
 ---
 
+## Configuration
+
+### `GET /config`
+
+Returns the active service configuration with all secret fields redacted. Useful for verifying which settings are active without exposing credentials.
+
+The fields `azure_client_secret`, `worker_secret`, and `event_grid_sas_key` are replaced with `"***"` when non-empty.
+
+**Response** `200 OK`
+
+```json
+{
+  "azure_tenant_id": "00000000-0000-0000-0000-000000000001",
+  "azure_client_id": "my-client-id",
+  "azure_client_secret": "***",
+  "retry_strategy": "queue",
+  "provisioning_queue_name": "vending-jobs",
+  "provisioning_dlq_name": "vending-jobs-dlq",
+  "worker_secret": "***",
+  "event_grid_sas_key": "***",
+  "mock_mode": false
+}
+```
+
+---
+
+## Jobs API
+
+The `/jobs/*` endpoints provide visibility into the Azure Storage Queue used by the `queue` retry strategy. They are always registered but are most useful when `VENDING_RETRY_STRATEGY=queue`.
+
+All `/jobs/*` endpoints connect to Azure Storage using the same credentials as the main service (`VENDING_STORAGE_CONNECTION_STRING` or `DefaultAzureCredential` + `VENDING_STORAGE_ACCOUNT_NAME`).
+
+---
+
+### `GET /jobs/stats`
+
+Returns approximate message counts for both the provisioning queue and the dead-letter queue.
+
+**Response** `200 OK`
+
+```json
+{
+  "provisioning": {
+    "queue": "vending-jobs",
+    "approximate_message_count": 3
+  },
+  "dead_letter": {
+    "queue": "vending-jobs-dlq",
+    "approximate_message_count": 1
+  }
+}
+```
+
+If a queue cannot be reached, the response includes `"error": "<message>"` in place of `approximate_message_count`.
+
+---
+
+### `GET /jobs/list`
+
+Peeks the top N messages in the provisioning queue without removing them.
+
+#### Query parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `count` | `5` | Number of messages to peek (1–32) |
+
+**Response** `200 OK`
+
+```json
+{
+  "queue": "vending-jobs",
+  "count": 1,
+  "messages": [
+    {
+      "job_id": "abc123",
+      "subscription_id": "00000000-0000-0000-0000-000000000001",
+      "subscription_name": "my-subscription",
+      "management_group_id": "ITL-Development",
+      "attempt": 1
+    }
+  ]
+}
+```
+
+---
+
+### `GET /jobs/dlq`
+
+Peeks the top N messages in the dead-letter queue. Response shape is identical to `/jobs/list`.
+
+#### Query parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `count` | `5` | Number of messages to peek |
+
+---
+
+### `DELETE /jobs/dlq`
+
+Clears **all** messages from the dead-letter queue. This is a destructive, irreversible operation.
+
+**Response** `200 OK`
+
+```json
+{"queue": "vending-jobs-dlq", "deleted": 3}
+```
+
+---
+
+### `POST /jobs/enqueue`
+
+Enqueues a provisioning job directly to the provisioning queue, bypassing the Event Grid webhook. Useful for manual re-queuing or testing without a real Event Grid event.
+
+**Response** `202 Accepted`
+
+#### Request body
+
+```json
+{
+  "subscription_id": "00000000-0000-0000-0000-000000000001",
+  "subscription_name": "my-subscription",
+  "management_group_id": "ITL-Development",
+  "job_id": "",
+  "attempt": 1
+}
+```
+
+| Field | Type | Required | Default | Description |
+|-------|------|----------|---------|-------------|
+| `subscription_id` | `string` | Yes | — | Azure subscription ID |
+| `subscription_name` | `string` | Yes | — | Display name |
+| `management_group_id` | `string` | No | `""` | Target management group |
+| `job_id` | `string` | No | auto | Custom job ID; a UUID is generated when empty |
+| `attempt` | `integer` | No | `1` | Attempt counter (informational) |
+
+**Response** `202 Accepted`
+
+```json
+{
+  "job_id": "abc123",
+  "message_id": "d3b07384-d113-4ec6-b7b7-d85b72a2f51b",
+  "queue": "vending-jobs"
+}
+```
+
+---
+
+### `GET /jobs/{job_id}`
+
+Looks up a specific job by ID, peeking across both the provisioning queue and the dead-letter queue (up to 32 messages each).
+
+**Response** `200 OK` — found
+
+```json
+{
+  "found": true,
+  "queue": "vending-jobs",
+  "job": {
+    "job_id": "abc123",
+    "subscription_id": "00000000-0000-0000-0000-000000000001",
+    "subscription_name": "my-subscription",
+    "management_group_id": "ITL-Development",
+    "attempt": 1
+  }
+}
+```
+
+**Response** `200 OK` — not found
+
+```json
+{"found": false, "queue": null, "job": null}
+```
+
+---
+
 ## OpenAPI / Swagger UI
 
 When the service is running locally, open the following URLs in your browser:

docs/architecture.md

@@ -74,7 +74,7 @@ extensions/                auto-discovered plugins
 | `extensions/` | Auto-discovered plugins. Each module self-registers at import. Only modules starting with `__` are excluded from discovery. |
 | `handlers/` | FastAPI routers, each as a sub-package (`event_grid/`, `worker/`, `preflight/`, `replay/`, `mock/`). |
 | `core/config.py` | All settings via `Pydantic BaseSettings`. Use `get_settings()` — never instantiate `Settings()` directly. |
-| `workflow/` | Package: `engine.py` hosts `WorkflowEngine` and the backward-compat `run_provisioning_workflow` wrapper; `steps.py` defines built-in steps 1–6. |
+| `workflow/` | Package: `engine.py` hosts `WorkflowEngine`; `steps.py` defines built-in steps 1–6. |
 
 ---
 
@@ -95,7 +95,7 @@ extensions/                auto-discovered plugins
 | `core/protocols.py` | Port contracts: `ManagementGroupPort`, `RbacPort`, `PolicyPort`, `NotificationPort`, `TagReaderPort` — all `@runtime_checkable Protocol` |
 | `core/exceptions.py` | Typed exception hierarchy: `AppError → ProvisioningError` (`GateCheckFailed`, `StepFailed`) `\| AzureIntegrationError` (`ManagementGroupError`, `RbacError`, `PolicyError`, `NotificationError`) `\| ConfigurationError \| AuthorizationError` |
 | `schemas/event_grid.py` | HTTP surface contracts: `EventGridEvent`, `EventGridEventData`, `WebhookResponse`, `HealthResponse` |
-| `workflow/engine.py` | `WorkflowEngine` class — `run()` method orchestrates gates + built-in steps + custom steps + lifecycle events. Also exports backward-compat `run_provisioning_workflow()` wrapper. |
+| `workflow/engine.py` | `WorkflowEngine` class — `run()` method orchestrates gates + built-in steps + custom steps + lifecycle events. |
 | `workflow/steps.py` | Built-in provisioning steps 1–6, each decorated with `@register_step` |
 | `handlers/event_grid/` | `POST /webhook/` — receives Event Grid deliveries, validates SAS key, dispatches to `WorkflowEngine` |
 | `handlers/worker/` | `POST /worker/process-job` — dequeues and processes a `ProvisioningJob` from Azure Storage Queue |

docs/demo.html

@@ -3,7 +3,7 @@
 <head>
   <meta charset="UTF-8" />
   <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-  <title>Automated Demo � ITL Subscription Vending</title>
+  <title>Automated Demo &mdash; ITL Subscription Vending</title>
   <style>
     :root {
       --bg:          #0d1117;
@@ -342,8 +342,8 @@
     <svg width="9" height="9" viewBox="0 0 24 24" fill="currentColor"><circle cx="12" cy="12" r="10"/></svg>
     Automated Walkthrough
   </div>
-  <h1>Subscription Vending � Live Demo</h1>
-  <p>Watch the full end-to-end provisioning workflow run automatically across five real-world scenarios. The inbound Event Grid webhook, all seven provisioning steps, and the HTTP response are shown in real time.</p>
+  <h1>Subscription Vending &mdash; Live Demo</h1>
+  <p>Watch the full end-to-end provisioning workflow run automatically across six real-world scenarios. The inbound Event Grid webhook, all seven provisioning steps, and the HTTP response are shown in real time.</p>
   <div class="hero-controls">
     <div class="status-badge">
       <span class="idle-dot" id="status-dot"></span>
@@ -371,7 +371,7 @@ <h1>Subscription Vending � Live Demo</h1>
           <div class="panel-header">
             <svg width="12" height="12" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><polyline points="22 12 18 12 15 21 9 3 6 12 2 12"/></svg>
             Scenarios
-            <span class="panel-header-right" id="scenario-counter">0 / 5</span>
+            <span class="panel-header-right" id="scenario-counter">0 / 6</span>
           </div>
           <div class="scenario-progress" id="scenario-progress"></div>
         </div>
@@ -539,28 +539,35 @@ <h1>Subscription Vending � Live Demo</h1>
   },
   {
     label:    "No budget tag",
-    desc:     "Step 5 skipped � budget alert omitted",
+    desc:     "Step 5 skipped \u2014 budget alert omitted",
     sub:      { name: "dev-sandbox-01",       env: "sandbox",     budget: "",     owner: "devteam@itlusions.com",  aks: "false" },
     scenario: "no-budget",
   },
   {
     label:    "Authorization service down",
-    desc:     "Step 2 fails � 503 from auth endpoint",
+    desc:     "Step 2 fails — 503 from auth endpoint",
     sub:      { name: "logistics-staging",    env: "staging",     budget: "500",  owner: "ops@itlusions.com",      aks: "false" },
     scenario: "auth-fail",
   },
   {
     label:    "RBAC partial failure",
-    desc:     "Step 3 partial � 3 of 4 assignments applied",
+    desc:     "Step 3 partial — 3 of 4 assignments applied",
     sub:      { name: "platform-development", env: "development", budget: "1000", owner: "platform@itlusions.com", aks: "true"  },
     scenario: "rbac-partial",
   },
   {
     label:    "Cascading failures",
-    desc:     "Steps 1�5 fail � worst-case scenario",
+    desc:     "Steps 1\u20135 fail \u2014 worst-case scenario",
     sub:      { name: "legacy-production",    env: "production",  budget: "5000", owner: "legacy@itlusions.com",   aks: "false" },
     scenario: "all-fail",
   },
+  {
+    label:    "Queue strategy — async worker",
+    desc:     "Webhook routes to Storage Queue; worker processes the job",
+    sub:      { name: "analytics-development", env: "development", budget: "750",  owner: "data@itlusions.com",     aks: "false" },
+    scenario: "queue",
+    queueMode: true,
+  },
 ];
 
 function getOutcomes(scenario, hasBudget) {
@@ -572,15 +579,16 @@ <h1>Subscription Vending � Live Demo</h1>
   return ({
     happy:         [ok(), ok(), ok(), ok(), ok(), bud, ok()],
     "no-budget":   [ok(), ok(), ok(), ok(), ok(), skip("itl-budget tag not set"), ok()],
-    "auth-fail":   [ok(), ok(), fail("503 Service Unavailable � authorization service unreachable"), ok(), ok(), bud, ok()],
-    "rbac-partial":[ok(), ok(), ok(), { result: "partial", delay: 460, note: "3/4 succeeded � ops-group assignment failed (403 Forbidden)" }, ok(), bud, ok()],
+    "auth-fail":   [ok(), ok(), fail("503 Service Unavailable — authorization service unreachable"), ok(), ok(), bud, ok()],
+    "rbac-partial":[ok(), ok(), ok(), { result: "partial", delay: 460, note: "3/4 succeeded — ops-group assignment failed (403 Forbidden)" }, ok(), bud, ok()],
     "all-fail":    [ok(),
-      fail("ResourceNotFound � management group 'ITL-Production' does not exist"),
-      fail("503 � authorization service unreachable"),
-      fail("AuthorizationFailed � insufficient permissions"),
-      fail("PolicyAssignmentFailed � scope locked"),
-      hasBudget ? fail("ConsumptionQuotaExceeded � budget limit reached") : skip("itl-budget tag not set"),
+      fail("ResourceNotFound — management group 'ITL-Production' does not exist"),
+      fail("503 — authorization service unreachable"),
+      fail("AuthorizationFailed — insufficient permissions"),
+      fail("PolicyAssignmentFailed — scope locked"),
+      hasBudget ? fail("ConsumptionQuotaExceeded — budget limit reached") : skip("itl-budget tag not set"),
       ok()],
+    queue:         [ok(), ok(), ok(), ok(), ok(), bud, ok()],
   })[scenario] || [ok(),ok(),ok(),ok(),ok(),ok(),ok()];
 }
 
@@ -625,7 +633,7 @@ <h1>Subscription Vending � Live Demo</h1>
     <div class="sp-desc">${esc(s.desc)}</div>`;
   }).join("");
   document.getElementById("scenario-counter").textContent =
-    `${Object.keys(doneMap).length} / 5`;
+    `${Object.keys(doneMap).length} / 6`;
 }
 
 function showWebhook(sub, subId, eventId) {
@@ -701,8 +709,8 @@ <h1>Subscription Vending � Live Demo</h1>
     ? `<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="#3fb950" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><polyline points="20 6 9 17 4 12"/></svg>`
     : `<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="#d29922" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M10.29 3.86L1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z"/><line x1="12" y1="9" x2="12" y2="13"/><line x1="12" y1="17" x2="12.01" y2="17"/></svg>`;
   title.textContent = success
-    ? "Provisioning completed � all steps succeeded"
-    : `Provisioning completed � ${errors.length} error${errors.length > 1 ? "s" : ""} recorded`;
+    ? "Provisioning completed — all steps succeeded"
+    : `Provisioning completed — ${errors.length} error${errors.length > 1 ? "s" : ""} recorded`;
 
   body.innerHTML = [
     ["subscription_id",   subId,                                 ""],
@@ -793,6 +801,23 @@ <h1>Subscription Vending � Live Demo</h1>
   tlog("l-muted", `  sub_name             ${sub.name}`);
   tlog("l-muted", `  scenario             ${def.label}`);
 
+  // Queue mode: show dispatch + worker pickup
+  if (def.queueMode) {
+    tlog("l-info",  `${ts()} retry_strategy=queue — dispatching to Azure Storage Queue`);
+    tlog("l-muted", `  queue                vending-jobs`);
+    tlog("l-muted", `  job_id               job-${uuid().slice(0,8)}`);
+    tlog("l-muted", `  attempt              1`);
+    tlog("l-muted", `  message_id           ${uuid()}`);
+    tlog("l-ok",    `  enqueued — returning 200 OK to Event Grid`);
+    await sleep(620);
+    if (adAbort) { adRunning = false; return; }
+    tlog("l-info",  `${ts()} Worker: dequeued message`);
+    tlog("l-muted", `  delivery_count       1`);
+    tlog("l-info",  `${ts()} Worker: starting provisioning`);
+    await sleep(280);
+    if (adAbort) { adRunning = false; return; }
+  }
+
   // -- Steps -------------------------------------------------
   const outcomes  = getOutcomes(def.scenario, hasBudget);
   const errors    = [];