Skip to content

Commit fca73a1

Browse files
committed
refactor: move budget alert and cost analysis to its own tutorial
Signed-off-by: Nilushan Costa <nilushan@wso2.com>
1 parent 5b334b3 commit fca73a1

4 files changed

Lines changed: 575 additions & 84 deletions

File tree

docs/tutorials/component-alerts-and-incidents.mdx

Lines changed: 6 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -413,9 +413,10 @@ Apply this step to create alert-rule trait instances for:
413413
- `cart` component (metric-based alert)
414414
- Trigger: memory usage of the cart component exceeds 70% for 2 minutes
415415
- Trait instance name: `cartservice-high-memory-alert`
416-
- `redis` component (budget-based alert)
417-
- Trigger: Cost of the redis component exceeds USD 2 in 5 minutes
418-
- Trait instance name: `redis-budget-alert`
416+
417+
:::tip
418+
To configure a **budget-based** alert (and have the **FinOps Agent** generate an AI cost analysis report when it fires), see [Configure Budget Alert and Perform Cost Analysis using FinOps Agent](./configure-budget-alert-and-cost-analysis.mdx).
419+
:::
419420

420421
Attach the alert-rule traits to existing components by appending them to spec.traits if the field already exists,
421422
or creating the array if it does not:
@@ -544,46 +545,6 @@ else
544545
]'
545546
fi
546547

547-
# redis component: budget-based alert
548-
if kubectl get component redis -n default \
549-
-o jsonpath='{.spec.traits[*].instanceName}' 2>/dev/null \
550-
| tr ' ' '\n' | grep -qx "redis-budget-alert"; then
551-
echo "Trait 'redis-budget-alert' already exists on component 'redis', skipping."
552-
else
553-
kubectl patch component redis -n default --type=json -p='[
554-
{
555-
"op": "add",
556-
"path": "/spec/traits/-",
557-
"value": {
558-
"name": "observability-alert-rule",
559-
"kind": "ClusterTrait",
560-
"instanceName": "redis-budget-alert",
561-
"parameters": {
562-
"description": "Alert when redis cost for 5mins exceeds USD 2",
563-
"severity": "warning",
564-
"source": { "type": "budget" },
565-
"condition": { "window": "5m", "interval": "1m", "operator": "gt", "threshold": 2 }
566-
}
567-
}
568-
}
569-
]' 2>/dev/null || kubectl patch component redis -n default --type=json -p='[
570-
{
571-
"op": "add",
572-
"path": "/spec/traits",
573-
"value": [{
574-
"name": "observability-alert-rule",
575-
"kind": "ClusterTrait",
576-
"instanceName": "redis-budget-alert",
577-
"parameters": {
578-
"description": "Alert when redis cost for 5mins exceeds USD 2",
579-
"severity": "warning",
580-
"source": { "type": "budget" },
581-
"condition": { "window": "5m", "interval": "1m", "operator": "gt", "threshold": 2 }
582-
}
583-
}]
584-
}
585-
]'
586-
fi
587548
```
588549

589550
#### Notification Channels Are Configured Per Environment
@@ -605,7 +566,6 @@ This step creates `ReleaseBinding`s that:
605566
- `frontend` component: overrides `PRODUCT_CATALOG_SERVICE_ADDR` to an invalid endpoint
606567
- `recommendation` component: reduces CPU requests/limits to make high CPU easier to hit
607568
- `cart` component: reduces memory requests/limits to make high memory easier to hit
608-
- `redis` component: inflates CPU and memory requests/limits so the projected cost crosses the budget threshold in a short period of time
609569
- Configure alert behavior for the `development` environment via `traitEnvironmentConfigs`:
610570
- Enable/disable alert rules
611571
- Select notification channels
@@ -693,42 +653,11 @@ spec:
693653
memory: "150Mi"
694654
# Note: traitEnvironmentConfigs is omitted here.
695655
# Defaults: alert enabled, incident creation disabled, no AI RCA, uses environment's default notification channel
696-
697-
---
698-
# ReleaseBinding for redis component with oversized resource requests/limits to trigger a budget alert
699-
apiVersion: openchoreo.dev/v1alpha1
700-
kind: ReleaseBinding
701-
metadata:
702-
name: redis-development
703-
namespace: default
704-
spec:
705-
owner:
706-
projectName: gcp-microservice-demo
707-
componentName: redis
708-
environment: development
709-
componentTypeEnvironmentConfigs:
710-
resources:
711-
requests:
712-
cpu: "500m"
713-
memory: "400Mi"
714-
limits:
715-
cpu: "1000m"
716-
memory: "1000Mi"
717-
traitEnvironmentConfigs:
718-
redis-budget-alert:
719-
enabled: true
720-
actions:
721-
notifications:
722-
channels:
723-
- "webhook-notification-channel-development"
724-
incident:
725-
enabled: true
726-
triggerAiCostAnalysis: true
727656
EOF
728657
```
729658

730659
:::note
731-
`actions.incident.triggerAiRca: true` and `actions.incident.triggerAiCostAnalysis: true` both require `actions.incident.enabled: true`. `triggerAiCostAnalysis` is only valid for alerts with `source.type: budget`.
660+
`actions.incident.triggerAiRca: true` requires `actions.incident.enabled: true`.
732661
:::
733662

734663
### Step 6: Trigger Alerts
@@ -772,14 +701,6 @@ Also you can acknowledge and resolve incidents via the Backstage portal when the
772701

773702
If you have properly configured the [SRE Agent](../ai/sre-agent.mdx), you can verify AI root cause analysis by checking the RCA reports in the Backstage portal when an incident is created.
774703

775-
### Step 9: Verify Budget Alert and AI Cost Analysis
776-
777-
Within a few minutes of applying the redis `ReleaseBinding`, the `redis-budget-alert` should fire because the inflated CPU/memory requests push the cost above the threshold.
778-
779-
- Confirm alert delivery to the configured webhook notification channel.
780-
- Confirm that an incident was created for the budget alert
781-
- If the FinOps Agent is configured, an **AI cost analysis** report is generated for the incident — view it in the Backstage portal alongside the incident. The cost analysis report provides a cost optimization recommendation and lets you apply the recommendation automatically.
782-
783704
## Summary
784705

785706
You attached OpenChoreo **observability alert rules** to existing components (as `observability-alert-rule` traits), configured **email** and **webhook** notification channels per environment, and enabled **incident creation** (plus AI root cause analysis) via `ReleaseBinding` `traitEnvironmentConfigs`.
@@ -789,6 +710,7 @@ Then you triggered the alerts using controlled misconfigurations, and verified a
789710
## Next Steps
790711

791712
- Configure and tune the [SRE Agent](../ai/sre-agent.mdx) for your observability/AI requirements.
713+
- Follow [Configure Budget Alert and Perform Cost Analysis using FinOps Agent](./configure-budget-alert-and-cost-analysis.mdx) to set up budget alerts and AI cost analysis.
792714
- Explore how to view and manage alerts/incidents via the Observer API (Backstage portal and OpenChoreo MCP server).
793715
- Customize webhook formatting using `payloadTemplate` for downstream systems (for example, Slack-compatible payloads).
794716
- Refer to [Observability Alerting](../platform-engineer-guide/observability-alerting.mdx) for how alerting architecture works in OpenChoreo.

0 commit comments

Comments
 (0)