P1-8: Helicone / Langsmith / Datadog logs / Slack alerts observability sinks

[moonming]

Problem

`crates/aisix-obs/` ships Langfuse + 4 Prometheus series + a marker OTLP. LiteLLM has 30+ named integrations. Four that customers ask for most:

1. Helicone — log every request via `POST https://api.helicone.ai/v1/log/request\` with a Bearer key.
2. Langsmith — `POST https://api.smith.langchain.com/runs/batch\`.
3. Datadog logs — HTTP intake `POST https://http-intake.logs.datadoghq.com/api/v2/logs\` with DD-API-KEY header.
4. Slack alerts — webhook on threshold events (budget at 80%, error spike, etc.).

Scope

Each sink is a separate exporter behind the same async-batched-mpsc shape Langfuse uses (`crates/aisix-obs/src/langfuse.rs` is the model).

DP (moonming/ai-gateway)

• 4 new sub-modules under `aisix-obs`: helicone.rs, langsmith.rs, datadog.rs, slack_alert.rs.
• Each: capacity-bounded mpsc → background batcher → HTTP POST → structured retry with backoff.
• Config (etcd-loaded): `observability: { sinks: [{kind, ...}] }`. Multiple sinks active simultaneously.
• Slack alerts wire into the budget tracker (P0-6) and rate-limit-rejection counter for threshold logic.

CP (api7/api7ee-3-control-plane)

• N/A initially (config in DP file). Optional later: `observability_sink` resource.

Dashboard UI (api7/AISIX-Cloud)

• Settings page (env-scoped or org-scoped — likely org-scoped for billing-related sinks like Datadog/Slack).
• Add/remove sinks form.

e2e

• Mock each external endpoint, send a request, assert payload shape on the wire (especially the LLM-specific ones — Helicone payload schema is finicky).

Estimate

DP 1d/each = 4d, CP 0d, UI 1.5d, e2e 1d