feat(examples): anchor the incident-investigation dataset to a fixed date (default 2026-06-18) (#80)

Follow-up to #65 (merged). The incident demo stack anchored its telemetry to wall-clock now (never-expiring but non-deterministic). For a reproducible RCA dataset the timestamps should be fixed, so the incident now anchors to a fixed instant by default: DEMO_ANCHOR (default 2026-06-18T02:17Z) anchors entities, logs, and the Prometheus backfill to the same instant — config ~anchor-24h, deploy ~anchor-12h, promotion ~anchor-4h, incident ~anchor — so every run produces the same ground-truth times.

DEMO_ANCHOR=now restores the wall-clock / never-expiring behaviour. The live exporter still serves the current breach so an instant get_metrics works on any day; verify.sh resolves the same anchor for its range queries. Scoped to examples/incident-investigation/deploy/ (generators + entrypoint + compose + scripts); no server/query changes.

Author: qiansheng91

examples/incident-investigation/deploy/README.md

@@ -83,12 +83,17 @@ sh examples/incident-investigation/deploy/stop.sh --all    # also remove the bui
 ## Notes
 
 - Telemetry is synthetic, shaped to match the modeled incident — a demo, not production data.
-- Everything is relative to "now", so the demo never expires. Metric history is generated relative
-  to now and loaded with `promtool tsdb create-blocks-from openmetrics` before Prometheus starts; the
-  exporter then continues the same series live. Logs are generated relative to now. The entity
-  timeline (deployment / config-change / incident / promotion timestamps) is re-anchored to now at
-  startup by the demo image's entrypoint — config ~now-24h, deploy ~now-12h, promo active ~now-4h,
-  incident ~now — matching the telemetry.
+- The incident is anchored to a **fixed date (2026-06-18) by default**, so the dataset is
+  reproducible — entities, logs, and the metric history all carry the same timestamps every run
+  (`config ~anchor-24h`, `deploy ~anchor-12h`, `promotion active ~anchor-4h`, `incident ~anchor`).
+  The metric history is loaded with `promtool tsdb create-blocks-from openmetrics` before Prometheus
+  starts; the entity timeline is re-anchored by the demo image's entrypoint; the live exporter still
+  serves the current breach so an instant `get_metrics` works on any day. Set `DEMO_ANCHOR=now` to
+  anchor everything to wall-clock instead (a never-expiring live demo):
+
+  ```bash
+  DEMO_ANCHOR=now sh examples/incident-investigation/deploy/start.sh
+  ```
 - The logs include correlated request traces: a failed checkout shares one `trace_id` across
   `checkout-service → payment-gateway → payment-router → channel → provider`, so you can follow a
   single request down the stack and see the timeout originate downstream and surface up as retry

examples/incident-investigation/deploy/README.zh-CN.md

@@ -67,6 +67,10 @@ sh examples/incident-investigation/deploy/stop.sh --all    # 连构建的镜像
 ## 说明
 
 - 遥测均为合成数据,按建模的故障塑形——是 demo,不是生产数据。
-- 一切都相对"现在",所以 demo 永不过期。指标历史相对 now 生成,在 Prometheus 启动前用 `promtool tsdb create-blocks-from openmetrics` 载入,exporter 随后实时续上;日志相对 now 生成;实体时间线(部署 / 配置变更 / 故障 / 促销时间戳)由 demo 镜像入口脚本在启动时统一位移到 now —— 配置 ~now-24h、部署 ~now-12h、促销激活 ~now-4h、故障 ~now —— 与遥测一致。
+- 故障**默认锚定在固定日期 2026-06-18**,所以数据集可复现 —— 实体、日志、指标历史每次运行都是同一组时间戳(`配置 ~锚点-24h`、`部署 ~锚点-12h`、`促销激活 ~锚点-4h`、`故障 ~锚点`)。指标历史在 Prometheus 启动前用 `promtool tsdb create-blocks-from openmetrics` 载入;实体时间线由 demo 镜像入口脚本位移;实时 exporter 仍提供当前击穿值,所以任何一天 instant `get_metrics` 都能查到。要让一切相对"现在"(永不过期的 live demo),设 `DEMO_ANCHOR=now`:
+
+  ```bash
+  DEMO_ANCHOR=now sh examples/incident-investigation/deploy/start.sh
+  ```
 - 日志包含**跨服务关联 trace**:一次失败的 checkout 用同一个 `trace_id` 串起 `checkout-service → payment-gateway → payment-router → channel → provider`,可顺着一条请求下钻、看到超时源自下游并上溯为重试耗尽;另含配置变更、部署、熔断等地标事件。
 - pack 还建模了 MySQL 部署事件集和一个 runbook;这里灌的可执行 plan 方法是 `get_metrics`(Prometheus)和 `get_logs`(Elasticsearch)。

examples/incident-investigation/deploy/docker-compose.yml

@@ -21,6 +21,9 @@ services:
       dockerfile: examples/incident-investigation/deploy/umodel.Dockerfile
     ports:
       - "${UMODEL_PORT:-8080}:8080"   # UModel API (umctl --addr / MCP target); override host port with UMODEL_PORT
+    environment:
+      # Fixed incident date by default (reproducible); set DEMO_ANCHOR=now for a live demo.
+      - DEMO_ANCHOR=${DEMO_ANCHOR:-}
 
   exporter:
     image: python:3.12-slim
@@ -40,6 +43,7 @@ services:
       - backfill:/backfill
     environment:
       - BACKFILL_OUT=/backfill/openmetrics.txt
+      - DEMO_ANCHOR=${DEMO_ANCHOR:-}
     command: ["python", "gen_backfill.py"]
     restart: "no"
 
@@ -88,6 +92,8 @@ services:
   es-seed:
     image: python:3.12-slim
     working_dir: /app
+    environment:
+      - DEMO_ANCHOR=${DEMO_ANCHOR:-}
     volumes:
       - .:/app:ro
     command: ["python", "gen_logs.py"]

examples/incident-investigation/deploy/entrypoint.py

@@ -1,17 +1,20 @@
 #!/usr/bin/env python3
-"""Demo entrypoint: slide the incident-investigation sample timeline so it is always relative
-to the current time, then start the UModel server.
+"""Demo entrypoint: slide the incident-investigation sample timeline to the demo anchor, then
+start the UModel server.
 
 The sample data carries real, valid ISO timestamps anchored to REF below (so the pack still
 loads correctly outside this demo — e.g. test-integration.sh). Here we shift every timestamp by
-(now - REF) before the server reads them, so the deployment / config-change / incident / promotion
-times track wall-clock now (T-N) and the demo never "expires":
+(anchor - REF) before the server reads them, so the deployment / config-change / incident /
+promotion times land on the anchor:
 
-    config change ~ now-24h, deploy ~ now-12h, promotion active ~ now-4h, incident ~ now.
+    config change ~ anchor-24h, deploy ~ anchor-12h, promotion active ~ anchor-4h, incident ~ anchor.
 
-This matches the now-relative Prometheus/Elasticsearch telemetry seeded by the rest of the stack.
-The shift is idempotent: it always works from a pristine copy (.orig), so restarts re-anchor to
-the current time instead of compounding.
+The anchor defaults to a FIXED instant (DEFAULT_ANCHOR) so the dataset is reproducible — the same
+ground-truth timestamps every run. Set DEMO_ANCHOR=now to anchor to wall-clock instead (a
+never-expiring live demo). This matches the DEMO_ANCHOR handling in gen_logs.py / gen_backfill.py.
+
+The shift is idempotent: it always works from a pristine copy (.orig), so restarts re-anchor
+cleanly instead of compounding.
 """
 import datetime as dt
 import os
@@ -23,13 +26,21 @@
 # Anchor baked into sample-data/entities.json (the INC-0042 P99 breach). If you re-date the
 # sample, update this to the latest "present" timestamp in the file.
 REF = dt.datetime(2026, 6, 17, 18, 10, 0, tzinfo=UTC)
+DEFAULT_ANCHOR = "2026-06-18T02:17:00Z"
 TARGETS = ["/app/examples/incident-investigation/sample-data/entities.json"]
 ISO = re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z")
 
 
+def resolve_anchor():
+    a = os.environ.get("DEMO_ANCHOR") or DEFAULT_ANCHOR
+    if a == "now":
+        return dt.datetime.now(tz=UTC)
+    return dt.datetime.strptime(a, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=UTC)
+
+
 def main():
-    now = dt.datetime.now(tz=UTC)
-    delta = now - REF
+    anchor = resolve_anchor()
+    delta = anchor - REF
     for path in TARGETS:
         if not os.path.exists(path):
             continue
@@ -44,7 +55,7 @@ def shift(m):
 
         open(path, "w").write(ISO.sub(shift, src))
         print(f"[entrypoint] re-anchored {os.path.basename(path)} by {delta} "
-              f"(REF {REF:%Y-%m-%d} -> now {now:%Y-%m-%d %H:%M}Z)", flush=True)
+              f"(REF {REF:%Y-%m-%d} -> anchor {anchor:%Y-%m-%d %H:%M}Z)", flush=True)
 
     os.execvp("umodel-server", ["umodel-server"] + sys.argv[1:])
 

examples/incident-investigation/deploy/gen_backfill.py

@@ -5,7 +5,8 @@
 continues the series from "now", so instant queries stay fresh while range queries see
 the full incident arc.
 
-The history follows the modeled timeline (README "Timeline"), anchored to wall-clock now:
+The history follows the modeled timeline (README "Timeline"), anchored to the demo anchor
+(a fixed date by default; DEMO_ANCHOR=now for wall-clock):
 
     P0  healthy           [now-72h, now-24h)   baseline; everything nominal
     P1  retries-up        [now-24h, now-4h)    T-24h config change -> checkout
@@ -20,6 +21,7 @@
 Reuses exporter.py's SERVICES / PROFILES / BUCKETS as the P2 peak, so the history and the
 live tail share one source of truth. Standard library only.
 """
+import datetime as _dt
 import math
 import os
 import time
@@ -28,9 +30,20 @@
 
 WINDOW_S = 72 * 3600     # history depth
 STEP_S = 300             # one sample per 5 min (range queries use [15m]/[30m] windows)
-END_OFFSET_S = 120       # stop just before now so the live exporter owns the present
+END_OFFSET_S = 120       # stop just before the anchor so the live exporter owns the present
 OUT = os.environ.get("BACKFILL_OUT", "/backfill/openmetrics.txt")
 
+# The incident is anchored to a fixed instant by default (reproducible dataset). Set
+# DEMO_ANCHOR=now to anchor to wall-clock instead (a never-expiring live demo).
+DEFAULT_ANCHOR = "2026-06-18T02:17:00Z"
+
+
+def resolve_now():
+    a = os.environ.get("DEMO_ANCHOR") or DEFAULT_ANCHOR
+    if a == "now":
+        return int(time.time())
+    return int(_dt.datetime.strptime(a, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=_dt.timezone.utc).timestamp())
+
 # Timeline boundaries, as age (seconds before now).
 T_CONFIG = 24 * 3600     # checkout retry config change
 T_PROMO = 4 * 3600       # promotion goes active
@@ -117,7 +130,7 @@ def instant(sid, ts, now):
 
 
 def main():
-    now = int(time.time())
+    now = resolve_now()
     end = now - END_OFFSET_S
     start = end - WINDOW_S
     times = list(range(start, end + 1, STEP_S))

examples/incident-investigation/deploy/gen_logs.py

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 """Generate ~72h of service logs for the incident-investigation demo and bulk-load them
 into the demo Elasticsearch, so the pack's get_logs plans return rows that span the whole
-incident (not a single burst). Timestamps are relative to wall-clock now, so the demo
-always shows "the last three days". Standard library only.
+incident (not a single burst). Timestamps are relative to the demo anchor (a fixed date by
+default; set DEMO_ANCHOR=now for wall-clock). Standard library only.
 
 The log volume and severity follow the modeled timeline (README "Timeline"):
 
@@ -18,6 +18,7 @@
 severity / log_message / trace_id / span_id / http_status / upstream_service /
 latency_ms / error_code / pod).
 """
+import datetime as _dt
 import json
 import os
 import random
@@ -29,6 +30,17 @@
 HOUR = 3600
 RNG = random.Random(0xC0FFEE)          # deterministic output run to run
 
+# The incident is anchored to a fixed instant by default (reproducible dataset). Set
+# DEMO_ANCHOR=now to anchor to wall-clock instead (a never-expiring live demo).
+DEFAULT_ANCHOR = "2026-06-18T02:17:00Z"
+
+
+def resolve_now():
+    a = os.environ.get("DEMO_ANCHOR") or DEFAULT_ANCHOR
+    if a == "now":
+        return int(time.time())
+    return int(_dt.datetime.strptime(a, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=_dt.timezone.utc).timestamp())
+
 # service_id -> short name, mirroring exporter.py / the entity ids the plans substitute.
 SVC = {
     "63718b78868895d2590551b27ec6f51c": "payment-gateway",
@@ -98,7 +110,7 @@ def chain(ts):
 
 
 def main():
-    now = int(time.time())
+    now = resolve_now()
     docs = []
 
     # --- P0 healthy: sparse INFO across the platform, one every ~25 min ---

examples/incident-investigation/deploy/start.sh

@@ -61,7 +61,12 @@ echo "==> Bringing up the incident-investigation demo stack with: $COMPOSE"
 $COMPOSE -f "$COMPOSE_FILE" up -d --build
 
 echo "==> Waiting for the 72h metric backfill + Elasticsearch seed + Prometheus scrapes (up to ~5 min)..."
-hist_ts=$(( $(date +%s) - 216000 ))   # 60h ago: confirms the history was backfilled
+# Probe the backfill 60h before the incident anchor (fixed date by default, or wall-clock when
+# DEMO_ANCHOR=now), so the check lands inside the seeded window regardless of the current date.
+anchor_ts=$(python3 -c 'import os,time,datetime as d
+a=os.environ.get("DEMO_ANCHOR") or "2026-06-18T02:17:00Z"
+print(int(time.time()) if a=="now" else int(d.datetime.strptime(a,"%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=d.timezone.utc).timestamp()))' 2>/dev/null || date +%s)
+hist_ts=$(( anchor_ts - 216000 ))   # 60h before the anchor: confirms the history was backfilled
 # Sentinel query: confirms localhost:$UMODEL_PORT is THIS demo (returns the degraded payment
 # path), not some other umodel. A quoted heredoc keeps the SPL single quotes literal, no escaping.
 um_body() { cat <<'JSON'

examples/incident-investigation/deploy/verify.sh

@@ -14,7 +14,11 @@ PROM="${PROM_URL:-http://localhost:${PROM_PORT:-9090}}"
 ES="${ES_URL:-http://localhost:${ES_PORT:-9200}}"
 PG="63718b78868895d2590551b27ec6f51c"   # payment-gateway
 CK="149632df43354373835df2717cb8fb19"   # checkout-service
-NOW=$(date +%s)
+# Anchor for the section-5 arc queries: the fixed incident instant by default (must match the
+# seeded data), or wall-clock when DEMO_ANCHOR=now. Python keeps the ISO->epoch parse portable.
+NOW=$(python3 -c 'import os,time,datetime as d
+a=os.environ.get("DEMO_ANCHOR") or "2026-06-18T02:17:00Z"
+print(int(time.time()) if a=="now" else int(d.datetime.strptime(a,"%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=d.timezone.utc).timestamp()))' 2>/dev/null || date +%s)
 
 uctl() {
   if command -v umctl >/dev/null 2>&1; then umctl "$@"; else