fix(observability): stop local dev from polluting prod Axiom datasets#546
Conversation
…d Axiom datasets Every service's evlog drain shipped wide events to its production Axiom dataset even in development: api/basket/uptime called batchedAxiomDrain unconditionally, and insights/slack gated only on the Axiom key being present (which a local .env often has). Result: dev noise polluted prod telemetry — e.g. a Jul 3-4 'spike' of 5,856 api errors was entirely local Bun module-resolution failures, dwarfing real prod errors >4:1 and misreading as a production incident. Gate the Axiom drain on NODE_ENV !== 'development' in all five services. Dev still writes to local FS logs; prod is unchanged.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
The latest updates on your projects. Learn more about Unkey Deploy
|
Greptile SummaryThis PR gates the Axiom log drain behind
Confidence Score: 3/5The dev-drain leak is fixed for The core fix is correct and stops the exact incident from recurring in dev. However, the All five evlog files share the Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Logger Drain Called] --> B{NODE_ENV?}
B -->|"=== 'development'"| C[devFsDrain only]
B -->|"!== 'development'\n(production, test, staging, ...)"| D[devFsDrain if enabled]
D --> E[batchedAxiomDrain]
D --> F[batchedSuperlogDrain]
C --> G[Local .evlog/logs]
E --> H[Axiom Dataset]
F --> I[Superlog]
style C fill:#90EE90
style E fill:#FFB347
style B fill:#87CEEB
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[Logger Drain Called] --> B{NODE_ENV?}
B -->|"=== 'development'"| C[devFsDrain only]
B -->|"!== 'development'\n(production, test, staging, ...)"| D[devFsDrain if enabled]
D --> E[batchedAxiomDrain]
D --> F[batchedSuperlogDrain]
C --> G[Local .evlog/logs]
E --> H[Axiom Dataset]
F --> I[Superlog]
style C fill:#90EE90
style E fill:#FFB347
style B fill:#87CEEB
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const useLocalEvlogFiles = | ||
| process.env.NODE_ENV === "development" || readBooleanEnv("API_EVLOG_FS"); | ||
|
|
||
| const drainToAxiom = process.env.NODE_ENV !== "development"; |
There was a problem hiding this comment.
NODE_ENV !== "development" permits non-production environments
The guard !== "development" allows "test", "staging", or any other unrecognized value to drain to Axiom. If CI runs are executed with NODE_ENV=test (a common Jest/Bun test default), they would write to the production Axiom dataset just as dev traffic did before this fix. A more conservative allowlist — process.env.NODE_ENV === "production" — would ensure only intentionally production-deployed processes drain. The same concern applies to the identical guard in apps/basket/src/lib/evlog-basket.ts, apps/uptime/src/lib/evlog-uptime.ts, apps/insights/src/lib/evlog-insights.ts, and apps/slack/src/lib/evlog-slack.ts.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
3 issues found across 5 files
Confidence score: 3/5
- In
apps/uptime/src/lib/evlog-uptime.tsandapps/api/src/lib/evlog-api.ts, the currentNODE_ENV !== "development"style guard can route non-production runs (unset/typo/test envs) into production Axiom, risking telemetry pollution and noisy operational signals if merged as-is — switch to a fail-closed allowlist (production-only) before merging. - In
apps/insights/src/lib/evlog-insights.ts,env.NODE_ENVandprocess.env.NODE_ENVare checked inconsistently, so local dev can miss the.evlog/logsfallback and silently lose expected local logging behavior — align both checks to the same normalized env source (or pass it through explicitly) before merging.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/insights/src/lib/evlog-insights.ts">
<violation number="1" location="apps/insights/src/lib/evlog-insights.ts:28">
P2: Local insights dev runs can now lose the intended `.evlog/logs` fallback: `env.NODE_ENV` defaults to `development` and disables Axiom, but `fsDrain` still checks raw `process.env.NODE_ENV`, and the dev script does not set it. Consider aligning the FS drain check to `env.NODE_ENV` or setting `NODE_ENV=development` in the insights dev script.</violation>
</file>
<file name="apps/uptime/src/lib/evlog-uptime.ts">
<violation number="1" location="apps/uptime/src/lib/evlog-uptime.ts:33">
P2: Direct local uptime runs can still ship to Axiom when `NODE_ENV` is unset, because this guard enables Axiom for every value except the exact string `development`. Consider fail-closing to production only so a missing local env var does not pollute the production datasets again.</violation>
</file>
<file name="apps/api/src/lib/evlog-api.ts">
<violation number="1" location="apps/api/src/lib/evlog-api.ts:33">
P2: The denylist guard `!== "development"` still permits non-production environments (e.g. `NODE_ENV=test` during CI, or an unset/typo'd value) to drain to the production Axiom dataset — the same class of pollution this PR aims to prevent. An allowlist check like `process.env.NODE_ENV === "production"` would be more defensive, ensuring only intentionally deployed processes write to Axiom. The same pattern is repeated in `apps/basket`, `apps/uptime`, `apps/insights`, and `apps/slack`.</violation>
</file>
Architecture diagram
sequenceDiagram
participant App as Service (api/basket/uptime)
participant Evlog as evlog Logger Drain
participant FS as Local FS (.evlog/logs)
participant Axiom as Axiom Dataset (prod)
participant Superlog as Superlog Drain
Note over App,Superlog: Logger drain flow for each service
App->>Evlog: loggerDrain(ctx)
alt NODE_ENV === "development"
Evlog->>FS: devFsDrain(ctx)
Note over Evlog: drainToAxiom = false
Evlog->>Superlog: batchedSuperlogDrain?.(ctx)
FS-->>App: Local log written
else NODE_ENV !== "development" (prod/staging)
Evlog->>FS: devFsDrain(ctx) (if configured)
Evlog->>Axiom: batchedAxiomDrain(ctx)
Evlog->>Superlog: batchedSuperlogDrain?.(ctx)
Axiom-->>Evlog: Event ingested
end
participant Insights as insights Service
participant Slack as slack Service
Note over Insights,Slack: Similar flow with pipeline wrapper
Insights->>Evlog: loggerDrain(ctx)
alt NODE_ENV === "development" OR no axiomApiKey
Evlog->>Superlog: batchedSuperlogDrain?.(ctx)
Note over Evlog: batchedAxiomDrain is null
else prod with axiomApiKey
Evlog->>Axiom: batchedAxiomDrain(ctx)
Evlog->>Superlog: batchedSuperlogDrain?.(ctx)
Axiom-->>Evlog: Event ingested
end
Slack->>Evlog: loggerDrain(ctx)
alt NODE_ENV === "development" OR no axiomApiKey
Evlog->>Superlog: batchedSuperlogDrain?.(ctx)
Note over Evlog: batchedAxiomDrain is null
else prod with axiomApiKey
Evlog->>Axiom: batchedAxiomDrain(ctx)
Evlog->>Superlog: batchedSuperlogDrain?.(ctx)
Axiom-->>Evlog: Event ingested
end
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
| ) | ||
| : null; | ||
| const batchedAxiomDrain = | ||
| axiomApiKey && env.NODE_ENV !== "development" |
There was a problem hiding this comment.
P2: Local insights dev runs can now lose the intended .evlog/logs fallback: env.NODE_ENV defaults to development and disables Axiom, but fsDrain still checks raw process.env.NODE_ENV, and the dev script does not set it. Consider aligning the FS drain check to env.NODE_ENV or setting NODE_ENV=development in the insights dev script.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/insights/src/lib/evlog-insights.ts, line 28:
<comment>Local insights dev runs can now lose the intended `.evlog/logs` fallback: `env.NODE_ENV` defaults to `development` and disables Axiom, but `fsDrain` still checks raw `process.env.NODE_ENV`, and the dev script does not set it. Consider aligning the FS drain check to `env.NODE_ENV` or setting `NODE_ENV=development` in the insights dev script.</comment>
<file context>
@@ -24,15 +24,16 @@ const pipeline = createDrainPipeline<DrainContext>({
- )
- : null;
+const batchedAxiomDrain =
+ axiomApiKey && env.NODE_ENV !== "development"
+ ? pipeline(
+ createAxiomDrain({
</file context>
| const useLocalEvlogFiles = | ||
| process.env.NODE_ENV === "development" || readBooleanEnv("UPTIME_EVLOG_FS"); | ||
|
|
||
| const drainToAxiom = process.env.NODE_ENV !== "development"; |
There was a problem hiding this comment.
P2: Direct local uptime runs can still ship to Axiom when NODE_ENV is unset, because this guard enables Axiom for every value except the exact string development. Consider fail-closing to production only so a missing local env var does not pollute the production datasets again.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/uptime/src/lib/evlog-uptime.ts, line 33:
<comment>Direct local uptime runs can still ship to Axiom when `NODE_ENV` is unset, because this guard enables Axiom for every value except the exact string `development`. Consider fail-closing to production only so a missing local env var does not pollute the production datasets again.</comment>
<file context>
@@ -30,6 +30,8 @@ const devFsLogsDir = join(
const useLocalEvlogFiles =
process.env.NODE_ENV === "development" || readBooleanEnv("UPTIME_EVLOG_FS");
+const drainToAxiom = process.env.NODE_ENV !== "development";
+
const devFsDrain = useLocalEvlogFiles
</file context>
| const drainToAxiom = process.env.NODE_ENV !== "development"; | |
| const drainToAxiom = process.env.NODE_ENV === "production"; |
| const useLocalEvlogFiles = | ||
| process.env.NODE_ENV === "development" || readBooleanEnv("API_EVLOG_FS"); | ||
|
|
||
| const drainToAxiom = process.env.NODE_ENV !== "development"; |
There was a problem hiding this comment.
P2: The denylist guard !== "development" still permits non-production environments (e.g. NODE_ENV=test during CI, or an unset/typo'd value) to drain to the production Axiom dataset — the same class of pollution this PR aims to prevent. An allowlist check like process.env.NODE_ENV === "production" would be more defensive, ensuring only intentionally deployed processes write to Axiom. The same pattern is repeated in apps/basket, apps/uptime, apps/insights, and apps/slack.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/api/src/lib/evlog-api.ts, line 33:
<comment>The denylist guard `!== "development"` still permits non-production environments (e.g. `NODE_ENV=test` during CI, or an unset/typo'd value) to drain to the production Axiom dataset — the same class of pollution this PR aims to prevent. An allowlist check like `process.env.NODE_ENV === "production"` would be more defensive, ensuring only intentionally deployed processes write to Axiom. The same pattern is repeated in `apps/basket`, `apps/uptime`, `apps/insights`, and `apps/slack`.</comment>
<file context>
@@ -30,6 +30,8 @@ const devFsLogsDir = join(
const useLocalEvlogFiles =
process.env.NODE_ENV === "development" || readBooleanEnv("API_EVLOG_FS");
+const drainToAxiom = process.env.NODE_ENV !== "development";
+
const devFsDrain = useLocalEvlogFiles
</file context>
| const drainToAxiom = process.env.NODE_ENV !== "development"; | |
| const drainToAxiom = process.env.NODE_ENV === "production"; |
1 participant
Problem
Every service's evlog→Axiom drain ran in development too, so local dev shipped wide events to the production Axiom datasets:
api,basket,uptime: calledbatchedAxiomDrain(ctx)unconditionally.insights,slack: gated only on the Axiom key being present — which a local.envusually has.Impact, observed: the Jul 3–4 "5,856 api server errors on deploy day" was 100%
environment: "development"— local BunCannot find module '@databuddy/services/...'failures from unbuilt workspace packages. Production errors stayed flat (785→498→200→single digits). Dev noise outweighed real prod errors >4:1 in the shared dataset, so error-rate views (and any alert built on them) misread it as a production incident.Fix
Gate the Axiom drain on
NODE_ENV !== "development"in all five services. Dev still writes to local FS logs (.evlog/logs); prod behavior is unchanged.Verification
check-typesgreen for all five apps; lint clean. Behavioral: in dev the drain is now a no-op/null; prod path untouched.Follow-up (separate)
Consider centralizing this in a shared
createAxiomDrainwrapper so new services can't reintroduce the leak.Summary by cubic
Stop local development from draining telemetry to the production Axiom datasets. This removes dev noise from prod metrics and alerts; production behavior is unchanged.
NODE_ENV !== "development"acrossapi,basket,insights,slack, anduptime..evlogfiles only; in production, the Axiom drain runs as before.Written for commit 7fab18d. Summary will update on new commits.