fix(observability): stamp tenant org on gateway traffic docs (G-GATEWAY-INDEX-ORG)

Insights → Analytics/Accounting read ZEROS while offgrid-gateway holds 2,899 real
docs: those surfaces filter by term:{org} for tenant isolation, but the aggregator
shipped docs with no org field, so the filter matched 0 (proven live: match_all →
2899 hits, term org=default → 0).

Producer-side fix (the tenant filter is correct — dropping it would let one
tenant's cost model count another's traffic):
- aggregator reads x-offgrid-org off the request and stamps  on the
  observability record (absent ⇒ null: honestly unattributed, never guessed).
- new PURE gatewayAttribution({orgId,userId}) in lib/gateway.ts — one place that
  builds the attribution headers, so call sites stop hand-writing them. Blank
  values are DROPPED rather than sent empty (an empty header would attribute the
  call to org "", worse than honestly unattributed).
- chat/stream (the interactive inference path, where orgId is already in scope)
  now sends both headers.

4 unit tests on the pure helper. Suite 5079 pass / 0 fail; coverage green.

HONEST REMAINDER: agent/app-run inference still ships unattributed — gatewayAnswer
threads  but not orgId, so that needs the param threaded through the
planner/compose signatures. Historical docs stay unattributed by design (guessing
an org for past traffic would fabricate attribution).

Author: alichherawalla

scripts/gateway-aggregator.mjs

@@ -593,6 +593,11 @@ async function handle(req, res, chunks) {
     // Request-side context for the observability record (A + D above).
     const caller = String(req.headers['user-agent'] || '').slice(0, 80);
     const corrId = String(req.headers['x-offgrid-run'] || req.headers['x-request-id'] || '');
+    // TENANT ORG attribution (G-GATEWAY-INDEX-ORG). The console's observability surfaces filter the
+    // offgrid-gateway index by `term: { org }` for tenant isolation, so a doc with no `org` is
+    // invisible to Insights → Analytics/Accounting. The console (which owns the request-scoped tenant)
+    // sends it as x-offgrid-org; absent ⇒ null (honestly unattributed, never guessed).
+    const org = String(req.headers['x-offgrid-org'] || '').trim().slice(0, 64) || null;
     // PA-15 — per-tenant gateway ATTRIBUTION. When this request arrived on a provisioned per-tenant
     // gateway host ("<slug5><rand5>-gateway.<apex>"), the tunnel/Caddy forwards the original host in
     // the Host header. Resolve it (pure) so the request is attributed to that tenant's gateway on the
@@ -697,12 +702,13 @@ async function handle(req, res, chunks) {
           tps, finish, toolCalls, reasoning: reasoning.slice(0, 2000), caller, corrId, params, msgs,
           tenantGateway: tenantGateway ? tenantGateway.label : null,
           tenantGatewayHost: tenantGateway ? inboundHost.toLowerCase() : null,
+          org,
           input: promptText(body), output: output.slice(0, 2000),
         });
       });
     });
     up.on('error', (e) => {
-      record({ gateway: target.name, model: body.model || target.model, modelServed: target.model, kind, status: 502, ms: Date.now() - started, bytes: 0, tokens: 0, caller, corrId, params, msgs, tenantGateway: tenantGateway ? tenantGateway.label : null, tenantGatewayHost: tenantGateway ? inboundHost.toLowerCase() : null, input: promptText(body), output: `(error: ${e.message})` });
+      record({ gateway: target.name, model: body.model || target.model, modelServed: target.model, kind, status: 502, ms: Date.now() - started, bytes: 0, tokens: 0, caller, corrId, params, msgs, tenantGateway: tenantGateway ? tenantGateway.label : null, tenantGatewayHost: tenantGateway ? inboundHost.toLowerCase() : null, org, input: promptText(body), output: `(error: ${e.message})` });
       json(res, 502, { error: { message: `gateway ${target.name} (${target.host}) error: ${e.message}`, type: 'upstream_error' } });
     });
     up.setTimeout(Number(process.env.OFFGRID_GATEWAY_UPSTREAM_TIMEOUT_MS || 300000), () => up.destroy(new Error('upstream timeout')));

src/app/api/v1/chat/stream/route.ts

@@ -61,7 +61,7 @@ import { DEFAULT_ORG } from '@/lib/tenancy-policy';
 export const dynamic = 'force-dynamic';
 export const maxDuration = 300;
 
-import { GATEWAY_URL, gatewayHeaders } from '@/lib/gateway';
+import { GATEWAY_URL, gatewayAttribution, gatewayHeaders } from '@/lib/gateway';
 const enc = new TextEncoder();
 
 type ContentPart =
@@ -588,7 +588,12 @@ export async function POST(req: Request) {
         method: 'POST',
         // x-offgrid-user attributes gateway spend to the real signed-in user (captured into the
         // gateway's OpenSearch log as `caller`) rather than the console's user-agent.
-        headers: gatewayHeaders({ 'content-type': 'application/json', 'x-offgrid-user': userId }),
+        // Attribution (user + tenant org) so the aggregator's observability doc is attributable and
+        // the org-scoped Insights surfaces can actually see it (G-GATEWAY-INDEX-ORG).
+        headers: gatewayHeaders({
+          'content-type': 'application/json',
+          ...gatewayAttribution({ userId, orgId }),
+        }),
         body: JSON.stringify(payload),
         signal: AbortSignal.timeout(290000),
       }).catch(() => null);

src/lib/gateway.ts

@@ -20,6 +20,27 @@ export const GATEWAY_CONTROL_URL = ENDPOINTS.controlUrl;
 
 const GATEWAY_API_KEY = process.env.OFFGRID_GATEWAY_API_KEY ?? '';
 
+/**
+ * ATTRIBUTION headers for an inference call (PURE). The aggregator stamps these onto the
+ * `offgrid-gateway` observability doc, and the console's Insights surfaces filter that index by
+ * `term: { org }` for tenant isolation — so a call that omits the org lands as an UNATTRIBUTED doc
+ * that those surfaces can never show (G-GATEWAY-INDEX-ORG). Every inference call site should spread
+ * this so attribution is stamped one way, in one place, instead of hand-written per call.
+ *
+ * Blank/whitespace values are DROPPED rather than sent empty, so an unknown user/org stays honestly
+ * absent instead of becoming an empty-string attribution.
+ */
+export function gatewayAttribution(
+  attribution: { orgId?: string | null; userId?: string | null } = {},
+): Record<string, string> {
+  const out: Record<string, string> = {};
+  const org = attribution.orgId?.trim();
+  const user = attribution.userId?.trim();
+  if (org) out['x-offgrid-org'] = org;
+  if (user) out['x-offgrid-user'] = user;
+  return out;
+}
+
 /**
  * Synchronous gateway headers (LEGACY seam). Emits exactly what the console sent before the broker:
  * the static `x-api-key` when configured, else no auth. Kept for the many synchronous call sites that

test/gateway-attribution.test.ts

@@ -0,0 +1,33 @@
+import assert from 'node:assert/strict';
+import { test } from 'node:test';
+import { gatewayAttribution } from '@/lib/gateway';
+
+// The aggregator stamps these headers onto the offgrid-gateway observability doc, and the console's
+// Insights surfaces filter that index by `term: { org }` for TENANT ISOLATION. So a call that omits
+// the org lands as a doc those surfaces can never show (G-GATEWAY-INDEX-ORG). PURE — no I/O.
+
+test('stamps both org and user when supplied', () => {
+  assert.deepEqual(gatewayAttribution({ orgId: 'org_bharat', userId: 'rm@bank.test' }), {
+    'x-offgrid-org': 'org_bharat',
+    'x-offgrid-user': 'rm@bank.test',
+  });
+});
+
+test('DROPS blank/whitespace/missing values instead of sending empty attribution', () => {
+  // An empty header would attribute the call to org "" — worse than honestly unattributed.
+  assert.deepEqual(gatewayAttribution({ orgId: '', userId: '   ' }), {});
+  assert.deepEqual(gatewayAttribution({}), {});
+  assert.deepEqual(gatewayAttribution(), {});
+  assert.deepEqual(gatewayAttribution({ orgId: null, userId: null }), {});
+});
+
+test('trims surrounding whitespace so the aggregator stores a clean term value', () => {
+  assert.deepEqual(gatewayAttribution({ orgId: '  org_suraksha  ' }), {
+    'x-offgrid-org': 'org_suraksha',
+  });
+});
+
+test('either half can stand alone (a system run may have an org but no user)', () => {
+  assert.deepEqual(gatewayAttribution({ orgId: 'default' }), { 'x-offgrid-org': 'default' });
+  assert.deepEqual(gatewayAttribution({ userId: 'ops@x' }), { 'x-offgrid-user': 'ops@x' });
+});