feat: add validation framework with conversation logger

Adds the /validate skill, conversation logging, and associated tooling.

**Validation framework**
- Add `src/skill/validate/SKILL.md` — 7-step validation workflow:
  steps 1–5 fetch and analyse traces, step 6 generates an HTML dashboard,
  step 7 categorises groundedness failures
- Add `src/skill/validate/batch_validate.py` — calls Altimate backend
  SSE endpoints (`/validate`, `/validate/session`, `/validate/date-range`)
  and writes structured JSON results; hardcoded prod URL and token, no
  dotenv dependency
- Embed both files at build time via `ALTIMATE_VALIDATE_SKILL_MD` and
  `ALTIMATE_VALIDATE_BATCH_PY` defines in `build.ts`
- Replace LLM-based intent classifier with deterministic `/validate`
  prefix check via `resolvePrompt` helper in `session.ts`, covering both
  `prompt` and `prompt_async` routes
- Delete `src/altimate/intent/validation-classifier.ts` and its directory

**Conversation logger**
- Add `src/session/conversation-logger.ts` — logs the last user+assistant
  turn (prompt, normalised tool calls, final response, cost/tokens) to
  `POST /log-conversation` on session idle; collects up to 500 messages
  to capture multi-step sessions; fire-and-forget, never blocks the UI
- Gate logger in `bootstrap.ts` behind `ALTIMATE_LOGGER_DISABLED=true`
- Add `docs/docs/configure/logging.md` documenting what is collected and
  how to opt out
- Add Data Collection section to `README.md` with opt-out instructions

**Build / cleanup**
- Remove duplicate `ALTIMATE_CLI_*` constants from `build.ts`
- Remove `load_dotenv` import and dead path-resolution block from
  `batch_validate.py`
- Remove Langfuse reference from `SKILL.md` comment
- Update all `uv run` invocations in `SKILL.md` to drop
  `--with python-dotenv`

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: Bharatram-altimate-ai

.github/workflows/pr-standards.yml

@@ -32,7 +32,7 @@ jobs:
               owner: context.repo.owner,
               repo: context.repo.repo,
               path: '.github/TEAM_MEMBERS',
-              ref: 'dev'
+              ref: 'main'
             });
             const members = Buffer.from(file.content, 'base64').toString().split('\n').map(l => l.trim()).filter(Boolean);
             if (members.includes(login)) {
@@ -179,7 +179,7 @@ jobs:
               owner: context.repo.owner,
               repo: context.repo.repo,
               path: '.github/TEAM_MEMBERS',
-              ref: 'dev'
+              ref: 'main'
             });
             const members = Buffer.from(file.content, 'base64').toString().split('\n').map(l => l.trim()).filter(Boolean);
             if (members.includes(login)) {

.gitignore

@@ -23,8 +23,37 @@ target
 .scripts
 .direnv/
 
+# Python
+__pycache__/
+*.pyc
+*.pyo
+*.egg-info/
+
+# SQLite databases (feedback store creates these at runtime)
+*.db
+
+# Runtime logs
+*.log
+logs/
+
+# Large intermediate files at repo root (generated during benchmark runs)
+/queries.json
+/queries_1k.json
+/results/
+
+# Local runtime config
+.altimate-code/
+
+# Commit message scratch files
+.github/meta/
+
+# Experiment / simulation artifacts
+/data/
+/experiments/
+/models/
+/simulation/
+
 # Local dev files
 opencode-dev
-logs/
 *.bun-build
 tsconfig.tsbuildinfo

README.md

@@ -157,6 +157,20 @@ Full docs at **[altimate-code.sh](https://altimate-code.sh)**.
 - [Agent Modes](https://altimate-code.sh/data-engineering/agent-modes/)
 - [Configuration](https://altimate-code.sh/configure/model-providers/)
 
+## Data Collection
+
+Altimate Code logs conversation turns (prompt, tool calls, and assistant response) to improve validation quality and agent behavior. Logs are sent to Altimate's backend and are not shared with third parties.
+
+**To opt out:**
+
+```bash
+export ALTIMATE_LOGGER_DISABLED=true
+```
+
+Add it to your shell profile (`~/.zshrc`, `~/.bashrc`) to make it permanent.
+
+See [`docs/docs/configure/logging.md`](docs/docs/configure/logging.md) for details on what is collected.
+
 ## Community & Contributing
 
 - **Issues**: [GitHub Issues](https://github.com/AltimateAI/altimate-code/issues)

bun.lock

@@ -0,0 +1,60 @@
+# Conversation Logging
+
+Altimate Code automatically logs each conversation turn to the Altimate backend. This powers validation, audit, and quality analysis features. Logging is **enabled by default** — no configuration is required to activate it.
+
+## What Is Logged
+
+Each turn (one user prompt + all assistant responses) sends the following to the Altimate backend:
+
+| Field | Description |
+|-------|-------------|
+| `session_id` | The current session identifier |
+| `conversation_id` | The assistant message ID for this turn |
+| `user_id` | Your email or username (from your Altimate account) |
+| `user_prompt` | The text of your message |
+| `parts` | All reasoning, text, and tool call/response parts from the assistant |
+| `final_response` | The last text response from the assistant |
+| `metadata` | Model ID, token counts, and cost for the turn |
+
+Logging fires after the session becomes idle (i.e., after the assistant finishes responding). Up to 500 messages are captured per turn to ensure complete coverage of multi-step agentic sessions.
+
+## Why We Log
+
+Conversation logs are used to:
+
+- **Validate AI responses** — power the `/validate` skill that audits factual claims against source data
+- **Quality analysis** — identify recurring failure patterns across sessions
+- **Audit trails** — provide a record of what the assistant did and why
+
+## Disabling Logging
+
+Logging is on by default. To disable it, set the following environment variable before starting Altimate Code:
+
+```bash
+export ALTIMATE_LOGGER_DISABLED=true
+```
+
+To make this permanent, add it to your shell profile (`~/.zshrc`, `~/.bashrc`, etc.):
+
+```bash
+echo 'export ALTIMATE_LOGGER_DISABLED=true' >> ~/.zshrc
+source ~/.zshrc
+```
+
+To re-enable logging, unset the variable:
+
+```bash
+unset ALTIMATE_LOGGER_DISABLED
+```
+
+Setting `ALTIMATE_LOGGER_DISABLED=false` is equivalent to not setting it — logging will be active.
+
+## Network
+
+Conversation logs are sent to:
+
+| Endpoint | Purpose |
+|----------|---------|
+| `apimi.tryaltimate.com` | Conversation log ingestion |
+
+Requests are fire-and-forget — a failed log request does not affect your session in any way.

docs/docs/configure/logging.md

@@ -0,0 +1,31 @@
+---
+name: altimate-setup
+description: Configure Altimate platform credentials for datamate and API access
+---
+
+# Altimate Setup
+
+Guide the user through configuring their Altimate platform credentials.
+
+## Steps
+
+1. **Check existing config**: Read `~/.altimate/altimate.json`. If it exists and is valid, show the current config (mask the API key) and ask if they want to update it.
+
+2. **Gather credentials**: Ask the user for:
+   - **Altimate URL** (default: `https://api.myaltimate.com`)
+   - **Instance name** (their tenant/org name, e.g. `megatenant`)
+   - **API key** (from Altimate platform settings)
+   - **MCP server URL** (optional, default: `https://mcpserver.getaltimate.com/sse`)
+
+3. **Write config**: Create `~/.altimate/` directory if needed, then write `~/.altimate/altimate.json`:
+   ```json
+   {
+     "altimateUrl": "<url>",
+     "altimateInstanceName": "<instance>",
+     "altimateApiKey": "<key>",
+     "mcpServerUrl": "<mcp-url>"
+   }
+   ```
+   Then set permissions to owner-only: `chmod 600 ~/.altimate/altimate.json`
+
+4. **Validate**: Call the `datamate_manager` tool with `operation: "list"` to verify the credentials work. Report success or failure to the user.

packages/altimate-code/src/provider/models-snapshot.ts

@@ -8,7 +8,7 @@
   "scripts": {
     "typecheck": "tsgo --noEmit",
     "test": "bun test --timeout 30000",
-    "build": "bun run script/build.ts",
+    "build": "bun run script/build.ts && bun run --conditions=browser ./src/index.ts validate install",
     "dev": "bun run --conditions=browser ./src/index.ts",
     "db": "bun drizzle-kit"
   },

packages/opencode/.opencode/skills/altimate-setup/SKILL.md

@@ -71,6 +71,11 @@ const migrations = await Promise.all(
 )
 console.log(`Loaded ${migrations.length} migrations`)
 
+// Load validate skill assets for embedding
+const validateSkillMd = await Bun.file(path.join(dir, "src/skill/validate/SKILL.md")).text()
+const validateBatchPy = await Bun.file(path.join(dir, "src/skill/validate/batch_validate.py")).text()
+console.log("Loaded validate skill assets")
+
 const singleFlag = process.argv.includes("--single")
 const baselineFlag = process.argv.includes("--baseline")
 const skipInstall = process.argv.includes("--skip-install")
@@ -220,6 +225,8 @@ for (const item of targets) {
       OPENCODE_LIBC: item.os === "linux" ? `'${item.abi ?? "glibc"}'` : "undefined",
       OPENCODE_MIGRATIONS: JSON.stringify(migrations),
       OPENCODE_CHANGELOG: JSON.stringify(changelog),
+      ALTIMATE_VALIDATE_SKILL_MD: JSON.stringify(validateSkillMd),
+      ALTIMATE_VALIDATE_BATCH_PY: JSON.stringify(validateBatchPy),
       OTUI_TREE_SITTER_WORKER_PATH: bunfsRoot + workerRelativePath,
     },
   })

packages/opencode/package.json

@@ -0,0 +1,178 @@
+import z from "zod"
+import path from "path"
+import { Global } from "../../global"
+import { Filesystem } from "../../util/filesystem"
+
+const DEFAULT_MCP_URL = "https://mcpserver.getaltimate.com/sse"
+
+const AltimateCredentials = z.object({
+  altimateUrl: z.string(),
+  altimateInstanceName: z.string(),
+  altimateApiKey: z.string(),
+  mcpServerUrl: z.string().optional(),
+})
+type AltimateCredentials = z.infer<typeof AltimateCredentials>
+
+const DatamateSummary = z.object({
+  id: z.coerce.string(),
+  name: z.string(),
+  description: z.string().nullable().optional(),
+  integrations: z
+    .array(
+      z.object({
+        id: z.coerce.string(),
+        tools: z.array(z.object({ key: z.string() })).optional(),
+      }),
+    )
+    .nullable()
+    .optional(),
+  memory_enabled: z.boolean().optional(),
+  privacy: z.string().optional(),
+})
+
+const IntegrationSummary = z.object({
+  id: z.coerce.string(),
+  name: z.string().optional(),
+  description: z.string().nullable().optional(),
+  tools: z
+    .array(
+      z.object({
+        key: z.string(),
+        name: z.string().optional(),
+        enable_all: z.array(z.string()).optional(),
+      }),
+    )
+    .optional(),
+})
+
+export namespace AltimateApi {
+  export function credentialsPath(): string {
+    return path.join(Global.Path.home, ".altimate", "altimate.json")
+  }
+
+  export async function isConfigured(): Promise<boolean> {
+    return Filesystem.exists(credentialsPath())
+  }
+
+  export async function getCredentials(): Promise<AltimateCredentials> {
+    const p = credentialsPath()
+    if (!(await Filesystem.exists(p))) {
+      throw new Error(`Altimate credentials not found at ${p}`)
+    }
+    const raw = JSON.parse(await Filesystem.readText(p))
+    return AltimateCredentials.parse(raw)
+  }
+
+  async function request(creds: AltimateCredentials, method: string, endpoint: string, body?: unknown) {
+    const url = `${creds.altimateUrl}${endpoint}`
+    const res = await fetch(url, {
+      method,
+      headers: {
+        "Content-Type": "application/json",
+        Authorization: `Bearer ${creds.altimateApiKey}`,
+        "x-tenant": creds.altimateInstanceName,
+      },
+      ...(body ? { body: JSON.stringify(body) } : {}),
+    })
+    if (!res.ok) {
+      throw new Error(`API ${method} ${endpoint} failed with status ${res.status}`)
+    }
+    return res.json()
+  }
+
+  export async function listDatamates() {
+    const creds = await getCredentials()
+    const data = await request(creds, "GET", "/datamates/")
+    const list = Array.isArray(data) ? data : (data.datamates ?? data.data ?? [])
+    return list.map((d: unknown) => DatamateSummary.parse(d)) as z.infer<typeof DatamateSummary>[]
+  }
+
+  export async function getDatamate(id: string) {
+    const creds = await getCredentials()
+    try {
+      const data = await request(creds, "GET", `/datamates/${id}/summary`)
+      const raw = data.datamate ?? data
+      return DatamateSummary.parse(raw)
+    } catch (e) {
+      // Fallback to list if single-item endpoint is unavailable (404)
+      if (e instanceof Error && e.message.includes("status 404")) {
+        const all = await listDatamates()
+        const found = all.find((d) => d.id === id)
+        if (!found) {
+          throw new Error(`Datamate with ID ${id} not found`)
+        }
+        return found
+      }
+      throw e
+    }
+  }
+
+  export async function createDatamate(payload: {
+    name: string
+    description?: string
+    integrations?: Array<{ id: string; tools: Array<{ key: string }> }>
+    memory_enabled?: boolean
+    privacy?: string
+  }) {
+    const creds = await getCredentials()
+    const data = await request(creds, "POST", "/datamates/", payload)
+    // Backend returns { id: number } for create
+    const id = String(data.id ?? data.datamate?.id)
+    return { id, name: payload.name }
+  }
+
+  export async function updateDatamate(
+    id: string,
+    payload: {
+      name?: string
+      description?: string
+      integrations?: Array<{ id: string; tools: Array<{ key: string }> }>
+      memory_enabled?: boolean
+      privacy?: string
+    },
+  ) {
+    const creds = await getCredentials()
+    const data = await request(creds, "PATCH", `/datamates/${id}`, payload)
+    const raw = data.datamate ?? data
+    return DatamateSummary.parse(raw)
+  }
+
+  export async function deleteDatamate(id: string) {
+    const creds = await getCredentials()
+    await request(creds, "DELETE", `/datamates/${id}`)
+  }
+
+  export async function listIntegrations() {
+    const creds = await getCredentials()
+    const data = await request(creds, "GET", "/datamate_integrations/")
+    const list = Array.isArray(data) ? data : (data.integrations ?? data.data ?? [])
+    return list.map((d: unknown) => IntegrationSummary.parse(d)) as z.infer<typeof IntegrationSummary>[]
+  }
+
+  /** Resolve integration IDs to full integration objects with all tools enabled (matching frontend behavior). */
+  export async function resolveIntegrations(
+    integrationIds: string[],
+  ): Promise<Array<{ id: string; tools: Array<{ key: string }> }>> {
+    const allIntegrations = await listIntegrations()
+    return integrationIds.map((id) => {
+      const def = allIntegrations.find((i) => i.id === id)
+      const tools =
+        def?.tools?.flatMap((t) => (t.enable_all ?? [t.key]).map((k) => ({ key: k }))) ?? []
+      return { id, tools }
+    })
+  }
+
+  export function buildMcpConfig(creds: AltimateCredentials, datamateId: string) {
+    return {
+      type: "remote" as const,
+      url: creds.mcpServerUrl ?? DEFAULT_MCP_URL,
+      oauth: false as const,
+      headers: {
+        Authorization: `Bearer ${creds.altimateApiKey}`,
+        "x-datamate-id": String(datamateId),
+        "x-tenant": creds.altimateInstanceName,
+        "x-altimate-url": creds.altimateUrl,
+      },
+    }
+  }
+}