Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/ai-metadata-public-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'evlog': minor
---

Expose AI SDK execution metadata as a public API on `AILogger`. Three new methods let app code read the same data that gets attached to wide events: `getMetadata()` returns an immutable snapshot of the run (model, provider, tokens, calls, steps, tool calls, cost, finish reason, embeddings), `getEstimatedCost()` returns the dollar cost computed from the configured pricing map, and `onUpdate(cb)` subscribes to incremental snapshots emitted on every step, embedding, error, and integration finish (returns an unsubscribe function). New types `AIMetadata` (alias for `AIEventData`) and `AIMetadataListener` are exported. `model` and `provider` on `AIMetadata` are now optional to reflect early-snapshot reality (e.g. embedding-only runs).
127 changes: 126 additions & 1 deletion apps/docs/content/2.logging/5.ai-sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,12 +115,15 @@ Your wide event now includes:

## How It Works

`createAILogger(log, options?)` returns an `AILogger` with two methods:
`createAILogger(log, options?)` returns an `AILogger` with the following methods:

| Method | Description |
|--------|-------------|
| `wrap(model)` | Wraps a language model with middleware. Accepts a model string (e.g. `'anthropic/claude-sonnet-4.6'`) or a `LanguageModelV3` object. Works with `generateText`, `streamText`, and `ToolLoopAgent`. Also works with pre-wrapped models (e.g. from supermemory). |
| `captureEmbed(result)` | Manually captures token usage, model info, and dimensions from `embed()` or `embedMany()` results (embedding models use a different type). |
| `getMetadata()` | Returns a snapshot of the current execution metadata (`AIMetadata`) — same shape as the `ai` field on the wide event. Safe to call inside `onFinish`, after `await generateText()`, or while a stream is in progress. |
| `getEstimatedCost()` | Returns the current estimated cost in dollars, or `undefined` if no `cost` map was provided. Convenience for `getMetadata().estimatedCost`. |
| `onUpdate(callback)` | Subscribes to metadata updates. Fires on every step, every `captureEmbed` call, on errors, and on `createEvlogIntegration`'s `onFinish`. Returns an unsubscribe function. |

The middleware intercepts calls at the provider level. It does not touch your callbacks, prompts, or responses. Captured data flows through the normal evlog pipeline (sampling, enrichers, drains) and ends up in Axiom, Better Stack, or wherever you drain to.

Expand Down Expand Up @@ -360,6 +363,128 @@ import { anthropic } from '@ai-sdk/anthropic'
const model = ai.wrap(anthropic('claude-sonnet-4.6'))
```

## Accessing Metadata in Your Code

The wide event already contains the full metadata object, but you often want the same data inside your handler — to persist it, surface it to end-users, bill against it, or stream incremental progress to the client.

`AILogger` exposes three methods for that, with no need to touch internal state:

### `getMetadata()` — final snapshot

Returns a structured `AIMetadata` object that mirrors the `ai` field on the wide event. Safe to call at any point, including after the run completes or inside the AI SDK's `onFinish`:

```typescript [server/api/chat.post.ts]
import { useLogger } from 'evlog'
import { createAILogger } from 'evlog/ai'
import { generateText } from 'ai'

export default defineEventHandler(async (event) => {
const log = useLogger(event)
const ai = createAILogger(log, {
cost: { 'claude-sonnet-4.6': { input: 3, output: 15 } },
})

await generateText({
model: ai.wrap('anthropic/claude-sonnet-4.6'),
prompt: 'Summarize this document',
})

const metadata = ai.getMetadata()

await db.aiRuns.insert({
userId: event.context.userId,
model: metadata.model,
inputTokens: metadata.inputTokens,
outputTokens: metadata.outputTokens,
estimatedCost: metadata.estimatedCost,
finishReason: metadata.finishReason,
responseId: metadata.responseId,
})

return { ok: true }
})
```

The snapshot is a fresh copy: mutating it never affects the underlying state or subsequent calls.

### `getEstimatedCost()` — quick cost check

Convenience for `getMetadata().estimatedCost`. Returns the cost in dollars, or `undefined` if no `cost` map was provided or the model is not in the map.

```typescript
const ai = createAILogger(log, {
cost: { 'claude-sonnet-4.6': { input: 3, output: 15 } },
})

await generateText({ model: ai.wrap('anthropic/claude-sonnet-4.6'), prompt })

const cost = ai.getEstimatedCost()
console.log(`This call cost $${cost?.toFixed(4)}`)
```

### `onUpdate(callback)` — incremental updates

Subscribe to metadata updates. The callback fires every time the underlying state flushes:

- Once per step in multi-step agent runs
- Once per `captureEmbed` call
- On model errors
- On `createEvlogIntegration`'s `onFinish`

Each invocation receives a fresh snapshot. Returns an unsubscribe function. Subscriber errors are isolated and never break the AI flow.

```typescript [server/api/agent.post.ts]
import { ToolLoopAgent, createAgentUIStreamResponse, stepCountIs } from 'ai'
import { useLogger } from 'evlog'
import { createAILogger } from 'evlog/ai'

export default defineEventHandler(async (event) => {
const log = useLogger(event)
const { messages } = await readBody(event)
const ai = createAILogger(log)

ai.onUpdate((metadata) => {
pushToClient(event, {
type: 'ai-progress',
step: metadata.steps,
tokens: metadata.totalTokens,
cost: metadata.estimatedCost,
})
})

const agent = new ToolLoopAgent({
model: ai.wrap('anthropic/claude-sonnet-4.6'),
tools: { searchWeb, queryDatabase },
stopWhen: stepCountIs(5),
})

return createAgentUIStreamResponse({ agent, uiMessages: messages })
})
```

For one-off cleanup:

```typescript
const off = ai.onUpdate((metadata) => { /* ... */ })
// later
off()
```

### `AIMetadata` shape

`AIMetadata` is a public type alias for the snapshot returned by `getMetadata()` and passed to `onUpdate` listeners. It has the same shape as the `ai` field on the wide event — see [Captured Data](#captured-data) for the full field reference.

```typescript
import type { AIMetadata, AIMetadataListener } from 'evlog/ai'

function handleProgress(metadata: AIMetadata) {
console.log(`${metadata.calls} calls, $${metadata.estimatedCost ?? 0}`)
}

const listener: AIMetadataListener = handleProgress
ai.onUpdate(listener)
```

## Telemetry Integration

For deeper observability — tool execution timing, success/failure tracking, and total generation wall time — use `createEvlogIntegration()`. It implements the AI SDK's `TelemetryIntegration` interface and captures data that middleware alone cannot see.
Expand Down
31 changes: 31 additions & 0 deletions apps/nuxthub-playground/app/components/AiChat.vue
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,25 @@ function getToolInput(part: any): { query?: string } {
function getToolOutput(part: any): { count?: number, error?: string } | undefined {
return part.output
}

interface AiMessageMetadata {
calls?: number
totalTokens?: number
estimatedCost?: number
finishReason?: string
}

function getAiMetadata(message: any): AiMessageMetadata | undefined {
const meta = message?.metadata as AiMessageMetadata | undefined
if (!meta || (meta.calls === undefined && meta.totalTokens === undefined)) return undefined
return meta
}

function formatCost(cost: number | undefined): string {
if (cost === undefined) return '—'
if (cost === 0) return '$0'
return `$${cost.toFixed(6)}`
}
</script>

<template>
Expand All @@ -74,6 +93,18 @@ function getToolOutput(part: any): { count?: number, error?: string } | undefine
</div>

<template v-for="(message, index) in chat.messages" :key="message.id">
<div
v-if="message.role === 'assistant' && getAiMetadata(message)"
style="flex-shrink: 0; align-self: flex-start; max-width: 85%; padding: 0.3rem 0.55rem; border-radius: 6px; font-size: 0.7rem; color: #4a5568; background: #edf2f7; border: 1px solid #e2e8f0; font-family: monospace; display: inline-flex; gap: 0.5rem; flex-wrap: wrap;"
>
<span>step {{ getAiMetadata(message)!.calls }}</span>
<span>·</span>
<span>{{ getAiMetadata(message)!.totalTokens }} tokens</span>
<span>·</span>
<span>{{ formatCost(getAiMetadata(message)!.estimatedCost) }}</span>
<span v-if="getAiMetadata(message)!.finishReason">·</span>
<span v-if="getAiMetadata(message)!.finishReason">{{ getAiMetadata(message)!.finishReason }}</span>
</div>
<template v-for="(part, pi) in message.parts" :key="`${message.id}-${part.type}-${pi}`">
<!-- User text -->
<div
Expand Down
91 changes: 91 additions & 0 deletions apps/nuxthub-playground/app/components/LogGenerator.vue
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
<script setup lang="ts">
import type { AIMetadata } from 'evlog/ai'

interface AiMetadataResponse {
status: string
text: string
metadata: AIMetadata
estimatedCost: number | undefined
history: Array<{ step: number, totalTokens: number, estimatedCost: number | undefined }>
}

const lastResult = ref('')
const aiMetadata = ref<AiMetadataResponse | null>(null)
const loadingAiMetadata = ref(false)

async function fire(url: string) {
try {
Expand All @@ -17,6 +29,24 @@ async function fireAll() {
fire('/api/test/warn'),
])
}

async function fetchAiMetadata() {
loadingAiMetadata.value = true
aiMetadata.value = null
try {
aiMetadata.value = await $fetch<AiMetadataResponse>('/api/test/ai-metadata')
} catch (err: any) {
lastResult.value = `/api/test/ai-metadata → Error: ${err.statusCode || err.message}`
} finally {
loadingAiMetadata.value = false
}
}

function formatCost(cost: number | undefined): string {
if (cost === undefined) return '—'
if (cost === 0) return '$0'
return `$${cost.toFixed(6)}`
}
</script>

<template>
Expand All @@ -42,5 +72,66 @@ async function fireAll() {
<p v-if="lastResult" style="margin-top: 0.5rem; color: #666; font-size: 0.85rem; word-break: break-all; overflow-wrap: anywhere;">
{{ lastResult }}
</p>

<div style="margin-top: 1rem; padding-top: 1rem; border-top: 1px solid #e2e8f0;">
<h2 style="margin-bottom: 0.25rem;">
AI Metadata API
</h2>
<p style="margin: 0 0 0.5rem; font-size: 0.78rem; color: #718096;">
Calls <code>generateText</code> and reads back <code>ai.getMetadata()</code>, <code>ai.getEstimatedCost()</code>, and snapshots collected via <code>ai.onUpdate()</code>.
</p>
<button :disabled="loadingAiMetadata" @click="fetchAiMetadata">
{{ loadingAiMetadata ? 'Running...' : 'Run AI metadata demo' }}
</button>

<div v-if="aiMetadata" style="margin-top: 0.75rem; display: grid; gap: 0.5rem;">
<div style="border: 1px solid #e2e8f0; border-radius: 8px; padding: 0.6rem 0.75rem; background: #f8fafc;">
<div style="font-size: 0.72rem; text-transform: uppercase; letter-spacing: 0.05em; color: #718096; margin-bottom: 0.35rem;">
Final snapshot · ai.getMetadata()
</div>
<div style="display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 0.25rem 0.75rem; font-size: 0.78rem;">
<span style="color: #4a5568;">Model</span>
<code>{{ aiMetadata.metadata.model ?? '—' }}</code>
<span style="color: #4a5568;">Provider</span>
<code>{{ aiMetadata.metadata.provider ?? '—' }}</code>
<span style="color: #4a5568;">Calls</span>
<code>{{ aiMetadata.metadata.calls }}</code>
<span style="color: #4a5568;">Input tokens</span>
<code>{{ aiMetadata.metadata.inputTokens }}</code>
<span style="color: #4a5568;">Output tokens</span>
<code>{{ aiMetadata.metadata.outputTokens }}</code>
<span style="color: #4a5568;">Total tokens</span>
<code>{{ aiMetadata.metadata.totalTokens }}</code>
<span style="color: #4a5568;">Finish reason</span>
<code>{{ aiMetadata.metadata.finishReason ?? '—' }}</code>
<span style="color: #4a5568;">Estimated cost</span>
<code>{{ formatCost(aiMetadata.estimatedCost) }}</code>
</div>
</div>

<div style="border: 1px solid #e2e8f0; border-radius: 8px; padding: 0.6rem 0.75rem; background: #f8fafc;">
<div style="font-size: 0.72rem; text-transform: uppercase; letter-spacing: 0.05em; color: #718096; margin-bottom: 0.35rem;">
ai.onUpdate() history · {{ aiMetadata.history.length }} update{{ aiMetadata.history.length === 1 ? '' : 's' }}
</div>
<div v-if="aiMetadata.history.length === 0" style="font-size: 0.78rem; color: #a0aec0;">
No updates received.
</div>
<ol v-else style="margin: 0; padding-left: 1.2rem; font-size: 0.78rem; color: #2d3748;">
<li v-for="(entry, i) in aiMetadata.history" :key="i" style="margin: 0.1rem 0;">
step {{ entry.step }} · {{ entry.totalTokens }} tokens · {{ formatCost(entry.estimatedCost) }}
</li>
</ol>
</div>

<details style="border: 1px solid #e2e8f0; border-radius: 8px; padding: 0.5rem 0.75rem; background: #fff;">
<summary style="cursor: pointer; font-size: 0.78rem; color: #4a5568;">
Model output
</summary>
<p style="margin: 0.4rem 0 0; font-size: 0.82rem; line-height: 1.45;">
{{ aiMetadata.text }}
</p>
</details>
</div>
</div>
</section>
</template>
28 changes: 28 additions & 0 deletions apps/nuxthub-playground/server/api/chat.post.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,17 @@ export default defineEventHandler(async (event) => {
},
})

ai.onUpdate((metadata) => {
logger.set({
aiLive: {
step: metadata.calls,
totalTokens: metadata.totalTokens,
estimatedCost: metadata.estimatedCost,
finishReason: metadata.finishReason,
},
})
})

try {
const agent = new ToolLoopAgent({
model: ai.wrap('google/gemini-3-flash'),
Expand All @@ -84,6 +95,23 @@ export default defineEventHandler(async (event) => {
return createAgentUIStreamResponse({
agent,
uiMessages: messages,
messageMetadata: ({ part }) => {
if (part.type === 'finish-step' || part.type === 'finish') {
const snapshot = ai.getMetadata()
return {
calls: snapshot.calls,
totalTokens: snapshot.totalTokens,
estimatedCost: snapshot.estimatedCost,
finishReason: snapshot.finishReason,
}
}
},
onFinish: () => {
logger.set({
aiFinalMetadata: ai.getMetadata(),
aiFinalCost: ai.getEstimatedCost(),
})
},
})
} catch (error) {
throw createError({
Expand Down
Loading
Loading