Skip to content

Commit d736844

Browse files
FEATURE: Added usagetap support for billing, metering, etc.
1 parent 5e0c3c2 commit d736844

16 files changed

Lines changed: 1588 additions & 0 deletions

README.md

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,71 @@ import { geminiAdapter } from 'promptopskit/gemini';
182182
import { openrouterAdapter } from 'promptopskit/openrouter';
183183
```
184184

185+
## Optional UsageTap Tracking
186+
187+
PromptOpsKit can also help you track provider calls with UsageTap.com while keeping the core render API body-only.
188+
189+
```typescript
190+
import { createPromptOpsKit } from 'promptopskit';
191+
import { createUsageTapClient, runOpenAIWithUsageTap } from 'promptopskit/usagetap';
192+
193+
const kit = createPromptOpsKit({ sourceDir: './prompts' });
194+
const usageTap = createUsageTapClient({ apiKey: process.env.USAGETAP_API_KEY! });
195+
196+
const { request } = await kit.renderPrompt({
197+
path: 'support/reply',
198+
provider: 'openai',
199+
variables: {
200+
user_message: 'How do I reset my password?',
201+
app_context: 'Account settings page',
202+
},
203+
});
204+
205+
const tracked = await runOpenAIWithUsageTap(usageTap, {
206+
begin: {
207+
customerId: 'user_123',
208+
feature: 'chat.send',
209+
requested: { standard: true, premium: true, search: true },
210+
idempotencyKey: 'chat-send-user-123-req-456',
211+
},
212+
request,
213+
entitlementMode: 'apply',
214+
modelTiers: {
215+
standard: 'gpt-5.4-mini',
216+
premium: 'gpt-5.4',
217+
},
218+
toolEntitlements: {
219+
image_tool: 'image',
220+
web_lookup: 'search',
221+
},
222+
invoke: async (requestUsed) => {
223+
const response = await fetch('https://api.openai.com/v1/chat/completions', {
224+
method: 'POST',
225+
headers: {
226+
'Content-Type': 'application/json',
227+
Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
228+
},
229+
body: JSON.stringify(requestUsed.body),
230+
});
231+
232+
return response.json();
233+
},
234+
});
235+
236+
// tracked.response -> vendor JSON response
237+
// tracked.begin -> UsageTap call_begin payload
238+
// tracked.end -> UsageTap call_end payload
239+
// tracked.requestUsed -> effective request after optional entitlement changes
240+
// tracked.effectiveUsage -> usage sent to UsageTap
241+
```
242+
243+
Notes:
244+
- `entitlementMode` defaults to `'off'`. Set it to `'apply'` only when you want UsageTap allowances to mutate a cloned provider request.
245+
- `runOpenRouterWithUsageTap`, `runAnthropicWithUsageTap`, and `runGeminiWithUsageTap` follow the same pattern.
246+
- `extractOpenAIUsage`, `extractAnthropicUsage`, and `extractGeminiUsage` are public if you want to manage UsageTap lifecycle yourself.
247+
248+
For explicit lifecycle control, use `beginUsageTapCall`, `endUsageTapCall`, or `withUsageTapCall` from `promptopskit/usagetap`. Full documentation: [docs/usagetap.md](./docs/usagetap.md).
249+
185250
## Overrides
186251

187252
Define environment and tier overrides in front matter. Precedence: **base → environment → tier → runtime**. Scalars and arrays are replaced, not merged.

docs/index.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ Open-source developer toolkit for managing prompts, system instructions, tools,
99
- [Composition](./composition.md) — Share system instructions across prompts with `includes`
1010
- [Overrides](./overrides.md) — Environment and tier-based overrides for dev/prod/free/pro
1111
- [Providers](./providers.md) — Provider adapters for OpenAI, Anthropic, Gemini, and OpenRouter
12+
- [UsageTap](./usagetap.md) — Optional begin/end call tracking and entitlement-aware provider helpers for UsageTap.com
1213
- [Inline Source](./inline-source.md) — Render prompts from strings without files
1314

1415
## Reference

docs/providers.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,17 @@ interface ProviderAdapter {
5050
}
5151
```
5252

53+
## Optional UsageTap tracking
54+
55+
If you want UsageTap begin/end tracking around a provider call, use the optional `promptopskit/usagetap` helper layer.
56+
57+
- The core adapters still only produce request bodies.
58+
- Provider-specific runners are available for OpenAI, OpenRouter, Anthropic, and Gemini.
59+
- Manual lifecycle control is available through `withUsageTapCall`.
60+
- Entitlement-aware request mutation is opt-in and runs on a cloned request.
61+
62+
See [UsageTap](./usagetap.md) for setup, lifecycle helpers, entitlement behavior, tool gating, standalone usage extractors, and provider examples.
63+
5364
## OpenAI
5465

5566
Body shape: [Chat Completions API](https://platform.openai.com/docs/api-reference/chat)

docs/usagetap.md

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
# UsageTap
2+
3+
PromptOpsKit includes an optional `promptopskit/usagetap` helper layer for UsageTap.com call tracking. It wraps your own provider call with `call_begin` and `call_end` requests while keeping the core prompt rendering API body-only.
4+
5+
## Install surface
6+
7+
UsageTap helpers ship inside the main package under a separate subpath:
8+
9+
```typescript
10+
import {
11+
createUsageTapClient,
12+
runOpenAIWithUsageTap,
13+
runOpenRouterWithUsageTap,
14+
runAnthropicWithUsageTap,
15+
runGeminiWithUsageTap,
16+
withUsageTapCall,
17+
extractOpenAIUsage,
18+
extractAnthropicUsage,
19+
extractGeminiUsage,
20+
} from 'promptopskit/usagetap';
21+
```
22+
23+
The helper layer does not add an SDK dependency. It uses `fetch` directly against `https://api.usagetap.com` and sends:
24+
25+
- `Authorization: Bearer ...`
26+
- `Accept: application/vnd.usagetap.v1+json`
27+
- `Content-Type: application/json`
28+
29+
## Create a client
30+
31+
```typescript
32+
import { createUsageTapClient } from 'promptopskit/usagetap';
33+
34+
const usageTap = createUsageTapClient({
35+
apiKey: process.env.USAGETAP_API_KEY!,
36+
});
37+
```
38+
39+
You can also override `baseUrl` or `fetch` for tests and custom runtimes.
40+
41+
## Track a provider call
42+
43+
```typescript
44+
import { createPromptOpsKit } from 'promptopskit';
45+
import { createUsageTapClient, runOpenAIWithUsageTap } from 'promptopskit/usagetap';
46+
47+
const kit = createPromptOpsKit({ sourceDir: './prompts' });
48+
const usageTap = createUsageTapClient({ apiKey: process.env.USAGETAP_API_KEY! });
49+
50+
const { request } = await kit.renderPrompt({
51+
path: 'support/reply',
52+
provider: 'openai',
53+
variables: {
54+
user_message: 'How do I reset my password?',
55+
app_context: 'Account settings page',
56+
},
57+
});
58+
59+
const tracked = await runOpenAIWithUsageTap(usageTap, {
60+
begin: {
61+
customerId: 'user_123',
62+
feature: 'chat.send',
63+
requested: { standard: true, premium: true, search: true },
64+
idempotencyKey: 'chat-send-user-123-req-456',
65+
},
66+
request,
67+
entitlementMode: 'apply',
68+
modelTiers: {
69+
standard: 'gpt-5.4-mini',
70+
premium: 'gpt-5.4',
71+
},
72+
toolEntitlements: {
73+
image_tool: 'image',
74+
web_lookup: 'search',
75+
},
76+
invoke: async (requestUsed) => {
77+
const response = await fetch('https://api.openai.com/v1/chat/completions', {
78+
method: 'POST',
79+
headers: {
80+
'Content-Type': 'application/json',
81+
Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
82+
},
83+
body: JSON.stringify(requestUsed.body),
84+
});
85+
86+
return response.json();
87+
},
88+
});
89+
90+
tracked.response;
91+
tracked.begin;
92+
tracked.end;
93+
tracked.requestUsed;
94+
tracked.effectiveUsage;
95+
tracked.allowed;
96+
```
97+
98+
`runOpenRouterWithUsageTap`, `runAnthropicWithUsageTap`, and `runGeminiWithUsageTap` follow the same pattern.
99+
100+
## Entitlement mode
101+
102+
`entitlementMode` defaults to `'off'`.
103+
104+
- `'off'`: track the call, but do not change the provider request.
105+
- `'apply'`: clone the provider request and apply UsageTap allowances before invoking the vendor.
106+
107+
When `entitlementMode: 'apply'` is enabled, helpers can:
108+
109+
- Swap to `modelTiers.standard` or `modelTiers.premium`.
110+
- Cap OpenAI or OpenRouter `reasoning_effort`.
111+
- Cap Gemini `thinkingConfig.thinkingBudget`. If no Gemini thinking budget was present, the helper now sets one from the allowed reasoning level.
112+
- Remove built-in OpenAI `web_search` when `allowed.search === false`.
113+
- Remove named tools according to `toolEntitlements` for OpenAI, Anthropic, and Gemini function declarations.
114+
115+
Notes:
116+
117+
- Mutations happen on a cloned request. The original `request` object is left unchanged.
118+
- Built-in tool gating is intentionally narrow today. Only OpenAI `web_search` is handled automatically; other built-ins must be mapped through your own tool policy.
119+
- Applying Gemini reasoning limits can introduce a `thinkingConfig` block even when the original request omitted one.
120+
121+
## Manual lifecycle control
122+
123+
Use `withUsageTapCall` when you want begin/invoke/end tracking without a provider-specific helper.
124+
125+
```typescript
126+
import { createUsageTapClient, withUsageTapCall } from 'promptopskit/usagetap';
127+
128+
const usageTap = createUsageTapClient({ apiKey: process.env.USAGETAP_API_KEY! });
129+
130+
const tracked = await withUsageTapCall(usageTap, {
131+
begin: {
132+
customerId: 'user_123',
133+
feature: 'embeddings.index',
134+
},
135+
invoke: async ({ setUsage, signal }) => {
136+
const response = await fetch('https://api.openai.com/v1/embeddings', {
137+
method: 'POST',
138+
signal,
139+
headers: {
140+
'Content-Type': 'application/json',
141+
Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
142+
},
143+
body: JSON.stringify({
144+
model: 'text-embedding-3-large',
145+
input: 'PromptOpsKit',
146+
}),
147+
}).then((result) => result.json());
148+
149+
setUsage({
150+
modelUsed: 'text-embedding-3-large',
151+
inputTokens: 6,
152+
});
153+
154+
return response;
155+
},
156+
});
157+
```
158+
159+
`withUsageTapCall` also accepts an invoke result shaped like `{ result, usage }` if you prefer to return the usage payload directly.
160+
161+
If the vendor call throws and UsageTap `call_end` also fails, the original vendor error is rethrown and the UsageTap failure is attached as `error.cause` when possible.
162+
163+
## Standalone usage extractors
164+
165+
The provider runners use public extractor helpers that you can call yourself:
166+
167+
- `extractOpenAIUsage(response, meta)`
168+
- `extractAnthropicUsage(response, meta)`
169+
- `extractGeminiUsage(response, meta)`
170+
171+
They map common token fields into the UsageTap `call_end` payload shape:
172+
173+
- OpenAI and OpenRouter: `usage.prompt_tokens`, `completion_tokens`, cached prompt tokens, reasoning tokens
174+
- Anthropic: `usage.input_tokens`, `output_tokens`, `cache_read_input_tokens`
175+
- Gemini: `usageMetadata.promptTokenCount`, `candidatesTokenCount`, `cachedContentTokenCount`, `thoughtsTokenCount`
176+
177+
## API surface
178+
179+
Main exports:
180+
181+
- `createUsageTapClient`
182+
- `beginUsageTapCall`
183+
- `endUsageTapCall`
184+
- `withUsageTapCall`
185+
- `applyUsageTapEntitlements`
186+
- `runOpenAIWithUsageTap`
187+
- `runOpenRouterWithUsageTap`
188+
- `runAnthropicWithUsageTap`
189+
- `runGeminiWithUsageTap`
190+
- `extractOpenAIUsage`
191+
- `extractAnthropicUsage`
192+
- `extractGeminiUsage`
193+
- `defaultUsageTapErrorMapper`
194+
195+
Relevant types:
196+
197+
- `UsageTapBeginRequest`
198+
- `UsageTapBeginResponse`
199+
- `UsageTapEndUsage`
200+
- `UsageTapCallOptions`
201+
- `UsageTapProviderRunOptions`
202+
- `UsageTapEntitlementOptions`
203+
- `UsageTapAllowed`
204+
205+
For the rest of the PromptOpsKit provider model, see [Providers](./providers.md).

package.json

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,16 @@
3030
"default": "./dist/testing.cjs"
3131
}
3232
},
33+
"./usagetap": {
34+
"import": {
35+
"types": "./dist/usagetap/index.d.ts",
36+
"default": "./dist/usagetap/index.js"
37+
},
38+
"require": {
39+
"types": "./dist/usagetap/index.d.cts",
40+
"default": "./dist/usagetap/index.cjs"
41+
}
42+
},
3343
"./openai": {
3444
"import": {
3545
"types": "./dist/providers/openai.d.ts",

src/index.ts

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,27 @@ export type { PromptValidationResult, ValidationError } from './validation/index
2121
export type { RenderedSections, RenderOptions } from './renderer/index.js';
2222
export type { ParseResult } from './parser/index.js';
2323
export type { OverrideOptions } from './overrides/index.js';
24+
export type {
25+
UsageTapAllowed,
26+
UsageTapAllowedCapability,
27+
UsageTapBeginRequest,
28+
UsageTapBeginResponse,
29+
UsageTapCallOptions,
30+
UsageTapCallResult,
31+
UsageTapClient,
32+
UsageTapClientConfig,
33+
UsageTapEndRequest,
34+
UsageTapEndResponse,
35+
UsageTapEndUsage,
36+
UsageTapEntitlementMode,
37+
UsageTapEntitlementOptions,
38+
UsageTapErrorPayload,
39+
UsageTapInvokeContext,
40+
UsageTapInvokeResult,
41+
UsageTapProviderRunOptions,
42+
UsageTapProviderRunResult,
43+
UsageTapReasoningLevel,
44+
} from './usagetap/index.js';
2445

2546
export { parsePrompt, loadPromptFile, extractSections } from './parser/index.js';
2647
export { interpolate, extractVariables } from './renderer/index.js';
@@ -32,6 +53,21 @@ export { anthropicAdapter } from './providers/anthropic.js';
3253
export { geminiAdapter } from './providers/gemini.js';
3354
export { openrouterAdapter } from './providers/openrouter.js';
3455
export { PromptAssetSchema, PromptAssetOverridesSchema } from './schema/index.js';
56+
export {
57+
applyUsageTapEntitlements,
58+
beginUsageTapCall,
59+
createUsageTapClient,
60+
defaultUsageTapErrorMapper,
61+
endUsageTapCall,
62+
extractAnthropicUsage,
63+
extractGeminiUsage,
64+
extractOpenAIUsage,
65+
runAnthropicWithUsageTap,
66+
runGeminiWithUsageTap,
67+
runOpenAIWithUsageTap,
68+
runOpenRouterWithUsageTap,
69+
withUsageTapCall,
70+
} from './usagetap/index.js';
3571

3672
// --- Config ---
3773

0 commit comments

Comments
 (0)