Skip to content

Commit 3121a7b

Browse files
pescnclaude
andauthored
fix: handle streaming usage chunk and soft-deleted provider filtering (#76)
* fix(adapters): handle separate usage chunk in OpenAI streaming responses Some OpenAI-compatible providers send token usage in a separate chunk (with choices=[]) after the finish_reason chunk, rather than bundling them together. Previously, this usage-only chunk was silently skipped because data.choices[0] was undefined, causing token counts to always be recorded as -1 for streaming requests. This broke TPM rate limiting and usage tracking. Now both patterns are supported: - Usage bundled with finish_reason in the same chunk - Usage in a separate subsequent chunk (choices=[]) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(db): filter soft-deleted providers in listUniqueSystemNames listUniqueSystemNames() only filtered NOT models.deleted but did not join the providers table or check NOT providers.deleted. When a provider was soft-deleted, its models (still deleted=false) would appear in the global model registry, but getModelsWithProviderBySystemName() correctly filtered them out, causing the UI to show models with no providers. Add innerJoin on ProvidersTable and NOT providers.deleted filter to match the behavior of getModelsWithProviderBySystemName(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(adapters): omit messageDelta in usage-only chunk to avoid overwriting stopReason Remove explicit stopReason: null from the usage-only message_delta yield to prevent overwriting a previously set stopReason. This aligns with the pattern used in openai-responses.ts for usage-only events. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 3d4016f commit 3121a7b

2 files changed

Lines changed: 27 additions & 10 deletions

File tree

backend/src/adapters/upstream/openai.ts

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,7 @@ import type {
1111
InternalResponse,
1212
InternalStreamChunk,
1313
InternalToolDefinition,
14-
InternalUsage,
15-
ProviderConfig,
14+
ProviderConfig,
1615
StopReason,
1716
TextContentBlock,
1817
ThinkingContentBlock,
@@ -387,7 +386,7 @@ export const openaiUpstreamAdapter: UpstreamAdapter = {
387386

388387
buildRequest(
389388
request: InternalRequest,
390-
provider: ProviderConfig,
389+
provider: ProviderConfig,
391390
): { url: string; init: RequestInit } {
392391
// Build messages array with system prompt
393392
const messages: OpenAIMessage[] = [];
@@ -510,6 +509,17 @@ export const openaiUpstreamAdapter: UpstreamAdapter = {
510509

511510
const choice = data.choices[0];
512511
if (!choice) {
512+
// Usage-only chunk (choices=[]) — some providers send usage separately
513+
// after the finish_reason chunk when stream_options.include_usage=true
514+
if (data.usage) {
515+
yield {
516+
type: "message_delta",
517+
usage: {
518+
inputTokens: data.usage.prompt_tokens,
519+
outputTokens: data.usage.completion_tokens,
520+
},
521+
};
522+
}
513523
continue;
514524
}
515525

@@ -581,18 +591,20 @@ export const openaiUpstreamAdapter: UpstreamAdapter = {
581591
// Handle finish reason
582592
if (choice.finish_reason) {
583593
yield { type: "content_block_stop", index: blockIndex };
584-
const usage: InternalUsage = data.usage
585-
? {
586-
inputTokens: data.usage.prompt_tokens,
587-
outputTokens: data.usage.completion_tokens,
588-
}
589-
: { inputTokens: -1, outputTokens: -1 };
594+
// Include usage only if upstream provided it in this chunk.
595+
// Some providers bundle usage with finish_reason; others send a
596+
// separate usage-only chunk (choices=[]) immediately after.
590597
yield {
591598
type: "message_delta",
592599
messageDelta: {
593600
stopReason: convertFinishReason(choice.finish_reason),
594601
},
595-
usage,
602+
...(data.usage && {
603+
usage: {
604+
inputTokens: data.usage.prompt_tokens,
605+
outputTokens: data.usage.completion_tokens,
606+
},
607+
}),
596608
};
597609
}
598610
}

backend/src/db/index.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -955,9 +955,14 @@ export async function listUniqueSystemNames(
955955
const r = await db
956956
.selectDistinct({ systemName: schema.ModelsTable.systemName })
957957
.from(schema.ModelsTable)
958+
.innerJoin(
959+
schema.ProvidersTable,
960+
eq(schema.ModelsTable.providerId, schema.ProvidersTable.id),
961+
)
958962
.where(
959963
and(
960964
not(schema.ModelsTable.deleted),
965+
not(schema.ProvidersTable.deleted),
961966
modelType ? eq(schema.ModelsTable.modelType, modelType) : undefined,
962967
),
963968
)

0 commit comments

Comments
 (0)