Skip to content

Commit 9c51a22

Browse files
shmsrclaudestefans-elastic
authored
[openai] Fix Usage Overview dashboard panels erroring when only a subset of collections is enabled (#19345)
The OpenAI Usage Overview dashboard aggregates usage fields across several metric data streams (token totals over completions/embeddings/moderations, plus the images and audio sections). Those panels reference fields from data streams that may not be enabled. When a stream is disabled it has no index, so its fields are absent from the logs-* data view and Lens hard-errors the whole panel ("missing field") instead of just showing the available data. Declare the 11 dashboard-referenced usage fields in every metric data stream's fields.yml (each stream's own fields plus its siblings'), so field_caps always resolves them whenever any one collection is enabled. The sibling fields are mapped but only populated by their owning stream. This is non-breaking: no field renames, the dashboard and ingest pipelines are unchanged, so historical data keeps rendering and existing queries/alerts keep working. Verified on a live stack with completions-only data: before, field_caps resolved 5/11 fields (panels error); after, 11/11 resolve (panels render). Fixes #19340 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: stefans-elastic <stefan.stas@elastic.co>
1 parent 81060d3 commit 9c51a22

9 files changed

Lines changed: 455 additions & 87 deletions

File tree

‎packages/openai/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "2.3.0"
3+
changes:
4+
- description: Map usage fields across the metric data streams (completions, embeddings, moderations, images, audio_transcriptions and audio_speeches) so the OpenAI Usage Overview dashboard panels render correctly when only a subset of the usage collections is enabled.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/19345
27
- version: "2.2.0"
38
changes:
49
- description: Add rate_limits data stream, a rate limit headroom dashboard panel, and a prebuilt rate limit headroom alert.

‎packages/openai/data_stream/audio_speeches/fields/fields.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,3 +39,57 @@
3939
- name: characters
4040
type: long
4141
description: Number of characters processed
42+
# Token/usage fields from sibling data streams are mapped (but not populated) here so the
43+
# shared OpenAI Usage Overview dashboard can aggregate usage across collections even when
44+
# only a subset of them is enabled. See https://github.com/elastic/integrations/issues/19340
45+
- name: completions
46+
type: group
47+
description: OpenAI completions usage metrics and metadata
48+
fields:
49+
- name: output_audio_tokens
50+
type: long
51+
description: Number of audio output tokens used
52+
- name: input_audio_tokens
53+
type: long
54+
description: Number of audio input tokens used, including cached tokens
55+
- name: input_cached_tokens
56+
type: long
57+
description: Number of text input tokens that has been cached from previous requests. For customers subscribe to scale tier, this includes scale tier tokens
58+
- name: input_tokens
59+
type: long
60+
description: Number of text input tokens used, including cached tokens. For customers subscribe to scale tier, this includes scale tier tokens
61+
- name: output_tokens
62+
type: long
63+
description: Number of text output tokens used. For customers subscribe to scale tier, this includes scale tier tokens
64+
- name: embeddings
65+
type: group
66+
description: OpenAI embeddings usage metrics and metadata
67+
fields:
68+
- name: input_tokens
69+
type: long
70+
description: Number of input tokens used.
71+
- name: moderations
72+
type: group
73+
description: OpenAI moderations usage metrics and metadata
74+
fields:
75+
- name: input_tokens
76+
type: long
77+
description: Number of input tokens used.
78+
- name: images
79+
type: group
80+
description: OpenAI images usage metrics and metadata
81+
fields:
82+
- name: images
83+
type: long
84+
description: Number of images processed
85+
- name: size
86+
type: keyword
87+
description: Image size (dimension of the generated image)
88+
- name: audio_transcriptions
89+
type: group
90+
description: OpenAI audio transcriptions usage metrics and metadata
91+
fields:
92+
- name: seconds
93+
type: long
94+
unit: s
95+
description: Number of seconds processed

‎packages/openai/data_stream/audio_transcriptions/fields/fields.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,3 +40,56 @@
4040
type: long
4141
unit: s
4242
description: Number of seconds processed
43+
# Token/usage fields from sibling data streams are mapped (but not populated) here so the
44+
# shared OpenAI Usage Overview dashboard can aggregate usage across collections even when
45+
# only a subset of them is enabled. See https://github.com/elastic/integrations/issues/19340
46+
- name: completions
47+
type: group
48+
description: OpenAI completions usage metrics and metadata
49+
fields:
50+
- name: output_audio_tokens
51+
type: long
52+
description: Number of audio output tokens used
53+
- name: input_audio_tokens
54+
type: long
55+
description: Number of audio input tokens used, including cached tokens
56+
- name: input_cached_tokens
57+
type: long
58+
description: Number of text input tokens that has been cached from previous requests. For customers subscribe to scale tier, this includes scale tier tokens
59+
- name: input_tokens
60+
type: long
61+
description: Number of text input tokens used, including cached tokens. For customers subscribe to scale tier, this includes scale tier tokens
62+
- name: output_tokens
63+
type: long
64+
description: Number of text output tokens used. For customers subscribe to scale tier, this includes scale tier tokens
65+
- name: embeddings
66+
type: group
67+
description: OpenAI embeddings usage metrics and metadata
68+
fields:
69+
- name: input_tokens
70+
type: long
71+
description: Number of input tokens used.
72+
- name: moderations
73+
type: group
74+
description: OpenAI moderations usage metrics and metadata
75+
fields:
76+
- name: input_tokens
77+
type: long
78+
description: Number of input tokens used.
79+
- name: images
80+
type: group
81+
description: OpenAI images usage metrics and metadata
82+
fields:
83+
- name: images
84+
type: long
85+
description: Number of images processed
86+
- name: size
87+
type: keyword
88+
description: Image size (dimension of the generated image)
89+
- name: audio_speeches
90+
type: group
91+
description: OpenAI audio speeches usage metrics and metadata
92+
fields:
93+
- name: characters
94+
type: long
95+
description: Number of characters processed

‎packages/openai/data_stream/completions/fields/fields.yml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,3 +57,45 @@
5757
- name: output_tokens
5858
type: long
5959
description: Number of text output tokens used. For customers subscribe to scale tier, this includes scale tier tokens
60+
# Token/usage fields from sibling data streams are mapped (but not populated) here so the
61+
# shared OpenAI Usage Overview dashboard can aggregate usage across collections even when
62+
# only a subset of them is enabled. See https://github.com/elastic/integrations/issues/19340
63+
- name: embeddings
64+
type: group
65+
description: OpenAI embeddings usage metrics and metadata
66+
fields:
67+
- name: input_tokens
68+
type: long
69+
description: Number of input tokens used.
70+
- name: moderations
71+
type: group
72+
description: OpenAI moderations usage metrics and metadata
73+
fields:
74+
- name: input_tokens
75+
type: long
76+
description: Number of input tokens used.
77+
- name: images
78+
type: group
79+
description: OpenAI images usage metrics and metadata
80+
fields:
81+
- name: images
82+
type: long
83+
description: Number of images processed
84+
- name: size
85+
type: keyword
86+
description: Image size (dimension of the generated image)
87+
- name: audio_transcriptions
88+
type: group
89+
description: OpenAI audio transcriptions usage metrics and metadata
90+
fields:
91+
- name: seconds
92+
type: long
93+
unit: s
94+
description: Number of seconds processed
95+
- name: audio_speeches
96+
type: group
97+
description: OpenAI audio speeches usage metrics and metadata
98+
fields:
99+
- name: characters
100+
type: long
101+
description: Number of characters processed

‎packages/openai/data_stream/embeddings/fields/fields.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,3 +42,57 @@
4242
- name: input_tokens
4343
type: long
4444
description: Number of input tokens used.
45+
# Token/usage fields from sibling data streams are mapped (but not populated) here so the
46+
# shared OpenAI Usage Overview dashboard can aggregate usage across collections even when
47+
# only a subset of them is enabled. See https://github.com/elastic/integrations/issues/19340
48+
- name: completions
49+
type: group
50+
description: OpenAI completions usage metrics and metadata
51+
fields:
52+
- name: output_audio_tokens
53+
type: long
54+
description: Number of audio output tokens used
55+
- name: input_audio_tokens
56+
type: long
57+
description: Number of audio input tokens used, including cached tokens
58+
- name: input_cached_tokens
59+
type: long
60+
description: Number of text input tokens that has been cached from previous requests. For customers subscribe to scale tier, this includes scale tier tokens
61+
- name: input_tokens
62+
type: long
63+
description: Number of text input tokens used, including cached tokens. For customers subscribe to scale tier, this includes scale tier tokens
64+
- name: output_tokens
65+
type: long
66+
description: Number of text output tokens used. For customers subscribe to scale tier, this includes scale tier tokens
67+
- name: moderations
68+
type: group
69+
description: OpenAI moderations usage metrics and metadata
70+
fields:
71+
- name: input_tokens
72+
type: long
73+
description: Number of input tokens used.
74+
- name: images
75+
type: group
76+
description: OpenAI images usage metrics and metadata
77+
fields:
78+
- name: images
79+
type: long
80+
description: Number of images processed
81+
- name: size
82+
type: keyword
83+
description: Image size (dimension of the generated image)
84+
- name: audio_transcriptions
85+
type: group
86+
description: OpenAI audio transcriptions usage metrics and metadata
87+
fields:
88+
- name: seconds
89+
type: long
90+
unit: s
91+
description: Number of seconds processed
92+
- name: audio_speeches
93+
type: group
94+
description: OpenAI audio speeches usage metrics and metadata
95+
fields:
96+
- name: characters
97+
type: long
98+
description: Number of characters processed

‎packages/openai/data_stream/images/fields/fields.yml‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,54 @@
4848
- name: size
4949
type: keyword
5050
description: Image size (dimension of the generated image)
51+
# Token/usage fields from sibling data streams are mapped (but not populated) here so the
52+
# shared OpenAI Usage Overview dashboard can aggregate usage across collections even when
53+
# only a subset of them is enabled. See https://github.com/elastic/integrations/issues/19340
54+
- name: completions
55+
type: group
56+
description: OpenAI completions usage metrics and metadata
57+
fields:
58+
- name: output_audio_tokens
59+
type: long
60+
description: Number of audio output tokens used
61+
- name: input_audio_tokens
62+
type: long
63+
description: Number of audio input tokens used, including cached tokens
64+
- name: input_cached_tokens
65+
type: long
66+
description: Number of text input tokens that has been cached from previous requests. For customers subscribe to scale tier, this includes scale tier tokens
67+
- name: input_tokens
68+
type: long
69+
description: Number of text input tokens used, including cached tokens. For customers subscribe to scale tier, this includes scale tier tokens
70+
- name: output_tokens
71+
type: long
72+
description: Number of text output tokens used. For customers subscribe to scale tier, this includes scale tier tokens
73+
- name: embeddings
74+
type: group
75+
description: OpenAI embeddings usage metrics and metadata
76+
fields:
77+
- name: input_tokens
78+
type: long
79+
description: Number of input tokens used.
80+
- name: moderations
81+
type: group
82+
description: OpenAI moderations usage metrics and metadata
83+
fields:
84+
- name: input_tokens
85+
type: long
86+
description: Number of input tokens used.
87+
- name: audio_transcriptions
88+
type: group
89+
description: OpenAI audio transcriptions usage metrics and metadata
90+
fields:
91+
- name: seconds
92+
type: long
93+
unit: s
94+
description: Number of seconds processed
95+
- name: audio_speeches
96+
type: group
97+
description: OpenAI audio speeches usage metrics and metadata
98+
fields:
99+
- name: characters
100+
type: long
101+
description: Number of characters processed

‎packages/openai/data_stream/moderations/fields/fields.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,3 +42,57 @@
4242
- name: input_tokens
4343
type: long
4444
description: Number of input tokens used.
45+
# Token/usage fields from sibling data streams are mapped (but not populated) here so the
46+
# shared OpenAI Usage Overview dashboard can aggregate usage across collections even when
47+
# only a subset of them is enabled. See https://github.com/elastic/integrations/issues/19340
48+
- name: completions
49+
type: group
50+
description: OpenAI completions usage metrics and metadata
51+
fields:
52+
- name: output_audio_tokens
53+
type: long
54+
description: Number of audio output tokens used
55+
- name: input_audio_tokens
56+
type: long
57+
description: Number of audio input tokens used, including cached tokens
58+
- name: input_cached_tokens
59+
type: long
60+
description: Number of text input tokens that has been cached from previous requests. For customers subscribe to scale tier, this includes scale tier tokens
61+
- name: input_tokens
62+
type: long
63+
description: Number of text input tokens used, including cached tokens. For customers subscribe to scale tier, this includes scale tier tokens
64+
- name: output_tokens
65+
type: long
66+
description: Number of text output tokens used. For customers subscribe to scale tier, this includes scale tier tokens
67+
- name: embeddings
68+
type: group
69+
description: OpenAI embeddings usage metrics and metadata
70+
fields:
71+
- name: input_tokens
72+
type: long
73+
description: Number of input tokens used.
74+
- name: images
75+
type: group
76+
description: OpenAI images usage metrics and metadata
77+
fields:
78+
- name: images
79+
type: long
80+
description: Number of images processed
81+
- name: size
82+
type: keyword
83+
description: Image size (dimension of the generated image)
84+
- name: audio_transcriptions
85+
type: group
86+
description: OpenAI audio transcriptions usage metrics and metadata
87+
fields:
88+
- name: seconds
89+
type: long
90+
unit: s
91+
description: Number of seconds processed
92+
- name: audio_speeches
93+
type: group
94+
description: OpenAI audio speeches usage metrics and metadata
95+
fields:
96+
- name: characters
97+
type: long
98+
description: Number of characters processed

0 commit comments

Comments
 (0)