Skip to content

Commit eb607da

Browse files
api-clients-generation-pipeline[bot]ci.datadog-api-spec
api-clients-generation-pipeline[bot]
and
ci.datadog-api-spec
authored
Add Annotations endpoints (#3881)
Co-authored-by: ci.datadog-api-spec <packages@datadoghq.com>
1 parent 095d494 commit eb607da

File tree

38 files changed

+4081
-90
lines changed

38 files changed

+4081
-90
lines changed

‎.generator/schemas/v2/openapi.yaml‎

Lines changed: 786 additions & 1 deletion
Large diffs are not rendered by default.

‎features/v2/llm_observability.feature‎

Lines changed: 153 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,40 @@
11
@endpoint(llm-observability) @endpoint(llm-observability-v2)
22
Feature: LLM Observability
3-
Manage LLM Observability projects, datasets, dataset records, and
4-
experiments via the Experiments API.
3+
Manage LLM Observability projects, datasets, dataset records, experiments,
4+
and annotations.
55

66
Background:
77
Given a valid "apiKeyAuth" key in the system
88
And a valid "appKeyAuth" key in the system
99
And an instance of "LLMObservability" API
1010

11+
@generated @skip @team:DataDog/ml-observability
12+
Scenario: Add annotation queue interactions returns "Bad Request" response
13+
Given operation "CreateLLMObsAnnotationQueueInteractions" enabled
14+
And new "CreateLLMObsAnnotationQueueInteractions" request
15+
And request contains "queue_id" parameter from "REPLACE.ME"
16+
And body with value {"data": {"attributes": {"interactions": [{"content_id": "trace-abc-123", "type": "trace"}]}, "type": "interactions"}}
17+
When the request is sent
18+
Then the response status is 400 Bad Request
19+
20+
@generated @skip @team:DataDog/ml-observability
21+
Scenario: Add annotation queue interactions returns "Created" response
22+
Given operation "CreateLLMObsAnnotationQueueInteractions" enabled
23+
And new "CreateLLMObsAnnotationQueueInteractions" request
24+
And request contains "queue_id" parameter from "REPLACE.ME"
25+
And body with value {"data": {"attributes": {"interactions": [{"content_id": "trace-abc-123", "type": "trace"}]}, "type": "interactions"}}
26+
When the request is sent
27+
Then the response status is 201 Created
28+
29+
@generated @skip @team:DataDog/ml-observability
30+
Scenario: Add annotation queue interactions returns "Not Found" response
31+
Given operation "CreateLLMObsAnnotationQueueInteractions" enabled
32+
And new "CreateLLMObsAnnotationQueueInteractions" request
33+
And request contains "queue_id" parameter from "REPLACE.ME"
34+
And body with value {"data": {"attributes": {"interactions": [{"content_id": "trace-abc-123", "type": "trace"}]}, "type": "interactions"}}
35+
When the request is sent
36+
Then the response status is 404 Not Found
37+
1138
@generated @skip @team:DataDog/ml-observability
1239
Scenario: Append records to an LLM Observability dataset returns "Bad Request" response
1340
Given operation "CreateLLMObsDatasetRecords" enabled
@@ -48,6 +75,22 @@ Feature: LLM Observability
4875
When the request is sent
4976
Then the response status is 200 OK
5077

78+
@generated @skip @team:DataDog/ml-observability
79+
Scenario: Create an LLM Observability annotation queue returns "Bad Request" response
80+
Given operation "CreateLLMObsAnnotationQueue" enabled
81+
And new "CreateLLMObsAnnotationQueue" request
82+
And body with value {"data": {"attributes": {"description": "Queue for annotating customer support traces", "name": "My annotation queue", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751"}, "type": "queues"}}
83+
When the request is sent
84+
Then the response status is 400 Bad Request
85+
86+
@generated @skip @team:DataDog/ml-observability
87+
Scenario: Create an LLM Observability annotation queue returns "Created" response
88+
Given operation "CreateLLMObsAnnotationQueue" enabled
89+
And new "CreateLLMObsAnnotationQueue" request
90+
And body with value {"data": {"attributes": {"description": "Queue for annotating customer support traces", "name": "My annotation queue", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751"}, "type": "queues"}}
91+
When the request is sent
92+
Then the response status is 201 Created
93+
5194
@generated @skip @team:DataDog/ml-observability
5295
Scenario: Create an LLM Observability dataset returns "Bad Request" response
5396
Given operation "CreateLLMObsDataset" enabled
@@ -221,6 +264,87 @@ Feature: LLM Observability
221264
When the request is sent
222265
Then the response status is 204 No Content
223266

267+
@generated @skip @team:DataDog/ml-observability
268+
Scenario: Delete an LLM Observability annotation queue returns "No Content" response
269+
Given operation "DeleteLLMObsAnnotationQueue" enabled
270+
And new "DeleteLLMObsAnnotationQueue" request
271+
And request contains "queue_id" parameter from "REPLACE.ME"
272+
When the request is sent
273+
Then the response status is 204 No Content
274+
275+
@generated @skip @team:DataDog/ml-observability
276+
Scenario: Delete an LLM Observability annotation queue returns "Not Found" response
277+
Given operation "DeleteLLMObsAnnotationQueue" enabled
278+
And new "DeleteLLMObsAnnotationQueue" request
279+
And request contains "queue_id" parameter from "REPLACE.ME"
280+
When the request is sent
281+
Then the response status is 404 Not Found
282+
283+
@generated @skip @team:DataDog/ml-observability
284+
Scenario: Delete annotation queue interactions returns "Bad Request" response
285+
Given operation "DeleteLLMObsAnnotationQueueInteractions" enabled
286+
And new "DeleteLLMObsAnnotationQueueInteractions" request
287+
And request contains "queue_id" parameter from "REPLACE.ME"
288+
And body with value {"data": {"attributes": {"interaction_ids": ["00000000-0000-0000-0000-000000000000", "00000000-0000-0000-0000-000000000001"]}, "type": "interactions"}}
289+
When the request is sent
290+
Then the response status is 400 Bad Request
291+
292+
@generated @skip @team:DataDog/ml-observability
293+
Scenario: Delete annotation queue interactions returns "No Content" response
294+
Given operation "DeleteLLMObsAnnotationQueueInteractions" enabled
295+
And new "DeleteLLMObsAnnotationQueueInteractions" request
296+
And request contains "queue_id" parameter from "REPLACE.ME"
297+
And body with value {"data": {"attributes": {"interaction_ids": ["00000000-0000-0000-0000-000000000000", "00000000-0000-0000-0000-000000000001"]}, "type": "interactions"}}
298+
When the request is sent
299+
Then the response status is 204 No Content
300+
301+
@generated @skip @team:DataDog/ml-observability
302+
Scenario: Delete annotation queue interactions returns "Not Found" response
303+
Given operation "DeleteLLMObsAnnotationQueueInteractions" enabled
304+
And new "DeleteLLMObsAnnotationQueueInteractions" request
305+
And request contains "queue_id" parameter from "REPLACE.ME"
306+
And body with value {"data": {"attributes": {"interaction_ids": ["00000000-0000-0000-0000-000000000000", "00000000-0000-0000-0000-000000000001"]}, "type": "interactions"}}
307+
When the request is sent
308+
Then the response status is 404 Not Found
309+
310+
@generated @skip @team:DataDog/ml-observability
311+
Scenario: Get annotated queue interactions returns "Bad Request" response
312+
Given operation "GetLLMObsAnnotatedInteractions" enabled
313+
And new "GetLLMObsAnnotatedInteractions" request
314+
And request contains "queue_id" parameter from "REPLACE.ME"
315+
When the request is sent
316+
Then the response status is 400 Bad Request
317+
318+
@generated @skip @team:DataDog/ml-observability
319+
Scenario: Get annotated queue interactions returns "Not Found" response
320+
Given operation "GetLLMObsAnnotatedInteractions" enabled
321+
And new "GetLLMObsAnnotatedInteractions" request
322+
And request contains "queue_id" parameter from "REPLACE.ME"
323+
When the request is sent
324+
Then the response status is 404 Not Found
325+
326+
@generated @skip @team:DataDog/ml-observability
327+
Scenario: Get annotated queue interactions returns "OK" response
328+
Given operation "GetLLMObsAnnotatedInteractions" enabled
329+
And new "GetLLMObsAnnotatedInteractions" request
330+
And request contains "queue_id" parameter from "REPLACE.ME"
331+
When the request is sent
332+
Then the response status is 200 OK
333+
334+
@generated @skip @team:DataDog/ml-observability
335+
Scenario: List LLM Observability annotation queues returns "Bad Request" response
336+
Given operation "ListLLMObsAnnotationQueues" enabled
337+
And new "ListLLMObsAnnotationQueues" request
338+
When the request is sent
339+
Then the response status is 400 Bad Request
340+
341+
@generated @skip @team:DataDog/ml-observability
342+
Scenario: List LLM Observability annotation queues returns "OK" response
343+
Given operation "ListLLMObsAnnotationQueues" enabled
344+
And new "ListLLMObsAnnotationQueues" request
345+
When the request is sent
346+
Then the response status is 200 OK
347+
224348
@generated @skip @team:DataDog/ml-observability
225349
Scenario: List LLM Observability dataset records returns "Bad Request" response
226350
Given operation "ListLLMObsDatasetRecords" enabled
@@ -357,6 +481,33 @@ Feature: LLM Observability
357481
When the request is sent
358482
Then the response status is 200 OK
359483

484+
@generated @skip @team:DataDog/ml-observability
485+
Scenario: Update an LLM Observability annotation queue returns "Bad Request" response
486+
Given operation "UpdateLLMObsAnnotationQueue" enabled
487+
And new "UpdateLLMObsAnnotationQueue" request
488+
And request contains "queue_id" parameter from "REPLACE.ME"
489+
And body with value {"data": {"attributes": {"description": "Updated description", "name": "Updated queue name"}, "type": "queues"}}
490+
When the request is sent
491+
Then the response status is 400 Bad Request
492+
493+
@generated @skip @team:DataDog/ml-observability
494+
Scenario: Update an LLM Observability annotation queue returns "Not Found" response
495+
Given operation "UpdateLLMObsAnnotationQueue" enabled
496+
And new "UpdateLLMObsAnnotationQueue" request
497+
And request contains "queue_id" parameter from "REPLACE.ME"
498+
And body with value {"data": {"attributes": {"description": "Updated description", "name": "Updated queue name"}, "type": "queues"}}
499+
When the request is sent
500+
Then the response status is 404 Not Found
501+
502+
@generated @skip @team:DataDog/ml-observability
503+
Scenario: Update an LLM Observability annotation queue returns "OK" response
504+
Given operation "UpdateLLMObsAnnotationQueue" enabled
505+
And new "UpdateLLMObsAnnotationQueue" request
506+
And request contains "queue_id" parameter from "REPLACE.ME"
507+
And body with value {"data": {"attributes": {"description": "Updated description", "name": "Updated queue name"}, "type": "queues"}}
508+
When the request is sent
509+
Then the response status is 200 OK
510+
360511
@generated @skip @team:DataDog/ml-observability
361512
Scenario: Update an LLM Observability dataset returns "Bad Request" response
362513
Given operation "UpdateLLMObsDataset" enabled

‎features/v2/undo.json‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3087,6 +3087,66 @@
30873087
"type": "idempotent"
30883088
}
30893089
},
3090+
"ListLLMObsAnnotationQueues": {
3091+
"tag": "LLM Observability",
3092+
"undo": {
3093+
"type": "safe"
3094+
}
3095+
},
3096+
"CreateLLMObsAnnotationQueue": {
3097+
"tag": "LLM Observability",
3098+
"undo": {
3099+
"operationId": "DeleteLLMObsAnnotationQueue",
3100+
"parameters": [
3101+
{
3102+
"name": "queue_id",
3103+
"source": "data.id"
3104+
}
3105+
],
3106+
"type": "unsafe"
3107+
}
3108+
},
3109+
"DeleteLLMObsAnnotationQueue": {
3110+
"tag": "LLM Observability",
3111+
"undo": {
3112+
"type": "idempotent"
3113+
}
3114+
},
3115+
"UpdateLLMObsAnnotationQueue": {
3116+
"tag": "LLM Observability",
3117+
"undo": {
3118+
"type": "idempotent"
3119+
}
3120+
},
3121+
"GetLLMObsAnnotatedInteractions": {
3122+
"tag": "LLM Observability",
3123+
"undo": {
3124+
"type": "safe"
3125+
}
3126+
},
3127+
"CreateLLMObsAnnotationQueueInteractions": {
3128+
"tag": "LLM Observability",
3129+
"undo": {
3130+
"operationId": "DeleteLLMObsAnnotationQueueInteractions",
3131+
"parameters": [
3132+
{
3133+
"name": "queue_id",
3134+
"source": "path.queue_id"
3135+
},
3136+
{
3137+
"name": "body",
3138+
"template": "{\"data\": {\"type\": \"interactions\", \"attributes\": {\"interaction_ids\": [\"{{ data.attributes.interactions[0].id }}\"]}}}"
3139+
}
3140+
],
3141+
"type": "unsafe"
3142+
}
3143+
},
3144+
"DeleteLLMObsAnnotationQueueInteractions": {
3145+
"tag": "LLM Observability",
3146+
"undo": {
3147+
"type": "idempotent"
3148+
}
3149+
},
30903150
"ListLLMObsExperiments": {
30913151
"tag": "LLM Observability",
30923152
"undo": {

‎private/bdd_runner/src/support/scenarios_model_mapping.ts‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7769,6 +7769,71 @@ export const ScenariosModelMappings: { [key: string]: OperationMapping } = {
77697769
},
77707770
operationResponseType: "IPAllowlistResponse",
77717771
},
7772+
"LLMObservabilityApi.V2.ListLLMObsAnnotationQueues": {
7773+
projectId: {
7774+
type: "string",
7775+
format: "",
7776+
},
7777+
queueIds: {
7778+
type: "Array<string>",
7779+
format: "",
7780+
},
7781+
operationResponseType: "LLMObsAnnotationQueuesResponse",
7782+
},
7783+
"LLMObservabilityApi.V2.CreateLLMObsAnnotationQueue": {
7784+
body: {
7785+
type: "LLMObsAnnotationQueueRequest",
7786+
format: "",
7787+
},
7788+
operationResponseType: "LLMObsAnnotationQueueResponse",
7789+
},
7790+
"LLMObservabilityApi.V2.DeleteLLMObsAnnotationQueue": {
7791+
queueId: {
7792+
type: "string",
7793+
format: "",
7794+
},
7795+
operationResponseType: "{}",
7796+
},
7797+
"LLMObservabilityApi.V2.UpdateLLMObsAnnotationQueue": {
7798+
queueId: {
7799+
type: "string",
7800+
format: "",
7801+
},
7802+
body: {
7803+
type: "LLMObsAnnotationQueueUpdateRequest",
7804+
format: "",
7805+
},
7806+
operationResponseType: "LLMObsAnnotationQueueResponse",
7807+
},
7808+
"LLMObservabilityApi.V2.GetLLMObsAnnotatedInteractions": {
7809+
queueId: {
7810+
type: "string",
7811+
format: "",
7812+
},
7813+
operationResponseType: "LLMObsAnnotatedInteractionsResponse",
7814+
},
7815+
"LLMObservabilityApi.V2.CreateLLMObsAnnotationQueueInteractions": {
7816+
queueId: {
7817+
type: "string",
7818+
format: "",
7819+
},
7820+
body: {
7821+
type: "LLMObsAnnotationQueueInteractionsRequest",
7822+
format: "",
7823+
},
7824+
operationResponseType: "LLMObsAnnotationQueueInteractionsResponse",
7825+
},
7826+
"LLMObservabilityApi.V2.DeleteLLMObsAnnotationQueueInteractions": {
7827+
queueId: {
7828+
type: "string",
7829+
format: "",
7830+
},
7831+
body: {
7832+
type: "LLMObsDeleteAnnotationQueueInteractionsRequest",
7833+
format: "",
7834+
},
7835+
operationResponseType: "{}",
7836+
},
77727837
"LLMObservabilityApi.V2.ListLLMObsExperiments": {
77737838
filterProjectId: {
77747839
type: "string",

‎services/llm_observability/README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
## Description
44

5-
Manage LLM Observability projects, datasets, dataset records, and experiments via the Experiments API.
5+
Manage LLM Observability projects, datasets, dataset records, experiments, and annotations.
66

77
## Navigation
88

@@ -28,15 +28,15 @@ const configuration = createConfiguration();
2828
// Enable unstable operations
2929
const configurationOpts = {
3030
unstableOperations: {
31-
"LLMObservabilityApi.v2.listLLMObsExperiments": true
31+
"LLMObservabilityApi.v2.listLLMObsAnnotationQueues": true
3232
}
3333
}
3434

3535
const configuration = createConfiguration(configurationOpts);
3636
const apiInstance = new LLMObservabilityApiV2(configuration);
3737
const params = {/* parameters */};
3838

39-
apiInstance.listLLMObsExperiments(params).then((data) => {
39+
apiInstance.listLLMObsAnnotationQueues(params).then((data) => {
4040
console.log("API called successfully. Returned data: " + JSON.stringify(data));
4141
}).catch((error) => {
4242
console.error("Error calling API: " + error);

0 commit comments

Comments
 (0)