Skip to content

Commit 0733181

Browse files
api-clients-generation-pipeline[bot]ci.datadog-api-spec
andauthored
Add Obs Pipelines Elasticsearch metrics destination and new fields (#3385)
Co-authored-by: ci.datadog-api-spec <packages@datadoghq.com>
1 parent bd26647 commit 0733181

9 files changed

+310
-55
lines changed

.generator/schemas/v2/openapi.yaml

Lines changed: 53 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -43722,6 +43722,7 @@ components:
4372243722
ObservabilityPipelineConfigDestinationItem:
4372343723
description: "A destination for the pipeline."
4372443724
oneOf:
43725+
- $ref: "#/components/schemas/ObservabilityPipelineElasticsearchDestination"
4372543726
- $ref: "#/components/schemas/ObservabilityPipelineHttpClientDestination"
4372643727
- $ref: "#/components/schemas/ObservabilityPipelineAmazonOpenSearchDestination"
4372743728
- $ref: "#/components/schemas/ObservabilityPipelineAmazonS3Destination"
@@ -43731,7 +43732,6 @@ components:
4373143732
- $ref: "#/components/schemas/ObservabilityPipelineCloudPremDestination"
4373243733
- $ref: "#/components/schemas/ObservabilityPipelineCrowdStrikeNextGenSiemDestination"
4373343734
- $ref: "#/components/schemas/ObservabilityPipelineDatadogLogsDestination"
43734-
- $ref: "#/components/schemas/ObservabilityPipelineElasticsearchDestination"
4373543735
- $ref: "#/components/schemas/ObservabilityPipelineGoogleChronicleDestination"
4373643736
- $ref: "#/components/schemas/ObservabilityPipelineGoogleCloudStorageDestination"
4373743737
- $ref: "#/components/schemas/ObservabilityPipelineGooglePubSubDestination"
@@ -44333,9 +44333,9 @@ components:
4433344333
type: object
4433444334
ObservabilityPipelineElasticsearchDestination:
4433544335
description: |-
44336-
The `elasticsearch` destination writes logs to an Elasticsearch cluster.
44336+
The `elasticsearch` destination writes logs or metrics to an Elasticsearch cluster.
4433744337

44338-
**Supported pipeline types:** logs
44338+
**Supported pipeline types:** logs, metrics
4433944339
properties:
4434044340
api_version:
4434144341
$ref: "#/components/schemas/ObservabilityPipelineElasticsearchDestinationApiVersion"
@@ -44344,9 +44344,11 @@ components:
4434444344
buffer:
4434544345
$ref: "#/components/schemas/ObservabilityPipelineBufferOptions"
4434644346
bulk_index:
44347-
description: The index to write logs to in Elasticsearch.
44347+
description: The name of the index to write events to in Elasticsearch.
4434844348
example: logs-index
4434944349
type: string
44350+
compression:
44351+
$ref: "#/components/schemas/ObservabilityPipelineElasticsearchDestinationCompression"
4435044352
data_stream:
4435144353
$ref: "#/components/schemas/ObservabilityPipelineElasticsearchDestinationDataStream"
4435244354
endpoint_url_key:
@@ -44357,21 +44359,34 @@ components:
4435744359
description: The unique identifier for this component.
4435844360
example: "elasticsearch-destination"
4435944361
type: string
44362+
id_key:
44363+
description: The name of the field used as the document ID in Elasticsearch.
44364+
example: id
44365+
type: string
4436044366
inputs:
4436144367
description: A list of component IDs whose output is used as the `input` for this component.
4436244368
example: ["filter-processor"]
4436344369
items:
4436444370
description: The ID of a component whose output is used as input for this destination.
4436544371
type: string
4436644372
type: array
44373+
pipeline:
44374+
description: The name of an Elasticsearch ingest pipeline to apply to events before indexing.
44375+
example: my-pipeline
44376+
type: string
44377+
request_retry_partial:
44378+
description: When `true`, retries failed partial bulk requests when some events in a batch fail while others succeed.
44379+
type: boolean
44380+
tls:
44381+
$ref: "#/components/schemas/ObservabilityPipelineTls"
4436744382
type:
4436844383
$ref: "#/components/schemas/ObservabilityPipelineElasticsearchDestinationType"
4436944384
required:
4437044385
- id
4437144386
- type
4437244387
- inputs
4437344388
type: object
44374-
x-pipeline-types: [logs]
44389+
x-pipeline-types: [logs, metrics]
4437544390
ObservabilityPipelineElasticsearchDestinationApiVersion:
4437644391
description: The Elasticsearch API version to use. Set to `auto` to auto-detect.
4437744392
enum: [auto, v6, v7, v8]
@@ -44400,18 +44415,48 @@ components:
4440044415
required:
4440144416
- strategy
4440244417
type: object
44418+
ObservabilityPipelineElasticsearchDestinationCompression:
44419+
description: Compression configuration for the Elasticsearch destination.
44420+
properties:
44421+
algorithm:
44422+
$ref: "#/components/schemas/ObservabilityPipelineElasticsearchDestinationCompressionAlgorithm"
44423+
level:
44424+
description: The compression level. Only applicable for `gzip`, `zlib`, and `zstd` algorithms.
44425+
example: 6
44426+
format: int64
44427+
type: integer
44428+
required:
44429+
- algorithm
44430+
type: object
44431+
ObservabilityPipelineElasticsearchDestinationCompressionAlgorithm:
44432+
description: The compression algorithm applied when sending data to Elasticsearch.
44433+
enum: [none, gzip, zlib, zstd, snappy]
44434+
example: gzip
44435+
type: string
44436+
x-enum-varnames:
44437+
- NONE
44438+
- GZIP
44439+
- ZLIB
44440+
- ZSTD
44441+
- SNAPPY
4440344442
ObservabilityPipelineElasticsearchDestinationDataStream:
4440444443
description: Configuration options for writing to Elasticsearch Data Streams instead of a fixed index.
4440544444
properties:
44445+
auto_routing:
44446+
description: When `true`, automatically routes events to the appropriate data stream based on the event content.
44447+
type: boolean
4440644448
dataset:
44407-
description: The data stream dataset for your logs. This groups logs by their source or application.
44449+
description: The data stream dataset. This groups events by their source or application.
4440844450
type: string
4440944451
dtype:
44410-
description: The data stream type for your logs. This determines how logs are categorized within the data stream.
44452+
description: The data stream type. This determines how events are categorized within the data stream.
4441144453
type: string
4441244454
namespace:
44413-
description: The data stream namespace for your logs. This separates logs into different environments or domains.
44455+
description: The data stream namespace. This separates events into different environments or domains.
4441444456
type: string
44457+
sync_fields:
44458+
description: When `true`, synchronizes data stream fields with the Elasticsearch index mapping.
44459+
type: boolean
4441544460
type: object
4441644461
ObservabilityPipelineElasticsearchDestinationType:
4441744462
default: elasticsearch

docs/datadog_api_client.v2.model.rst

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19730,6 +19730,20 @@ datadog\_api\_client.v2.model.observability\_pipeline\_elasticsearch\_destinatio
1973019730
:members:
1973119731
:show-inheritance:
1973219732

19733+
datadog\_api\_client.v2.model.observability\_pipeline\_elasticsearch\_destination\_compression module
19734+
-----------------------------------------------------------------------------------------------------
19735+
19736+
.. automodule:: datadog_api_client.v2.model.observability_pipeline_elasticsearch_destination_compression
19737+
:members:
19738+
:show-inheritance:
19739+
19740+
datadog\_api\_client.v2.model.observability\_pipeline\_elasticsearch\_destination\_compression\_algorithm module
19741+
----------------------------------------------------------------------------------------------------------------
19742+
19743+
.. automodule:: datadog_api_client.v2.model.observability_pipeline_elasticsearch_destination_compression_algorithm
19744+
:members:
19745+
:show-inheritance:
19746+
1973319747
datadog\_api\_client.v2.model.observability\_pipeline\_elasticsearch\_destination\_data\_stream module
1973419748
------------------------------------------------------------------------------------------------------
1973519749

src/datadog_api_client/v2/model/observability_pipeline_config.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@
2626
from datadog_api_client.v2.model.observability_pipeline_config_source_item import (
2727
ObservabilityPipelineConfigSourceItem,
2828
)
29+
from datadog_api_client.v2.model.observability_pipeline_elasticsearch_destination import (
30+
ObservabilityPipelineElasticsearchDestination,
31+
)
2932
from datadog_api_client.v2.model.observability_pipeline_http_client_destination import (
3033
ObservabilityPipelineHttpClientDestination,
3134
)
@@ -51,9 +54,6 @@
5154
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import (
5255
ObservabilityPipelineDatadogLogsDestination,
5356
)
54-
from datadog_api_client.v2.model.observability_pipeline_elasticsearch_destination import (
55-
ObservabilityPipelineElasticsearchDestination,
56-
)
5757
from datadog_api_client.v2.model.observability_pipeline_google_chronicle_destination import (
5858
ObservabilityPipelineGoogleChronicleDestination,
5959
)
@@ -172,6 +172,7 @@ def __init__(
172172
destinations: List[
173173
Union[
174174
ObservabilityPipelineConfigDestinationItem,
175+
ObservabilityPipelineElasticsearchDestination,
175176
ObservabilityPipelineHttpClientDestination,
176177
ObservabilityPipelineAmazonOpenSearchDestination,
177178
ObservabilityPipelineAmazonS3Destination,
@@ -181,7 +182,6 @@ def __init__(
181182
ObservabilityPipelineCloudPremDestination,
182183
ObservabilityPipelineCrowdStrikeNextGenSiemDestination,
183184
ObservabilityPipelineDatadogLogsDestination,
184-
ObservabilityPipelineElasticsearchDestination,
185185
ObservabilityPipelineGoogleChronicleDestination,
186186
ObservabilityPipelineGoogleCloudStorageDestination,
187187
ObservabilityPipelineGooglePubSubDestination,

src/datadog_api_client/v2/model/observability_pipeline_config_destination_item.py

Lines changed: 46 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -15,52 +15,70 @@ def __init__(self, **kwargs):
1515
"""
1616
A destination for the pipeline.
1717
18-
:param auth_strategy: HTTP authentication strategy.
19-
:type auth_strategy: ObservabilityPipelineHttpClientDestinationAuthStrategy, optional
18+
:param api_version: The Elasticsearch API version to use. Set to `auto` to auto-detect.
19+
:type api_version: ObservabilityPipelineElasticsearchDestinationApiVersion, optional
2020
21-
:param compression: Compression configuration for HTTP requests.
22-
:type compression: ObservabilityPipelineHttpClientDestinationCompression, optional
21+
:param auth: Authentication settings for the Elasticsearch destination.
22+
When `strategy` is `basic`, use `username_key` and `password_key` to reference credentials stored in environment variables or secrets.
23+
:type auth: ObservabilityPipelineElasticsearchDestinationAuth, optional
2324
24-
:param custom_key: Name of the environment variable or secret that holds a custom header value (used with custom auth strategies).
25-
:type custom_key: str, optional
25+
:param buffer: Configuration for buffer settings on destination components.
26+
:type buffer: ObservabilityPipelineBufferOptions, optional
2627
27-
:param encoding: Encoding format for log events.
28-
:type encoding: ObservabilityPipelineHttpClientDestinationEncoding
28+
:param bulk_index: The name of the index to write events to in Elasticsearch.
29+
:type bulk_index: str, optional
30+
31+
:param compression: Compression configuration for the Elasticsearch destination.
32+
:type compression: ObservabilityPipelineElasticsearchDestinationCompression, optional
33+
34+
:param data_stream: Configuration options for writing to Elasticsearch Data Streams instead of a fixed index.
35+
:type data_stream: ObservabilityPipelineElasticsearchDestinationDataStream, optional
36+
37+
:param endpoint_url_key: Name of the environment variable or secret that holds the Elasticsearch endpoint URL.
38+
:type endpoint_url_key: str, optional
2939
3040
:param id: The unique identifier for this component.
3141
:type id: str
3242
33-
:param inputs: A list of component IDs whose output is used as the input for this component.
43+
:param id_key: The name of the field used as the document ID in Elasticsearch.
44+
:type id_key: str, optional
45+
46+
:param inputs: A list of component IDs whose output is used as the `input` for this component.
3447
:type inputs: [str]
3548
36-
:param password_key: Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`).
37-
:type password_key: str, optional
49+
:param pipeline: The name of an Elasticsearch ingest pipeline to apply to events before indexing.
50+
:type pipeline: str, optional
51+
52+
:param request_retry_partial: When `true`, retries failed partial bulk requests when some events in a batch fail while others succeed.
53+
:type request_retry_partial: bool, optional
3854
3955
:param tls: Configuration for enabling TLS encryption between the pipeline component and external services.
4056
:type tls: ObservabilityPipelineTls, optional
4157
58+
:param type: The destination type. The value should always be `elasticsearch`.
59+
:type type: ObservabilityPipelineElasticsearchDestinationType
60+
61+
:param auth_strategy: HTTP authentication strategy.
62+
:type auth_strategy: ObservabilityPipelineHttpClientDestinationAuthStrategy, optional
63+
64+
:param custom_key: Name of the environment variable or secret that holds a custom header value (used with custom auth strategies).
65+
:type custom_key: str, optional
66+
67+
:param encoding: Encoding format for log events.
68+
:type encoding: ObservabilityPipelineHttpClientDestinationEncoding
69+
70+
:param password_key: Name of the environment variable or secret that holds the password (used when `auth_strategy` is `basic`).
71+
:type password_key: str, optional
72+
4273
:param token_key: Name of the environment variable or secret that holds the bearer token (used when `auth_strategy` is `bearer`).
4374
:type token_key: str, optional
4475
45-
:param type: The destination type. The value should always be `http_client`.
46-
:type type: ObservabilityPipelineHttpClientDestinationType
47-
4876
:param uri_key: Name of the environment variable or secret that holds the HTTP endpoint URI.
4977
:type uri_key: str, optional
5078
5179
:param username_key: Name of the environment variable or secret that holds the username (used when `auth_strategy` is `basic`).
5280
:type username_key: str, optional
5381
54-
:param auth: Authentication settings for the Amazon OpenSearch destination.
55-
The `strategy` field determines whether basic or AWS-based authentication is used.
56-
:type auth: ObservabilityPipelineAmazonOpenSearchDestinationAuth
57-
58-
:param buffer: Configuration for buffer settings on destination components.
59-
:type buffer: ObservabilityPipelineBufferOptions, optional
60-
61-
:param bulk_index: The index to write logs to.
62-
:type bulk_index: str, optional
63-
6482
:param bucket: S3 bucket name.
6583
:type bucket: str
6684
@@ -88,18 +106,9 @@ def __init__(self, **kwargs):
88106
:param container_name: The name of the Azure Blob Storage container to store logs in.
89107
:type container_name: str
90108
91-
:param endpoint_url_key: Name of the environment variable or secret that holds the CloudPrem endpoint URL.
92-
:type endpoint_url_key: str, optional
93-
94109
:param routes: A list of routing rules that forward matching logs to Datadog using dedicated API keys.
95110
:type routes: [ObservabilityPipelineDatadogLogsDestinationRoute], optional
96111
97-
:param api_version: The Elasticsearch API version to use. Set to `auto` to auto-detect.
98-
:type api_version: ObservabilityPipelineElasticsearchDestinationApiVersion, optional
99-
100-
:param data_stream: Configuration options for writing to Elasticsearch Data Streams instead of a fixed index.
101-
:type data_stream: ObservabilityPipelineElasticsearchDestinationDataStream, optional
102-
103112
:param customer_id: The Google Chronicle customer ID.
104113
:type customer_id: str
105114
@@ -217,6 +226,9 @@ def _composed_schemas(_):
217226
# code would be run when this module is imported, and these composed
218227
# classes don't exist yet because their module has not finished
219228
# loading
229+
from datadog_api_client.v2.model.observability_pipeline_elasticsearch_destination import (
230+
ObservabilityPipelineElasticsearchDestination,
231+
)
220232
from datadog_api_client.v2.model.observability_pipeline_http_client_destination import (
221233
ObservabilityPipelineHttpClientDestination,
222234
)
@@ -242,9 +254,6 @@ def _composed_schemas(_):
242254
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import (
243255
ObservabilityPipelineDatadogLogsDestination,
244256
)
245-
from datadog_api_client.v2.model.observability_pipeline_elasticsearch_destination import (
246-
ObservabilityPipelineElasticsearchDestination,
247-
)
248257
from datadog_api_client.v2.model.observability_pipeline_google_chronicle_destination import (
249258
ObservabilityPipelineGoogleChronicleDestination,
250259
)
@@ -288,6 +297,7 @@ def _composed_schemas(_):
288297

289298
return {
290299
"oneOf": [
300+
ObservabilityPipelineElasticsearchDestination,
291301
ObservabilityPipelineHttpClientDestination,
292302
ObservabilityPipelineAmazonOpenSearchDestination,
293303
ObservabilityPipelineAmazonS3Destination,
@@ -297,7 +307,6 @@ def _composed_schemas(_):
297307
ObservabilityPipelineCloudPremDestination,
298308
ObservabilityPipelineCrowdStrikeNextGenSiemDestination,
299309
ObservabilityPipelineDatadogLogsDestination,
300-
ObservabilityPipelineElasticsearchDestination,
301310
ObservabilityPipelineGoogleChronicleDestination,
302311
ObservabilityPipelineGoogleCloudStorageDestination,
303312
ObservabilityPipelineGooglePubSubDestination,

0 commit comments

Comments
 (0)