Skip to content

Commit a8dde96

Browse files
api-clients-generation-pipeline[bot]ci.datadog-api-spec
api-clients-generation-pipeline[bot]
and
ci.datadog-api-spec
authored
Add OpenAPI documentation for cache property in dedupe processor in observability pipelines (DataDog#2993)
Co-authored-by: ci.datadog-api-spec <packages@datadoghq.com>
1 parent b81a7c9 commit a8dde96

11 files changed

Lines changed: 379 additions & 1 deletion

‎.generator/schemas/v2/openapi.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39514,6 +39514,8 @@ components:
3951439514

3951539515
**Supported pipeline types:** logs'
3951639516
properties:
39517+
cache:
39518+
$ref: '#/components/schemas/ObservabilityPipelineDedupeProcessorCache'
3951739519
display_name:
3951839520
$ref: '#/components/schemas/ObservabilityPipelineComponentDisplayName'
3951939521
enabled:
@@ -39551,6 +39553,19 @@ components:
3955139553
type: object
3955239554
x-pipeline-types:
3955339555
- logs
39556+
ObservabilityPipelineDedupeProcessorCache:
39557+
description: Configuration for the cache used to detect duplicates.
39558+
properties:
39559+
num_events:
39560+
description: The number of events to cache for duplicate detection.
39561+
example: 5000
39562+
format: int64
39563+
maximum: 1000000000
39564+
minimum: 1
39565+
type: integer
39566+
required:
39567+
- num_events
39568+
type: object
3955439569
ObservabilityPipelineDedupeProcessorMode:
3955539570
description: The deduplication mode to apply to the fields.
3955639571
enum:

‎cassettes/features/v2/observability_pipelines/Create-a-pipeline-with-dedupe-processor-with-cache-returns-OK-response.frozen‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
2026-02-09T09:53:31.840Z

‎cassettes/features/v2/observability_pipelines/Create-a-pipeline-with-dedupe-processor-with-cache-returns-OK-response.yml‎

Lines changed: 46 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎cassettes/features/v2/observability_pipelines/Create-a-pipeline-with-dedupe-processor-without-cache-returns-OK-response.frozen‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
2026-02-09T09:53:33.945Z

‎cassettes/features/v2/observability_pipelines/Create-a-pipeline-with-dedupe-processor-without-cache-returns-OK-response.yml‎

Lines changed: 46 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎examples/v2/observability-pipelines/CreatePipeline_3363445359.rb‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# Create a pipeline with dedupe processor with cache returns "OK" response
2+
3+
require "datadog_api_client"
4+
api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new
5+
6+
body = DatadogAPIClient::V2::ObservabilityPipelineSpec.new({
7+
data: DatadogAPIClient::V2::ObservabilityPipelineSpecData.new({
8+
attributes: DatadogAPIClient::V2::ObservabilityPipelineDataAttributes.new({
9+
config: DatadogAPIClient::V2::ObservabilityPipelineConfig.new({
10+
destinations: [
11+
DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestination.new({
12+
id: "datadog-logs-destination",
13+
inputs: [
14+
"my-processor-group",
15+
],
16+
type: DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS,
17+
}),
18+
],
19+
processor_groups: [
20+
DatadogAPIClient::V2::ObservabilityPipelineConfigProcessorGroup.new({
21+
enabled: true,
22+
id: "my-processor-group",
23+
include: "service:my-service",
24+
inputs: [
25+
"datadog-agent-source",
26+
],
27+
processors: [
28+
DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessor.new({
29+
enabled: true,
30+
id: "dedupe-processor",
31+
include: "service:my-service",
32+
type: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorType::DEDUPE,
33+
fields: [
34+
"message",
35+
],
36+
mode: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorMode::MATCH,
37+
cache: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorCache.new({
38+
num_events: 5000,
39+
}),
40+
}),
41+
],
42+
}),
43+
],
44+
sources: [
45+
DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSource.new({
46+
id: "datadog-agent-source",
47+
type: DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT,
48+
}),
49+
],
50+
}),
51+
name: "Pipeline with Dedupe Cache",
52+
}),
53+
type: "pipelines",
54+
}),
55+
})
56+
p api_instance.create_pipeline(body)

‎examples/v2/observability-pipelines/CreatePipeline_581245895.rb‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Create a pipeline with dedupe processor without cache returns "OK" response
2+
3+
require "datadog_api_client"
4+
api_instance = DatadogAPIClient::V2::ObservabilityPipelinesAPI.new
5+
6+
body = DatadogAPIClient::V2::ObservabilityPipelineSpec.new({
7+
data: DatadogAPIClient::V2::ObservabilityPipelineSpecData.new({
8+
attributes: DatadogAPIClient::V2::ObservabilityPipelineDataAttributes.new({
9+
config: DatadogAPIClient::V2::ObservabilityPipelineConfig.new({
10+
destinations: [
11+
DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestination.new({
12+
id: "datadog-logs-destination",
13+
inputs: [
14+
"my-processor-group",
15+
],
16+
type: DatadogAPIClient::V2::ObservabilityPipelineDatadogLogsDestinationType::DATADOG_LOGS,
17+
}),
18+
],
19+
processor_groups: [
20+
DatadogAPIClient::V2::ObservabilityPipelineConfigProcessorGroup.new({
21+
enabled: true,
22+
id: "my-processor-group",
23+
include: "service:my-service",
24+
inputs: [
25+
"datadog-agent-source",
26+
],
27+
processors: [
28+
DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessor.new({
29+
enabled: true,
30+
id: "dedupe-processor",
31+
include: "service:my-service",
32+
type: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorType::DEDUPE,
33+
fields: [
34+
"message",
35+
],
36+
mode: DatadogAPIClient::V2::ObservabilityPipelineDedupeProcessorMode::MATCH,
37+
}),
38+
],
39+
}),
40+
],
41+
sources: [
42+
DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSource.new({
43+
id: "datadog-agent-source",
44+
type: DatadogAPIClient::V2::ObservabilityPipelineDatadogAgentSourceType::DATADOG_AGENT,
45+
}),
46+
],
47+
}),
48+
name: "Pipeline with Dedupe No Cache",
49+
}),
50+
type: "pipelines",
51+
}),
52+
})
53+
p api_instance.create_pipeline(body)

‎features/v2/observability_pipelines.feature‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,24 @@ Feature: Observability Pipelines
3535
And the response "data.attributes.config.processor_groups" has length 1
3636
And the response "data.attributes.config.destinations" has length 1
3737

38+
@team:DataDog/observability-pipelines
39+
Scenario: Create a pipeline with dedupe processor with cache returns "OK" response
40+
Given new "CreatePipeline" request
41+
And body with value {"data": {"attributes": {"config": {"destinations": [{"id": "datadog-logs-destination", "inputs": ["my-processor-group"], "type": "datadog_logs"}], "processor_groups": [{"enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": ["datadog-agent-source"], "processors": [{"enabled": true, "id": "dedupe-processor", "include": "service:my-service", "type": "dedupe", "fields": ["message"], "mode": "match", "cache": {"num_events": 5000}}]}], "sources": [{"id": "datadog-agent-source", "type": "datadog_agent"}]}, "name": "Pipeline with Dedupe Cache"}, "type": "pipelines"}}
42+
When the request is sent
43+
Then the response status is 201 OK
44+
And the response "data.attributes.config.processor_groups[0].processors[0].type" is equal to "dedupe"
45+
And the response "data.attributes.config.processor_groups[0].processors[0].cache.num_events" is equal to 5000
46+
47+
@team:DataDog/observability-pipelines
48+
Scenario: Create a pipeline with dedupe processor without cache returns "OK" response
49+
Given new "CreatePipeline" request
50+
And body with value {"data": {"attributes": {"config": {"destinations": [{"id": "datadog-logs-destination", "inputs": ["my-processor-group"], "type": "datadog_logs"}], "processor_groups": [{"enabled": true, "id": "my-processor-group", "include": "service:my-service", "inputs": ["datadog-agent-source"], "processors": [{"enabled": true, "id": "dedupe-processor", "include": "service:my-service", "type": "dedupe", "fields": ["message"], "mode": "match"}]}], "sources": [{"id": "datadog-agent-source", "type": "datadog_agent"}]}, "name": "Pipeline with Dedupe No Cache"}, "type": "pipelines"}}
51+
When the request is sent
52+
Then the response status is 201 OK
53+
And the response "data.attributes.config.processor_groups[0].processors[0].type" is equal to "dedupe"
54+
And the response "data.attributes.config.processor_groups[0].processors[0].fields[0]" is equal to "message"
55+
3856
@generated @skip @team:DataDog/observability-pipelines
3957
Scenario: Delete a pipeline returns "Conflict" response
4058
Given new "DeletePipeline" request

‎lib/datadog_api_client/inflector.rb‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3412,6 +3412,7 @@ def overrides
34123412
"v2.observability_pipeline_datadog_tags_processor_type" => "ObservabilityPipelineDatadogTagsProcessorType",
34133413
"v2.observability_pipeline_decoding" => "ObservabilityPipelineDecoding",
34143414
"v2.observability_pipeline_dedupe_processor" => "ObservabilityPipelineDedupeProcessor",
3415+
"v2.observability_pipeline_dedupe_processor_cache" => "ObservabilityPipelineDedupeProcessorCache",
34153416
"v2.observability_pipeline_dedupe_processor_mode" => "ObservabilityPipelineDedupeProcessorMode",
34163417
"v2.observability_pipeline_dedupe_processor_type" => "ObservabilityPipelineDedupeProcessorType",
34173418
"v2.observability_pipeline_disk_buffer_options" => "ObservabilityPipelineDiskBufferOptions",

‎lib/datadog_api_client/v2/models/observability_pipeline_dedupe_processor.rb‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,9 @@ module DatadogAPIClient::V2
2323
class ObservabilityPipelineDedupeProcessor
2424
include BaseGenericModel
2525

26+
# Configuration for the cache used to detect duplicates.
27+
attr_accessor :cache
28+
2629
# The display name for a component.
2730
attr_accessor :display_name
2831

@@ -50,6 +53,7 @@ class ObservabilityPipelineDedupeProcessor
5053
# @!visibility private
5154
def self.attribute_map
5255
{
56+
:'cache' => :'cache',
5357
:'display_name' => :'display_name',
5458
:'enabled' => :'enabled',
5559
:'fields' => :'fields',
@@ -64,6 +68,7 @@ def self.attribute_map
6468
# @!visibility private
6569
def self.openapi_types
6670
{
71+
:'cache' => :'ObservabilityPipelineDedupeProcessorCache',
6772
:'display_name' => :'String',
6873
:'enabled' => :'Boolean',
6974
:'fields' => :'Array<String>',
@@ -92,6 +97,10 @@ def initialize(attributes = {})
9297
end
9398
}
9499

100+
if attributes.key?(:'cache')
101+
self.cache = attributes[:'cache']
102+
end
103+
95104
if attributes.key?(:'display_name')
96105
self.display_name = attributes[:'display_name']
97106
end
@@ -222,6 +231,7 @@ def to_hash
222231
def ==(o)
223232
return true if self.equal?(o)
224233
self.class == o.class &&
234+
cache == o.cache &&
225235
display_name == o.display_name &&
226236
enabled == o.enabled &&
227237
fields == o.fields &&
@@ -236,7 +246,7 @@ def ==(o)
236246
# @return [Integer] Hash code
237247
# @!visibility private
238248
def hash
239-
[display_name, enabled, fields, id, include, mode, type, additional_properties].hash
249+
[cache, display_name, enabled, fields, id, include, mode, type, additional_properties].hash
240250
end
241251
end
242252
end

0 commit comments

Comments
 (0)