Skip to content

Commit fd2349d

Browse files
MichelLosierMichelLosier
authored
Otel profiles input package support (#1092)
This PR adds profiles as a policy_template type for OTel input packages. And it also adds the index template settings auto_expand_replicas, max_result_window, and refresh_interval.
1 parent 0887128 commit fd2349d

14 files changed

Lines changed: 180 additions & 14 deletions

File tree

‎code/go/pkg/validator/validator_test.go‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ func TestValidateFile(t *testing.T) {
3737
"good_input": {},
3838
"good_input_otel": {},
3939
"good_input_dynamic_signal_type": {},
40+
"good_input_profiles": {},
4041
"good_input_template_paths": {},
4142
"good_integration_template_paths": {},
4243
"good_content": {},
@@ -390,6 +391,12 @@ func TestValidateFile(t *testing.T) {
390391
"field policy_templates.0.input: Must not be present",
391392
},
392393
},
394+
"bad_input_profiles_non_otel": {
395+
"manifest.yml",
396+
[]string{
397+
"field policy_templates.0.input: policy_templates.0.input must be one of the following: \"otelcol\"",
398+
},
399+
},
393400
"bad_input_dynamic_signal_types_non_otel": {
394401
"manifest.yml",
395402
[]string{

‎compliance/features/basic.feature‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,3 +40,9 @@ Feature: Basic package types support
4040
Given the "good_v3" package is installed
4141
And a policy is created with "good_v3" package, "1.1.0" version, "otel" template, "otelcol" input type and dataset "good_v3.otel_test"
4242
Then there is an index template "logs-good_v3.otel_test" with pattern "logs-good_v3.otel_test.otel-*"
43+
44+
@3.6.0
45+
Scenario: OTel input package with profiles type can be installed
46+
Given the "good_input_profiles" package is installed
47+
And a policy is created with "good_input_profiles" package, "0.0.1" version, "profilingreceiver" template, "profilingreceiver" input, "otelcol" input type and dataset "spec.otel_input_test"
48+
# We omit assertions here because 'profiles' OTel packages should not produce ES assets which is currently handled by an ES plugin. https://github.com/elastic/package-spec/pull/1092#discussion_r2857893043

‎spec/changelog.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,12 @@
5858
- description: Allow to force the format used in the policy API in system and policy tests.
5959
type: enhancement
6060
link: https://github.com/elastic/package-spec/pull/1103
61+
- description: Add support for auto_expand_replicas, max_result_window, and refresh_interval index settings in data stream templates.
62+
type: enhancement
63+
link: https://github.com/elastic/package-spec/pull/1092
64+
- description: Add support for `profiles` policy template type in input packages
65+
type: enhancement
66+
link: https://github.com/elastic/package-spec/pull/1092
6167
- version: 3.5.7
6268
changes:
6369
- description: Allow _dev directory for content-only packages; use _dev/shared for development files (e.g. dashboard YML sources).

‎spec/input/manifest.spec.yml‎

Lines changed: 30 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ spec:
5555
- logs
5656
- synthetics
5757
- traces
58+
- profiles
5859
examples:
5960
- metrics
6061
title:
@@ -96,23 +97,36 @@ spec:
9697
allOf:
9798
# Either template_path or template_paths must be present (mutually exclusive)
9899
- oneOf:
99-
- required:
100-
- template_path
101-
not:
102-
required:
103-
- template_paths
104-
- required:
100+
- required:
101+
- template_path
102+
not:
103+
required:
105104
- template_paths
106-
not:
107-
required:
108-
- template_path
105+
- required:
106+
- template_paths
107+
not:
108+
required:
109+
- template_path
109110
# Either type is required, or dynamic_signal_types must be true
110111
- oneOf:
112+
- allOf:
111113
- required:
112-
- type
113-
- properties:
114-
dynamic_signal_types:
115-
const: true
114+
- type
115+
# If type is profiles, can only use otelcol inputs
116+
- if:
117+
properties:
118+
type:
119+
const: profiles
120+
then:
121+
properties:
122+
input:
123+
enum:
124+
- otelcol
125+
- required:
126+
- dynamic_signal_types
127+
properties:
128+
dynamic_signal_types:
129+
const: true
116130
required:
117131
- name
118132
- title
@@ -155,7 +169,7 @@ versions:
155169
- op: remove
156170
path: "/properties/policy_templates/items/properties/template_paths" # removes template_paths field for policy templates
157171
- op: remove
158-
path: "/properties/policy_templates/items/allOf" # removes allOf constraint for template_path/template_paths and dynamic_signal_types/type
172+
path: "/properties/policy_templates/items/allOf" # removes allOf constraint for template_path/template_paths, dynamic_signal_types/type, and profiles otelcol input
159173
- op: add
160174
path: "/properties/policy_templates/items/required"
161175
value:
@@ -173,6 +187,8 @@ versions:
173187
path: "/properties/deprecated" # removes deprecated field for package
174188
- op: remove
175189
path: "/properties/policy_templates/items/properties/deprecated" # removes deprecated field for policy templates
190+
- op: remove
191+
path: "/properties/policy_templates/items/properties/type/enum/4" # removes 'profiles' as a valid policy template type, since it requires Kibana version >= 9.4.0 support
176192
# Reserve otelcol input name before 3.5.0.
177193
- before: 3.5.0
178194
patch:

‎spec/integration/data_stream/manifest.spec.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -263,6 +263,12 @@ spec:
263263
$ref: "#/definitions/index_template_setting_index"
264264
number_of_shards:
265265
$ref: "#/definitions/index_template_setting_number_of_shards"
266+
auto_expand_replicas:
267+
$ref: "#/definitions/index_template_setting_auto_expand_replicas"
268+
max_result_window:
269+
$ref: "#/definitions/index_template_setting_max_result_window"
270+
refresh_interval:
271+
$ref: "#/definitions/index_template_setting_refresh_interval"
266272
mappings:
267273
description: Mappings section of index template
268274
type: object
@@ -512,6 +518,32 @@ spec:
512518
type: integer
513519
minimum: 1
514520
default: 1
521+
index_template_setting_auto_expand_replicas:
522+
description: |
523+
Auto-expand the number of replicas based on the number of data nodes in the cluster.
524+
Set to a dash delimited lower and upper bound (e.g. 0-5) or use `all` for the upper bound (e.g. 0-all). Defaults to false (i.e. disabled).
525+
type: string
526+
examples:
527+
- "0-1"
528+
- "0-all"
529+
- "false"
530+
index_template_setting_max_result_window:
531+
description: |
532+
The maximum value of `from + size` for searches to this index. Defaults to 10000.
533+
Search requests take heap memory and time proportional to `from + size` and this limits that memory.
534+
type: integer
535+
minimum: 1
536+
examples:
537+
- 50000
538+
index_template_setting_refresh_interval:
539+
description: |
540+
How often to perform a refresh operation, which makes recent changes to the index visible to search. The value defaults to `1s` in Elastic Stack and `5s` in Serverless. In Serverless, `5s` is also the minimum value that can be set.
541+
Can be set to `-1` to disable refresh.
542+
type: string
543+
examples:
544+
- "10s"
545+
- "1m"
546+
- "-1"
515547
properties:
516548
dataset:
517549
description: Name of data set.

‎test/packages/bad_input_profiles_non_otel/agent/input/input.yml.hbs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
receivers:
2+
profiling:
3+
{{#if samples_per_second}}
4+
samples_per_second: {{samples_per_second}}
5+
{{/if}}
6+
7+
service:
8+
pipelines:
9+
profiles:
10+
receivers: [ profiling ]

‎test/packages/bad_input_profiles_non_otel/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
- version: 0.0.1
2+
changes:
3+
- type: enhancement
4+
description: Initial release for testing input policy templates with type `profiles` with a non otelcol input type.
5+
link: https://github.com/elastic/package-spec/pull/1092

‎test/packages/bad_input_profiles_non_otel/docs/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
This is a template for the package README.

‎test/packages/bad_input_profiles_non_otel/manifest.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
format_version: 3.6.0
2+
name: bad_input_profiles_non_otel
3+
title: "Bad OpenTelemetry Profiling"
4+
version: 0.0.1
5+
source:
6+
license: "Elastic-2.0"
7+
description: "Bad OpenTelemetry Profiling"
8+
type: input
9+
categories:
10+
- custom
11+
- observability
12+
- opentelemetry
13+
conditions:
14+
kibana:
15+
version: "^9.4.0"
16+
elastic:
17+
subscription: "basic"
18+
policy_templates:
19+
- name: profilingreceiver
20+
type: profiles
21+
title: OpenTelemetry Profiling receiver
22+
description: Collect profiling data
23+
input: logs
24+
template_path: input.yml.hbs
25+
vars:
26+
- name: samples_per_second
27+
type: integer
28+
required: false
29+
description: Defines the sampling frequency.
30+
owner:
31+
github: elastic/ingest-otel-data
32+
type: elastic

‎test/packages/good_input_profiles/agent/input/input.yml.hbs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
receivers:
2+
profiling:
3+
{{#if samples_per_second}}
4+
samples_per_second: {{samples_per_second}}
5+
{{/if}}
6+
7+
service:
8+
pipelines:
9+
profiles:
10+
receivers: [ profiling ]

0 commit comments

Comments
 (0)