Skip to content

Commit 08e2f1d

Browse files
vamsimanoharclaude
vamsimanohar
and
claude
authored
Unified Node/NodeOperationDetail model for APM service map processor (#6536)
* Refactor to unified Node/NodeOperationDetail with SERVICE_MAP_V2 events - Replace Service/ServiceConnection/ServiceOperationDetail with unified Node/NodeOperationDetail model - Node adds type field for future entity types (database, queue, etc) - Operation simplified to name + attributes - NodeOperationDetail single entity with dual hash fields (nodeConnectionHash, operationConnectionHash) - CLIENT-span-primary emission: CLIENT spans emit full NodeOperationDetail, leaf SERVER spans for services with no outgoing calls - Update eventType to SERVICE_MAP_V2 for new index pattern otel-v2-apm-service-map Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Refactor model to unified Node/NodeOperationDetail with CLIENT-primary emission Refactors the APM service map processor to use a unified model structure replacing Service/ServiceConnection/ServiceOperationDetail with Node/NodeOperationDetail. Implements CLIENT-span-primary emission algorithm to eliminate duplicate events. Model Changes: - Node.java: Unified service entity with type field for future extensibility - NodeOperationDetail: Single entity for both topology and operation events - Operation: Simplified to name + attributes structure - CLIENT-primary algorithm: CLIENT spans emit complete events using decoration data OpenSearch Integration: - Added OTEL_APM_SERVICE_MAP index type following PR #6435 patterns - Index template with dynamic mappings for groupByAttributes and operation attributes - ISM policies for automated rollover (10gb/24h) - Updated IndexManagerFactory, IndexConstants, IndexConfiguration Configuration: - Changed eventType from SERVICE_MAP_V2 to SERVICE_MAP - Updated README with proper index_type: otel-v2-apm-service-map configuration - Coexists with legacy service_map processor using different index patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Update README with algorithm docs and remove standalone MD files - Merged NodeOperationDetail algorithm documentation into processor README - Updated output events section to reflect unified Node/NodeOperationDetail model - Added detailed metrics generation documentation - Removed standalone node-operation-detail-algorithm.md from tracking - Added algorithm design MD files to .gitignore Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Fix index-template version to use composable template format The index-template/ copy needs the "template" wrapper for the modern OpenSearch composable index template API, while the root copy uses the V1 legacy format. Previously both were identical (V1 format). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Clean up .gitignore and optimize index templates - Remove local-only entries from .gitignore (algorithm MDs, claude.md) - Collapse 12 dynamic templates to 6 using wildcard path matching - Remove ignore_above restrictions from all keyword fields - Fix index-template/ copy to use composable template format Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Remove eventType from index templates and fix test issues - Remove eventType field from both index templates (V1 and composable) eventType is pipeline routing metadata, not part of NodeOperationDetail model - Remove broken testWindowProcessingWithInterruptedException test Test doesn't work with @DataPrepperPluginTest annotation due to override restrictions Addresses review feedback from @kkondaka Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> --------- Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 00d41cb commit 08e2f1d

24 files changed

Lines changed: 1237 additions & 885 deletions

File tree

‎.gitignore‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,5 +27,5 @@ pipelines.yaml
2727
run-dataprepper.sh
2828
trace-tester.py
2929

30-
# Claude session notes
31-
claude.md
30+
# Claude project instructions
31+
CLAUDE.md

‎data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexConfiguration.java‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -434,6 +434,8 @@ private Map<String, Object> readIndexTemplate(final String templateFile, final I
434434
templateURL = loadExistingTemplate(templateType, IndexConstants.RAW_STANDARD_TEMPLATE_FILE);
435435
} else if (indexType.equals(IndexType.TRACE_ANALYTICS_SERVICE_MAP)) {
436436
templateURL = loadExistingTemplate(templateType, IndexConstants.SERVICE_MAP_DEFAULT_TEMPLATE_FILE);
437+
} else if (indexType.equals(IndexType.OTEL_APM_SERVICE_MAP)) {
438+
templateURL = loadExistingTemplate(templateType, IndexConstants.OTEL_APM_SERVICE_MAP_TEMPLATE_FILE);
437439
} else if (indexType.equals(IndexType.LOG_ANALYTICS)) {
438440
templateURL = loadExistingTemplate(templateType, IndexConstants.LOGS_DEFAULT_TEMPLATE_FILE);
439441
} else if (indexType.equals(IndexType.LOG_ANALYTICS_PLAIN)) {

‎data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexConstants.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,15 @@ public class IndexConstants {
3737
public static final String ISM_ROLLOVER_ALIAS_SETTING = "opendistro.index_state_management.rollover_alias";
3838
// TODO: extract out version number into version enum
3939
public static final String SERVICE_MAP_DEFAULT_TEMPLATE_FILE = "otel-v1-apm-service-map-index-template.json";
40+
public static final String OTEL_APM_SERVICE_MAP_TEMPLATE_FILE = "otel-v2-apm-service-map-index-template.json";
41+
public static final String OTEL_APM_SERVICE_MAP_ISM_POLICY = "otel-v2-apm-service-map-policy";
42+
public static final String OTEL_APM_SERVICE_MAP_ISM_FILE_NO_ISM_TEMPLATE = "otel-v2-apm-service-map-policy-no-ism-template.json";
43+
public static final String OTEL_APM_SERVICE_MAP_ISM_FILE_WITH_ISM_TEMPLATE = "otel-v2-apm-service-map-policy-with-ism-template.json";
4044

4145
static {
4246
// TODO: extract out version number into version enum
4347
TYPE_TO_DEFAULT_ALIAS.put(IndexType.TRACE_ANALYTICS_SERVICE_MAP, "otel-v1-apm-service-map");
48+
TYPE_TO_DEFAULT_ALIAS.put(IndexType.OTEL_APM_SERVICE_MAP, "otel-v2-apm-service-map");
4449
TYPE_TO_DEFAULT_ALIAS.put(IndexType.TRACE_ANALYTICS_RAW, "otel-v1-apm-span");
4550
TYPE_TO_DEFAULT_ALIAS.put(IndexType.TRACE_ANALYTICS_RAW_PLAIN, "otel-v1-apm-span");
4651
TYPE_TO_DEFAULT_ALIAS.put(IndexType.LOG_ANALYTICS, "logs-otel-v1");

‎data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexManagerFactory.java‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,10 @@ public final IndexManager getIndexManager(final IndexType indexType,
5959
indexManager = new TraceAnalyticsServiceMapIndexManager(
6060
restHighLevelClient, openSearchClient, openSearchSinkConfiguration, clusterSettingsParser, templateStrategy, indexAlias);
6161
break;
62+
case OTEL_APM_SERVICE_MAP:
63+
indexManager = new OTelAPMServiceMapIndexManager(
64+
restHighLevelClient, openSearchClient, openSearchSinkConfiguration, clusterSettingsParser, templateStrategy, indexAlias);
65+
break;
6266
case LOG_ANALYTICS:
6367
case LOG_ANALYTICS_PLAIN:
6468
indexManager = new LogAnalyticsIndexManager(
@@ -151,6 +155,24 @@ public TraceAnalyticsServiceMapIndexManager(final RestHighLevelClient restHighLe
151155
}
152156
}
153157

158+
private static class OTelAPMServiceMapIndexManager extends AbstractIndexManager {
159+
160+
public OTelAPMServiceMapIndexManager(final RestHighLevelClient restHighLevelClient,
161+
final OpenSearchClient openSearchClient,
162+
final OpenSearchSinkConfiguration openSearchSinkConfiguration,
163+
final ClusterSettingsParser clusterSettingsParser,
164+
final TemplateStrategy templateStrategy,
165+
final String indexAlias) {
166+
super(restHighLevelClient, openSearchClient, openSearchSinkConfiguration, clusterSettingsParser, templateStrategy, indexAlias);
167+
this.ismPolicyManagementStrategy = new IsmPolicyManagement(
168+
openSearchClient,
169+
restHighLevelClient,
170+
IndexConstants.OTEL_APM_SERVICE_MAP_ISM_POLICY,
171+
IndexConstants.OTEL_APM_SERVICE_MAP_ISM_FILE_WITH_ISM_TEMPLATE,
172+
IndexConstants.OTEL_APM_SERVICE_MAP_ISM_FILE_NO_ISM_TEMPLATE);
173+
}
174+
}
175+
154176
private static class LogAnalyticsIndexManager extends AbstractIndexManager {
155177

156178
public LogAnalyticsIndexManager(final RestHighLevelClient restHighLevelClient,

‎data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/index/IndexType.java‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ public enum IndexType {
1515
TRACE_ANALYTICS_RAW("trace-analytics-raw"),
1616
TRACE_ANALYTICS_RAW_PLAIN("trace-analytics-plain-raw"),
1717
TRACE_ANALYTICS_SERVICE_MAP("trace-analytics-service-map"),
18+
OTEL_APM_SERVICE_MAP("otel-v2-apm-service-map"),
1819
LOG_ANALYTICS("log-analytics"),
1920
LOG_ANALYTICS_PLAIN("log-analytics-plain"),
2021
METRIC_ANALYTICS("metric-analytics"),

‎data-prepper-plugins/opensearch/src/main/resources/index-template/otel-v2-apm-service-map-index-template.json‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
{
2+
"version": 0,
3+
"template": {
4+
"mappings": {
5+
"date_detection": false,
6+
"dynamic_templates": [
7+
{
8+
"long_group_by_attributes": {
9+
"path_match": "*.groupByAttributes.*",
10+
"match_mapping_type": "long",
11+
"mapping": {
12+
"type": "long"
13+
}
14+
}
15+
},
16+
{
17+
"double_group_by_attributes": {
18+
"path_match": "*.groupByAttributes.*",
19+
"match_mapping_type": "double",
20+
"mapping": {
21+
"type": "double"
22+
}
23+
}
24+
},
25+
{
26+
"string_group_by_attributes": {
27+
"path_match": "*.groupByAttributes.*",
28+
"match_mapping_type": "string",
29+
"mapping": {
30+
"type": "keyword"
31+
}
32+
}
33+
},
34+
{
35+
"long_operation_attributes": {
36+
"path_match": "*.attributes.*",
37+
"match_mapping_type": "long",
38+
"mapping": {
39+
"type": "long"
40+
}
41+
}
42+
},
43+
{
44+
"double_operation_attributes": {
45+
"path_match": "*.attributes.*",
46+
"match_mapping_type": "double",
47+
"mapping": {
48+
"type": "double"
49+
}
50+
}
51+
},
52+
{
53+
"string_operation_attributes": {
54+
"path_match": "*.attributes.*",
55+
"match_mapping_type": "string",
56+
"mapping": {
57+
"type": "keyword"
58+
}
59+
}
60+
}
61+
],
62+
"_source": {
63+
"enabled": true
64+
},
65+
"properties": {
66+
"sourceNode": {
67+
"properties": {
68+
"type": {
69+
"type": "keyword"
70+
},
71+
"keyAttributes": {
72+
"properties": {
73+
"environment": {
74+
"type": "keyword"
75+
},
76+
"name": {
77+
"type": "keyword"
78+
}
79+
}
80+
},
81+
"groupByAttributes": {
82+
"type": "object",
83+
"dynamic": "true"
84+
}
85+
}
86+
},
87+
"targetNode": {
88+
"properties": {
89+
"type": {
90+
"type": "keyword"
91+
},
92+
"keyAttributes": {
93+
"properties": {
94+
"environment": {
95+
"type": "keyword"
96+
},
97+
"name": {
98+
"type": "keyword"
99+
}
100+
}
101+
},
102+
"groupByAttributes": {
103+
"type": "object",
104+
"dynamic": "true"
105+
}
106+
}
107+
},
108+
"sourceOperation": {
109+
"properties": {
110+
"name": {
111+
"type": "keyword"
112+
},
113+
"attributes": {
114+
"type": "object",
115+
"dynamic": "true"
116+
}
117+
}
118+
},
119+
"targetOperation": {
120+
"properties": {
121+
"name": {
122+
"type": "keyword"
123+
},
124+
"attributes": {
125+
"type": "object",
126+
"dynamic": "true"
127+
}
128+
}
129+
},
130+
"nodeConnectionHash": {
131+
"type": "keyword"
132+
},
133+
"operationConnectionHash": {
134+
"type": "keyword"
135+
},
136+
"timestamp": {
137+
"type": "date",
138+
"format": "strict_date_optional_time||epoch_millis"
139+
}
140+
}
141+
}
142+
}
143+
}

‎data-prepper-plugins/opensearch/src/main/resources/otel-v2-apm-service-map-index-template.json‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
{
2+
"version": 0,
3+
"mappings": {
4+
"date_detection": false,
5+
"dynamic_templates": [
6+
{
7+
"long_group_by_attributes": {
8+
"path_match": "*.groupByAttributes.*",
9+
"match_mapping_type": "long",
10+
"mapping": {
11+
"type": "long"
12+
}
13+
}
14+
},
15+
{
16+
"double_group_by_attributes": {
17+
"path_match": "*.groupByAttributes.*",
18+
"match_mapping_type": "double",
19+
"mapping": {
20+
"type": "double"
21+
}
22+
}
23+
},
24+
{
25+
"string_group_by_attributes": {
26+
"path_match": "*.groupByAttributes.*",
27+
"match_mapping_type": "string",
28+
"mapping": {
29+
"type": "keyword"
30+
}
31+
}
32+
},
33+
{
34+
"long_operation_attributes": {
35+
"path_match": "*.attributes.*",
36+
"match_mapping_type": "long",
37+
"mapping": {
38+
"type": "long"
39+
}
40+
}
41+
},
42+
{
43+
"double_operation_attributes": {
44+
"path_match": "*.attributes.*",
45+
"match_mapping_type": "double",
46+
"mapping": {
47+
"type": "double"
48+
}
49+
}
50+
},
51+
{
52+
"string_operation_attributes": {
53+
"path_match": "*.attributes.*",
54+
"match_mapping_type": "string",
55+
"mapping": {
56+
"type": "keyword"
57+
}
58+
}
59+
}
60+
],
61+
"_source": {
62+
"enabled": true
63+
},
64+
"properties": {
65+
"sourceNode": {
66+
"properties": {
67+
"type": {
68+
"type": "keyword"
69+
},
70+
"keyAttributes": {
71+
"properties": {
72+
"environment": {
73+
"type": "keyword"
74+
},
75+
"name": {
76+
"type": "keyword"
77+
}
78+
}
79+
},
80+
"groupByAttributes": {
81+
"type": "object",
82+
"dynamic": "true"
83+
}
84+
}
85+
},
86+
"targetNode": {
87+
"properties": {
88+
"type": {
89+
"type": "keyword"
90+
},
91+
"keyAttributes": {
92+
"properties": {
93+
"environment": {
94+
"type": "keyword"
95+
},
96+
"name": {
97+
"type": "keyword"
98+
}
99+
}
100+
},
101+
"groupByAttributes": {
102+
"type": "object",
103+
"dynamic": "true"
104+
}
105+
}
106+
},
107+
"sourceOperation": {
108+
"properties": {
109+
"name": {
110+
"type": "keyword"
111+
},
112+
"attributes": {
113+
"type": "object",
114+
"dynamic": "true"
115+
}
116+
}
117+
},
118+
"targetOperation": {
119+
"properties": {
120+
"name": {
121+
"type": "keyword"
122+
},
123+
"attributes": {
124+
"type": "object",
125+
"dynamic": "true"
126+
}
127+
}
128+
},
129+
"nodeConnectionHash": {
130+
"type": "keyword"
131+
},
132+
"operationConnectionHash": {
133+
"type": "keyword"
134+
},
135+
"timestamp": {
136+
"type": "date",
137+
"format": "strict_date_optional_time||epoch_millis"
138+
}
139+
}
140+
}
141+
}

‎data-prepper-plugins/opensearch/src/main/resources/otel-v2-apm-service-map-policy-no-ism-template.json‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
{
2+
"policy": {
3+
"description": "Managing OTel v2 APM service map indices",
4+
"default_state": "current_write_index",
5+
"states": [
6+
{
7+
"name": "current_write_index",
8+
"actions": [
9+
{
10+
"rollover": {
11+
"min_size": "10gb",
12+
"min_index_age": "24h"
13+
}
14+
}
15+
]
16+
}
17+
]
18+
}
19+
}

0 commit comments

Comments
 (0)