Otel apm OpenSearch template#6435
Open
san81 wants to merge 3 commits into
Open
Conversation
Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>
Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>
san81
requested review from
KarstenSchnitter,
chenqi0805,
dinujoh,
dlvenable,
engechas,
graytaylor0,
kkondaka,
oeyh,
sb2k16 and
srikanthjg
as code owners
Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>
vamsimanohar
added a commit
to vamsimanohar/data-prepper
that referenced
this pull request
Feb 18, 2026
…y emission Refactors the APM service map processor to use a unified model structure replacing Service/ServiceConnection/ServiceOperationDetail with Node/NodeOperationDetail. Implements CLIENT-span-primary emission algorithm to eliminate duplicate events. Model Changes: - Node.java: Unified service entity with type field for future extensibility - NodeOperationDetail: Single entity for both topology and operation events - Operation: Simplified to name + attributes structure - CLIENT-primary algorithm: CLIENT spans emit complete events using decoration data OpenSearch Integration: - Added OTEL_APM_SERVICE_MAP index type following PR opensearch-project#6435 patterns - Index template with dynamic mappings for groupByAttributes and operation attributes - ISM policies for automated rollover (10gb/24h) - Updated IndexManagerFactory, IndexConstants, IndexConfiguration Configuration: - Changed eventType from SERVICE_MAP_V2 to SERVICE_MAP - Updated README with proper index_type: otel-v2-apm-service-map configuration - Coexists with legacy service_map processor using different index patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
kkondaka
pushed a commit
that referenced
this pull request
Feb 19, 2026
…6536) * Refactor to unified Node/NodeOperationDetail with SERVICE_MAP_V2 events - Replace Service/ServiceConnection/ServiceOperationDetail with unified Node/NodeOperationDetail model - Node adds type field for future entity types (database, queue, etc) - Operation simplified to name + attributes - NodeOperationDetail single entity with dual hash fields (nodeConnectionHash, operationConnectionHash) - CLIENT-span-primary emission: CLIENT spans emit full NodeOperationDetail, leaf SERVER spans for services with no outgoing calls - Update eventType to SERVICE_MAP_V2 for new index pattern otel-v2-apm-service-map Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Refactor model to unified Node/NodeOperationDetail with CLIENT-primary emission Refactors the APM service map processor to use a unified model structure replacing Service/ServiceConnection/ServiceOperationDetail with Node/NodeOperationDetail. Implements CLIENT-span-primary emission algorithm to eliminate duplicate events. Model Changes: - Node.java: Unified service entity with type field for future extensibility - NodeOperationDetail: Single entity for both topology and operation events - Operation: Simplified to name + attributes structure - CLIENT-primary algorithm: CLIENT spans emit complete events using decoration data OpenSearch Integration: - Added OTEL_APM_SERVICE_MAP index type following PR #6435 patterns - Index template with dynamic mappings for groupByAttributes and operation attributes - ISM policies for automated rollover (10gb/24h) - Updated IndexManagerFactory, IndexConstants, IndexConfiguration Configuration: - Changed eventType from SERVICE_MAP_V2 to SERVICE_MAP - Updated README with proper index_type: otel-v2-apm-service-map configuration - Coexists with legacy service_map processor using different index patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Update README with algorithm docs and remove standalone MD files - Merged NodeOperationDetail algorithm documentation into processor README - Updated output events section to reflect unified Node/NodeOperationDetail model - Added detailed metrics generation documentation - Removed standalone node-operation-detail-algorithm.md from tracking - Added algorithm design MD files to .gitignore Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Fix index-template version to use composable template format The index-template/ copy needs the "template" wrapper for the modern OpenSearch composable index template API, while the root copy uses the V1 legacy format. Previously both were identical (V1 format). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Clean up .gitignore and optimize index templates - Remove local-only entries from .gitignore (algorithm MDs, claude.md) - Collapse 12 dynamic templates to 6 using wildcard path matching - Remove ignore_above restrictions from all keyword fields - Fix index-template/ copy to use composable template format Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> * Remove eventType from index templates and fix test issues - Remove eventType field from both index templates (V1 and composable) eventType is pipeline routing metadata, not part of NodeOperationDetail model - Remove broken testWindowProcessingWithInterruptedException test Test doesn't work with @DataPrepperPluginTest annotation due to override restrictions Addresses review feedback from @kkondaka Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> --------- Signed-off-by: Vamsi Manohar <reddyvam@amazon.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Introducing Otel APM related
That will be used when the index type is
index_type: "otel-v2-apm-service-map"Required as pre work for otel-apm-service-map processor that we will be added in the next PRs
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.