Skip to content

fix: expand necessary OpenSearch permissions for data prepper#6649

Merged
dlvenable merged 1 commit into
opensearch-project:mainfrom
JongminChung:fix/gh-6009
Apr 2, 2026
Merged

fix: expand necessary OpenSearch permissions for data prepper#6649
dlvenable merged 1 commit into
opensearch-project:mainfrom
JongminChung:fix/gh-6009

Conversation

@JongminChung

@JongminChung JongminChung commented Mar 18, 2026

Copy link
Copy Markdown
Contributor

Description

PR expands OpenSearch permissions documentation for Data Prepper.

Additional permissions were compiled by trial and error based on logs from Data Prepper and OpenSearch.

An error occurred in the process of inquiring the composable index templates through index_template, adding the corresponding permissions.

Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,271 [service-map-pipeline-sink-worker-6-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,271 [service-map-pipeline-sink-worker-6-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,271 [service-map-pipeline-sink-worker-6-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the cert provided in the config.
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,351 [traces-raw-pipeline-sink-worker-8-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: Forbidden access
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,372 [service-map-pipeline-sink-worker-6-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: Forbidden access
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,431 [otel-logs-pipeline-sink-worker-10-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,431 [otel-logs-pipeline-sink-worker-10-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,431 [otel-logs-pipeline-sink-worker-10-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the cert provided in the config.
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,511 [otel-logs-pipeline-sink-worker-10-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: Forbidden access
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,558 [traces-raw-pipeline-sink-worker-8-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,558 [traces-raw-pipeline-sink-worker-8-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,558 [traces-raw-pipeline-sink-worker-8-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the cert provided in the config.
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,603 [service-map-pipeline-sink-worker-6-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,603 [service-map-pipeline-sink-worker-6-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,609 [service-map-pipeline-sink-worker-6-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the cert provided in the config.
Mar 18 16:07:40 org-dev-gcldep-ctl-1.ad.gabiacloud.internal systemd-gcloud-opensearch-dataprepper[1793440]: 2026-03-18T16:07:40,644 [traces-raw-pipeline-sink-worker-8-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: Forbidden access
otel-logs-pipeline:
  source:
    pipeline: { name: "otlp-pipeline" }
  sink:
    - opensearch:
        hosts: [ "$HOST" ]
        username: $USERNAME
        password: $RANDOM_PWD
        # index_type: log-analytics-plain (before: https://github.com/opensearch-project/data-prepper/pull/6647)
        template_type: index-template
        template_file: /usr/share/data-prepper/index-template/logs-otel-v1-index-standard-template.json
        ism_policy_file: /usr/share/data-prepper/index-template/logs-policy-with-ism-template.json
        index_type: custom
        index: logs-otel-v1

Issues Resolved

Resolves #6646

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Jongmin Chung <chungjm0711@gmail.com>
@dlvenable dlvenable merged commit 76a9a2b into opensearch-project:main Apr 2, 2026
91 of 97 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] logs-otel-v1 has error syntax

2 participants