Passing http request headers in the event metadata for http source by divbok · Pull Request #6671 · opensearch-project/data-prepper

divbok · 2026-03-24T20:34:37Z

Description

Adds a new metadata_headers configuration option to the HTTP source plugin that allows users to extract HTTP request headers and attach them as event metadata. This enables downstream processors and sinks to route, filter, or enrich events based on HTTP headers.

How it works

Whitelist mode: Configure specific header names to extract:
```
metadata_headers: ["X-Tenant-Id", "X-Region"]
```
Disabled (default): When metadata_headers is not set or null, no headers are extracted.

Header key normalization

All header keys are lowercased in event metadata regardless of how they were sent by the client or configured in the YAML. This is because Armeria (the underlying HTTP server) normalizes all header names to lowercase per the HTTP/2 specification. For example, a header sent as X-Tenant-Id is stored in metadata as x-tenant-id.
Sensitive header filtering
A hardcoded blocklist of sensitive headers is always filtered out, regardless of configuration. This includes authorization, cookie, x-api-key, x-amz-security-token, x-amz-credential, and others.
Multi-value headers
Headers with multiple values (e.g., X-Forwarded-For appearing twice) are stored as a List. Single-value headers are stored as a plain String.
Current limitations
This feature is currently only supported for the in-memory buffer (BlockingBuffer). The byte buffer / Kafka buffer path does not propagate metadata headers yet.

Example pipeline

log-pipeline:
source:
  http:
    port: 2021
    path: "/log/ingest"
    metadata_headers: ["X-Tenant-Id", "X-Region"]
route:
  - tenant_alpha: 'getMetadata("headers/x-tenant-id") == "team-alpha"'
  - us_west: 'getMetadata("headers/x-region") == "us-west-2"'
sink:
  - file:
      path: "/tmp/alpha.log"
      routes: ["tenant_alpha"]
  - file:
      path: "/tmp/us_west.log"
      routes: ["us_west"]

curl -X POST http://localhost:2021/log/ingest \
  -H "Content-Type: application/json" \
  -H "X-Tenant-Id: team-alpha" \
  -H "X-Region: us-west-2" \
  -d '[{"message": "hello"}]'

Issues Resolved

Resolves #6239

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dlvenable

Thanks @divbok for this contribution!

dlvenable · 2026-03-25T18:35:41Z

+            return Collections.emptyMap();
+        }
+
+        final boolean includeAll = metadataHeaders.size() == 1 && "*".equals(metadataHeaders.get(0));


I'm not sure about supporting * without supporting regex in general. I think if we wanted this, we should add metadata_headers_regex instead. This would be consistent with other processors that support literals and regex.

I will add the regex config in a follow up PR

dlvenable · 2026-03-25T18:36:06Z

+        final Map<String, Object> headers = new HashMap<>();
+        for (String headerName : headerNames) {
+            if (isSensitiveHeader(headerName)) {
+                LOG.warn("Skipping sensitive header '{}' from metadata_headers configuration", headerName);


I think we don't need any logging here. Maybe a TRACE if you want.

dlvenable · 2026-03-25T18:38:31Z

+        return headers;
+    }
+
+    static boolean isSensitiveHeader(final String headerName) {


Put this in its own class and test it there.

dlvenable · 2026-03-25T18:39:17Z

+    public void processRequestAttachesHeadersToEventMetadata() throws Exception {
+        final String tenantId = UUID.randomUUID().toString();
+        final String region = UUID.randomUUID().toString();
+        BlockingBuffer<Record<Log>> blockingBuffer = new BlockingBuffer<>(TEST_BUFFER_CAPACITY, 8, "test-pipeline");


Use a mocked Buffer instead. Then you don't need to use read. Verify that it is called and use an ArgumentCaptor.

dlvenable · 2026-03-25T18:41:01Z

        return HttpResponse.of(HttpStatus.OK);
    }

+    private Map<String, Object> extractHeaders(final AggregatedHttpRequest aggregatedHttpRequest) {


I tend to think that this should go into it's own class. You can unit test this more easily. Then verify an interaction with the mock in the unit tests for LogHTTPService.

dlvenable · 2026-03-25T18:42:31Z

-        return new Record<>(log);
+                .getThis();
+        if (!headerAttributes.isEmpty()) {
+            builder.withEventMetadataAttributes(headerAttributes);


This approach will put all the headers as metadata in the root of the metadata. I wonder if it would make more sense to have these under headers in the metadata. Do we support nested metadata? If not, then I guess this is the approach we should take.

github-actions · 2026-03-26T16:14:06Z

✅ License Header Check Passed

All newly added files have proper license headers. Great work! 🎉

oeyh · 2026-04-02T04:26:24Z

    private PluginModel codec;

+    @JsonProperty("metadata_headers")
+    private List<String> metadataHeaders;


Suggest using empty list as default. Otherwise it's null by default and will need null check wherever it's used.

Suggested change

private List<String> metadataHeaders;

private List<String> metadataHeaders = Collections.emptyList();

oeyh · 2026-04-02T04:26:29Z


    HttpResponse processRequest(final AggregatedHttpRequest aggregatedHttpRequest) throws Exception {
        final HttpData content = aggregatedHttpRequest.content();
+        final Map<String, Object> extractedHeaders = httpHeaderExtractor.extractHeaders(aggregatedHttpRequest);


All records in a batch share the exact same extractedHeaders map instance in their metadata. If a processor downstream somehow mutates the map, it will affect other records in the batch, might cause a surprise. We can wrap extractedHeaders in Collections.unmodifiableMap() to prevents accidental in-place mutation.

dlvenable · 2026-04-07T15:20:50Z

        assertEquals(HTTPSourceConfig.DEFAULT_LOG_INGEST_URI, sourceConfig.getPath());
        assertEquals(HTTPSourceConfig.DEFAULT_PORT, sourceConfig.getDefaultPort());
        assertEquals(HTTPSourceConfig.DEFAULT_LOG_INGEST_URI, sourceConfig.getDefaultPath());
+        assertNull(sourceConfig.getMetadataHeaders());


I believe this is causing the failure:

HTTPSourceConfigTest > testDefault() FAILED org.opentest4j.AssertionFailedError: expected: <null> but was: <[]>

This is now an empty collection by default which is good.

dlvenable · 2026-04-07T15:22:43Z

-
-        return new Record<>(log);
+                .getThis();
+        if (!headerAttributes.isEmpty()) {


How can we consolidate this block with line 138? We seem to have a split in how we create records that could result in discrepancies.

dlvenable · 2026-04-07T15:23:00Z

 import java.util.stream.Collectors;

-
-/*


Did you mean to remove this?

Added it back.

dlvenable · 2026-04-07T15:25:07Z

+                          final Buffer<Record<Log>> buffer,
+                          final PluginMetrics pluginMetrics,
+                          final InputCodec codec) {
+        this(bufferWriteTimeoutInMillis, buffer, pluginMetrics, codec, new HttpHeaderExtractor(null));


You should use an empty collection for consistency. new HttpHeaderExtractor() or new HttpHeaderExtractor(Collections.emptySet())

dlvenable · 2026-04-07T15:25:29Z

+    private final List<String> metadataHeaders;
+
+    public HttpHeaderExtractor(final List<String> metadataHeaders) {
+        this.metadataHeaders = metadataHeaders;


Require this to be non-null.

dlvenable · 2026-04-07T15:25:44Z

+    }
+
+    public Map<String, Object> extractHeaders(final AggregatedHttpRequest aggregatedHttpRequest) {
+        if (metadataHeaders == null || metadataHeaders.isEmpty()) {


Remove metadataHeaders == null after other changes I suggested.

dlvenable · 2026-04-07T15:26:00Z

+
+    private final List<String> metadataHeaders;
+
+    public HttpHeaderExtractor(final List<String> metadataHeaders) {


This can be Collection<String> You don't require ordering.

Signed-off-by: Divyansh Bokadia <dbokadia@amazon.com>

dlvenable

Thank you for the contribution @divbok !

divbok requested review from KarstenSchnitter, dinujoh, dlvenable, engechas, graytaylor0, kkondaka, oeyh, san81, sb2k16 and srikanthjg as code owners March 24, 2026 20:34

divbok changed the title ~~Passing http request headers as metadata in the event for http source~~ Passing http request headers in the event metadata for http source Mar 24, 2026

dlvenable requested changes Mar 25, 2026

View reviewed changes

divbok force-pushed the main branch from ae56f0e to b50c01e Compare March 26, 2026 16:12

divbok force-pushed the main branch 2 times, most recently from 79c50d1 to 788c85b Compare March 26, 2026 16:18

oeyh reviewed Apr 2, 2026

View reviewed changes

divbok force-pushed the main branch from 788c85b to b129198 Compare April 7, 2026 14:58

dlvenable requested changes Apr 7, 2026

View reviewed changes

Passing http request headers as metadata in the event for http source

fe38250

Signed-off-by: Divyansh Bokadia <dbokadia@amazon.com>

divbok force-pushed the main branch from b129198 to fe38250 Compare April 7, 2026 16:30

oeyh approved these changes Apr 7, 2026

View reviewed changes

dlvenable approved these changes Apr 8, 2026

View reviewed changes

dlvenable merged commit 1ac64aa into opensearch-project:main Apr 8, 2026
71 of 72 checks passed

dlvenable mentioned this pull request Apr 29, 2026

Adds Divyansh Bokadia (divbok) to the maintainers group. #6789

Merged

4 tasks

	private List<String> metadataHeaders;
	private List<String> metadataHeaders = Collections.emptyList();


		private final List<String> metadataHeaders;

		public HttpHeaderExtractor(final List<String> metadataHeaders) {

Conversation

divbok commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How it works

Header key normalization

Example pipeline

Issues Resolved

Check List

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ License Header Check Passed

Uh oh!

oeyh Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

divbok commented Mar 24, 2026 •

edited

Loading

github-actions Bot commented Mar 26, 2026 •

edited

Loading

oeyh Apr 2, 2026 •

edited

Loading