Duplicate header check logic issue

[mashhurs]

Please check the official documentation for information on how to use the client, examples for the most frequently used cases, configuration and more.

For help with Elasticsearch, check out our discussion forums.

Please note that this issue tracker is not a help forum and this issue may be closed.

This commit ff9ec8f#r186029959 recently introduced duplicate header check. After the 9.4.0 release, Logstash CI https://app.travis-ci.com/github/logstash-plugins/logstash-filter-elasticsearch/jobs/639388937 started complaining in a way that ES is rejecting due to duplicate headers. of [Content-Type, content-type] (capital and small letters difference), logs:

logstash-1       | === DEBUG: Testing connection to Elasticsearch ===
logstash-1       | Client class: Elasticsearch::Client
logstash-1       | Client transport class: Elastic::Transport::Transport::HTTP::Manticore
logstash-1       | Transport options: {hosts: ["http://elasticsearch:9200"], transport_class: Elastic::Transport::Transport::HTTP::Manticore, transport_options: {headers: {"user-agent" => "logstash/9.4.1 (OS=Linux-6.12.76-linuxkit-aarch64; JVM=Eclipse Adoptium-21.0.11) logstash-filter-elasticsearch/4.3.1", "x-elastic-product-origin" => "logstash-filter-elasticsearch", "content-type" => "application/vnd.elasticsearch+json; compatible-with=9", "accept" => "application/vnd.elasticsearch+json; compatible-with=9", "x-elastic-client-meta" => "es=9.4.0,rb=3.4.5,t=8.5.1,jv=21.0.11,jr=10.0.5.0,mc=0.9.2", "Content-Type" => "application/vnd.elasticsearch+json; compatible-with=9", "User-Agent" => "logstash/9.4.1 (OS=Linux-6.12.76-linuxkit-aarch64; JVM=Eclipse Adoptium-21.0.11) logstash-filter-elasticsearch/4.3.1"}, pool_max: 1000, pool_max_per_route: 100, ssl: {enabled: false}}, ssl: {enabled: false}, retry_on_failure: 0, retry_on_status: [500, 502, 503, 504], logger: nil, tracer: nil, reload_connections: false, delay_on_retry: 0, reload_on_failure: false, randomize_hosts: false, http: {user: nil, password: nil}, enable_meta_header: true}
logstash-1       | Methods with 'header': []
logstash-1       | Transport instance variables: [:@block, :@connections, :@compression, :@hosts, :@manticore, :@request_options, :@last_request_at, :@resurrect_after, :@options, :@reload_after, :@counter_mtx, :@logger, :@retry_on_status, :@tracer, :@serializer, :@state_mutex, :@protocol, :@counter, :@reload_connections, :@sniffer]
logstash-1       | Transport state: #<Elastic::Transport::Transport::HTTP::Manticore:0xa422390 @block=nil, @connections=#<Elastic::Transport::Transport::Connections::Collection:0x53c39950 @connections=[#<Elastic::Transport::Transport::Connections::Connection:0x769d0ef2 @dead=false, @verified=false, @failures=0, @state_mutex=#<Thread::Mutex:0x6d58ed12>, @host={:scheme=>"http", :host=>"elasticsearch", :path=>"", :port=>9200, :protocol=>"http"}, @options={resurrect_timeout: 60}, @connection=#<Manticore::Client:0x300b839c @async_requests=#<Thread::Queue:0x34ad750f>, @keepalive=true, @stubs={}, @pool=#<Java::OrgApacheHttpImplConn::PoolingHttpClientConnectionManager:0x2a5917f6>, @finalizer=#<Manticore::Client::Finalizer:0x558fa64a @_objs=[[#<Java::JavaLangRef::WeakReference:0x72769bf5>, [:close]], [#<Java::JavaLangRef::WeakReference:0x5cc85ffc>, [:shutdown]]]>, @use_cookies=false, @client=#<Java::OrgApacheHttpImplClient::InternalHttpClient:0x7388d94d>, @max_pool_size=1000, @options={headers: {"user-agent" => "logstash/9.4.1 (OS=Linux-6.12.76-linuxkit-aarch64; JVM=Eclipse Adoptium-21.0.11) logstash-filter-elasticsearch/4.3.1", "x-elastic-product-origin" => "logstash-filter-elasticsearch", "content-type" => "application/vnd.elasticsearch+json; compatible-with=9", "accept" => "application/vnd.elasticsearch+json; compatible-with=9", "x-elastic-client-meta" => "es=9.4.0,rb=3.4.5,t=8.5.1,jv=21.0.11,jr=10.0.5.0,mc=0.9.2", "Content-Type" => "application/vnd.elasticsearch+json; compatible-with=9", "User-Agent" => "logstash/9.4.1 (OS=Linux-6.12.76-linuxkit-aarch64; JVM=Eclipse Adoptium-21.0.11) logstash-filter-elasticsearch/4.3.1"}, pool_max: 1000, pool_max_per_route: 100, ssl: {enabled: false}}>>], @selector=#<Elastic::Transport::Transport::Connections::Selector::RoundRobin:0xc53c04e @connections=#<Elastic::Transport::Transport::Connections::Collection:0x53c39950 ...>, @mutex=#<Thread::Mutex:0x18f493d7>, @current=nil>>, @compression=false, @hosts=[{scheme: "http", host: "elasticsearch", path: "", port: 9200, protocol: "http"}], @manticore=#<Manticore::Client:0x300b839c @async_requests=#<Thread::Queue:0x34ad750f>, @keepalive=true, @stubs={}, @pool=#<Java::OrgApacheHttpImplConn::PoolingHttpClientConnectionManager:0x2a5917f6>, @finalizer=#<Manticore::Client::Finalizer:0x558fa64a @_objs=[[#<Java::JavaLangRef::WeakReference:0x72769bf5>, [:close]], [#<Java::JavaLangRef::WeakReference:0x5cc85ffc>, [:shutdown]]]>, @use_cookies=false, @client=#<Java::OrgApacheHttpImplClient::InternalHttpClient:0x7388d94d>, @max_pool_size=1000, @options={headers: {"user-agent" => "logstash/9.4.1 (OS=Linux-6.12.76-linuxkit-aarch64; JVM=Eclipse Adoptium-21.0.11) logstash-filter-elasticsearch/4.3.1", "x-elastic-product-origin" => "logstash-filter-elasticsearch", "content-type" => "application/vnd.elasticsearch+json; compatible-with=9", "accept" => "application/vnd.elasticsearch+json; compatible-with=9", "x-elastic-client-meta" => "es=9.4.0,rb=3.4.5,t=8.5.1,jv=21.0.11,jr=10.0.5.0,mc=0.9.2", "Content-Type" => "application/vnd.elasticsearch+json; compatible-with=9", "User-Agent" => "logstash/9.4.1 (OS=Linux-6.12.76-linuxkit-aarch64; JVM=Eclipse Adoptium-21.0.11) logstash-filter-elasticsearch/4.3.1"}, pool_max: 1000, pool_max_per_route: 100, ssl: {enabled: false}}>, @request_options={headers: {"user-agent" => "logstash/9.4.1 (OS=Linux-6.12.76-linuxkit-aarch64; JVM=Eclipse Adoptium-21.0.11) logstash-filter-elasticsearch/4.3.1", "x-elastic-product-origin" => "logstash-filter-elasticsearch", "content-type" => "application/vnd.elasticsearch+json; compatible-with=9", "accept" => "application/vnd.elasticsearch+json; compatible-with=9", "x-elastic-client-meta" => "es=9.4.0,rb=3.4.5,t=8.5.1,jv=21.0.11,jr=10.0.5.0,mc=0.9.2", "Content-Type" => "application/vnd.elasticsearch+json; compatible-with=9", "User-Agent" => "logstash/9.4.1 (OS=Linux-6.12.76-linuxkit-aarch64; JVM=Eclipse Adoptium-21.0.11) logstash-filter-elasticsearch/4.3.1"}}, @last_request_at=2026-05-20 23:16:58.047224 UTC, @resurrect_after=60, @options={hosts: ["http://elasticsearch:9200"], transport_class: Elastic::Transport::Transport::HTTP::Manticore, transport_options: {headers: {"user-agent" => "logstash/9.4.1 (OS=Linux-6.12.76-linuxkit-aarch64; JVM=Eclipse Adoptium-21.0.11) logstash-filter-elasticsearch/4.3.1", "x-elastic-product-origin" => "logstash-filter-elasticsearch", "content-type" => "application/vnd.elasticsearch+json; compatible-with=9", "accept" => "application/vnd.elasticsearch+json; compatible-with=9", "x-elastic-client-meta" => "es=9.4.0,rb=3.4.5,t=8.5.1,jv=21.0.11,jr=10.0.5.0,mc=0.9.2", "Content-Type" => "application/vnd.elasticsearch+json; compatible-with=9", "User-Agent" => "logstash/9.4.1 (OS=Linux-6.12.76-linuxkit-aarch64; JVM=Eclipse Adoptium-21.0.11) logstash-filter-elasticsearch/4.3.1"}, pool_max: 1000, pool_max_per_route: 100, ssl: {enabled: false}}, ssl: {enabled: false}, retry_on_failure: 0, retry_on_status: [500, 502, 503, 504], logger: nil, tracer: nil, reload_connections: false, delay_on_retry: 0, reload_on_failure: false, randomize_hosts: false, http: {user: nil, password: nil}, enable_meta_header: true}, @reload_after=10000, @counter_mtx=#<Thread::Mutex:0x64abb917>, @logger=nil, @retry_on_status=[500, 502, 503, 504], @tracer=nil, @serializer=#<Elastic::Transport::Transport::Serializer::MultiJson:0x68596992 @transport=#<Elastic::Transport::Transport::HTTP::Manticore:0xa422390 ...>>, @state_mutex=#<Thread::Mutex:0x5f0c45ae>, @protocol="http", @counter=0, @reload_connections=false, @sniffer=#<Elastic::Transport::Transport::Sniffer:0x66d62c59 @transport=#<Elastic::Transport::Transport::HTTP::Manticore:0xa422390 ...>, @timeout=1>>
....

logstash-1       |   1) LogStash::Filters::Elasticsearch run ES|QL queries with simple FROM query with LIMIT processes the event
logstash-1       |      Failure/Error: @client.info
logstash-1       |      
logstash-1       |      Elastic::Transport::Transport::Errors::BadRequest:
logstash-1       |        [400] {"error":{"root_cause":[{"type":"media_type_header_exception","reason":"Invalid media-type value on headers [Content-Type]"}],"type":"media_type_header_exception","reason":"Invalid media-type value on headers [Content-Type]","caused_by":{"type":"illegal_argument_exception","reason":"Incorrect header [Content-Type]. Only one value should be provided"}},"status":400}
logstash-1       |      Shared Example Group: "ESQL query execution" called from ./spec/filters/integration/elasticsearch_esql_spec.rb:112
logstash-1       |      # ./lib/logstash/filters/elasticsearch/client.rb:66:in 'info'
logstash-1       |      # ./lib/logstash/filters/elasticsearch/client.rb:70:in 'es_version'
logstash-1       |      # ./lib/logstash/filters/elasticsearch.rb:526:in 'validate_es_for_esql_support!'
logstash-1       |      # ./lib/logstash/filters/elasticsearch.rb:210:in 'register'
logstash-1       |      # ./spec/filters/integration/elasticsearch_esql_spec.rb:94:in 'block in <main>'

As per my analysis, logstash-filter-elasticsearch plugin initializes transport client where Content-Type (case sensitive, capital letters) header will be set, srouces:

• https://github.com/logstash-plugins/logstash-filter-elasticsearch/blob/962681d623affdd825b9c24611e6dcf7d131867c/lib/logstash/filters/elasticsearch/client.rb#L127
• https://github.com/elastic/elastic-transport-ruby/blob/2a70012f6f57afcacf705fbf08f4c755de9a8dc7/lib/elastic/transport/transport/base.rb#L398
• https://github.com/elastic/elastic-transport-ruby/blob/2a70012f6f57afcacf705fbf08f4c755de9a8dc7/lib/elastic/transport/transport/http/manticore.rb#L172

Then, this set_content_type! method adds content-type. It looks like we are using case sensitivity logic here.

It looks like using i flag will be a solution here:

unless user_headers&.keys&.detect { |h| h =~ /content-?_?type/i }