Affected Version
Apache Druid 36.0.0
Description
When a Druid data server (Historical or Broker) is under load and rejects a /druid/v2/ POST with HTTP 429 (via LimitRequestsFilter) or HTTP 503 (via Jetty's QoSFilter suspend timeout), the response body is HTML, not JSON, because Druid relies on Jetty's default ErrorHandler. The upstream Druid Broker/Router does not check Content-Type before parsing, so Jackson fails on the very first byte (0x3c = <), and the user-visible error is a generic Unknown exception HTTP 500 instead of a meaningful Too Many Requests / capacity-exceeded error.
Cluster size
Reproduced on a single-server micro-quickstart deployment (1× Coordinator+Overlord, 1× Broker, 1× Historical, 1× Router, 1× MiddleManager).
Configurations in use
Historical runtime.properties (intentionally constrained to force saturation; production-realistic values would be larger but the same code path triggers under sufficient load):
druid.server.http.numThreads=4
druid.server.http.enableRequestLimit=true
druid.server.http.maxIdleTime=PT5S
druid.processing.numThreads=1
druid.processing.numMergeBuffers=2
druid.processing.buffer.sizeBytes=64MiB
Broker runtime.properties:
druid.broker.http.numConnections=20
druid.broker.http.readTimeout=PT2M
druid.broker.http.unusedConnectionTimeout=PT4S
druid.server.http.numThreads=20
Debug logging on Broker / Historical:
org.apache.druid.jetty.RequestLog=DEBUGorg.apache.druid.client.DirectDruidClient=DEBUGorg.apache.druid.client.JsonParserIterator=DEBUGorg.eclipse.jetty.server.handler.ErrorHandler=DEBUGorg.eclipse.jetty.util.thread.QueuedThreadPool=DEBUG
Steps to reproduce
- Apply the constrained Historical config above and restart the Historical.
- Drive a concurrent load against the Router using a small set of queries on the
wikipedia datasource (e.g., 1000 requests at concurrency 200).
- Observe HTTP 500 responses on the client containing
JsonParseException, and matching HTTP 429 in the Historical request log.
A direct probe against the Historical (or any inspection of historical.log) shows the HTML body:
HTTP/1.1 429 Too Many Requests
Content-Type: text/html;charset=ISO-8859-1
<html>
<head><meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 429 Too Many Requests</title></head>
<body><h2>HTTP ERROR 429 Too Many Requests</h2>
...
</body></html>
Error message / stack trace
Broker:
WARN [ForkJoinPool-1-worker-3] org.apache.druid.client.JsonParserIterator -
Query [5710bfb8-ade6-4d13-8211-641c7e2f512f] to host [localhost:8083] interrupted
com.fasterxml.jackson.core.JsonParseException: Invalid type marker byte 0x3c
for expected value token
at [Source: (SequenceInputStream); byte offset: #1]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:724)
...
at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:216)
at org.apache.druid.client.DirectDruidClient.run(DirectDruidClient.java:381)
Client (through Router):
HTTP/1.1 500 Internal Server Error
{"error":"Unknown exception",
"errorClass":"com.fasterxml.jackson.core.JsonParseException",
"host":"localhost:8083",
"errorCode":"legacyQueryException",
"persona":"OPERATOR",
"category":"RUNTIME_FAILURE"}Debugging done
Cross-checked against Druid sources:
server/src/main/java/org/apache/druid/server/initialization/jetty/LimitRequestsFilter.java
emits ((HttpServletResponse) response).sendError(429, "Too Many Requests"),
which dispatches to Jetty's default ErrorHandler — producing HTML.server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java (around line 469)
only customizes the ErrorHandler when druid.server.http.showDetailedJettyErrors=false,
and even then it merely clears RequestDispatcher.ERROR_EXCEPTION and delegates to
super.handle(...). The body rendering remains Jetty's default (HTML when no
Accept: application/json header is sent, which is the case for DirectDruidClient).server/src/main/java/org/apache/druid/client/DirectDruidClient.java does not check
Content-Type. The InputStream is handed straight to
JsonParserIterator.init() (server/src/main/java/org/apache/druid/client/JsonParserIterator.java),
which calls objectMapper.getFactory().createParser(is); jp.nextToken(); —
exactly where Jackson trips over the leading <.- The
QoSFilter registered in
services/src/main/java/org/apache/druid/cli/QueryJettyServerInitializer.java
produces HTTP 503 on suspend timeout via the same code path, so 503 responses can
trigger the identical failure.
docs/configuration/index.md description of druid.server.http.showDetailedJettyErrors
("the JSON response only includes...") is misleading: the default error handler still
performs Accept-header based content negotiation and falls back to HTML for clients
that do not advertise application/json.
Affected Version
Apache Druid 36.0.0
Description
When a Druid data server (Historical or Broker) is under load and rejects a
/druid/v2/POST with HTTP 429 (viaLimitRequestsFilter) or HTTP 503 (via Jetty'sQoSFiltersuspend timeout), the response body is HTML, not JSON, because Druid relies on Jetty's defaultErrorHandler. The upstream Druid Broker/Router does not checkContent-Typebefore parsing, so Jackson fails on the very first byte (0x3c=<), and the user-visible error is a genericUnknown exceptionHTTP 500 instead of a meaningfulToo Many Requests/ capacity-exceeded error.Cluster size
Reproduced on a single-server
micro-quickstartdeployment (1× Coordinator+Overlord, 1× Broker, 1× Historical, 1× Router, 1× MiddleManager).Configurations in use
Historical
runtime.properties(intentionally constrained to force saturation; production-realistic values would be larger but the same code path triggers under sufficient load):Broker
runtime.properties:Debug logging on Broker / Historical:
org.apache.druid.jetty.RequestLog=DEBUGorg.apache.druid.client.DirectDruidClient=DEBUGorg.apache.druid.client.JsonParserIterator=DEBUGorg.eclipse.jetty.server.handler.ErrorHandler=DEBUGorg.eclipse.jetty.util.thread.QueuedThreadPool=DEBUGSteps to reproduce
wikipediadatasource (e.g., 1000 requests at concurrency 200).JsonParseException, and matching HTTP 429 in the Historical request log.A direct probe against the Historical (or any inspection of
historical.log) shows the HTML body:Error message / stack trace
Broker:
Client (through Router):
Debugging done
Cross-checked against Druid sources:
server/src/main/java/org/apache/druid/server/initialization/jetty/LimitRequestsFilter.javaemits
((HttpServletResponse) response).sendError(429, "Too Many Requests"),which dispatches to Jetty's default
ErrorHandler— producing HTML.server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java(around line 469)only customizes the
ErrorHandlerwhendruid.server.http.showDetailedJettyErrors=false,and even then it merely clears
RequestDispatcher.ERROR_EXCEPTIONand delegates tosuper.handle(...). The body rendering remains Jetty's default (HTML when noAccept: application/jsonheader is sent, which is the case forDirectDruidClient).server/src/main/java/org/apache/druid/client/DirectDruidClient.javadoes not checkContent-Type. TheInputStreamis handed straight toJsonParserIterator.init()(server/src/main/java/org/apache/druid/client/JsonParserIterator.java),which calls
objectMapper.getFactory().createParser(is); jp.nextToken();—exactly where Jackson trips over the leading
<.QoSFilterregistered inservices/src/main/java/org/apache/druid/cli/QueryJettyServerInitializer.javaproduces HTTP 503 on suspend timeout via the same code path, so 503 responses can
trigger the identical failure.
docs/configuration/index.mddescription ofdruid.server.http.showDetailedJettyErrors("the JSON response only includes...") is misleading: the default error handler still
performs
Accept-header based content negotiation and falls back to HTML for clientsthat do not advertise
application/json.