Skip to content

Fix search profile fetch schema to match server serialization#1147

Open
minseoky wants to merge 3 commits into
opensearch-project:mainfrom
minseoky:fix/shard-profile-fetch-array
Open

Fix search profile fetch schema to match server serialization#1147
minseoky wants to merge 3 commits into
opensearch-project:mainfrom
minseoky:fix/shard-profile-fetch-array

Conversation

@minseoky

@minseoky minseoky commented Jun 11, 2026

Copy link
Copy Markdown

Description

The profile section of the search response is incorrectly specified, causing generated clients to crash when deserializing responses with profile: true
(reported in opensearch-project/opensearch-java#1965).

1. ShardProfile.fetch is an array, not a single object (the reported bug)

The server always serializes fetch as an array — see FetchProfileShardResult#toXContent
(builder.startArray(FETCH)). The spec defined it as a single FetchProfile object, so the generated Java client fails with:

UnexpectedJsonEventException: Unexpected JSON event 'START_ARRAY' instead of '[START_OBJECT, KEY_NAME]'

A spec test is required for this change, and the test framework validates the entire response payload against the spec (any field returned by the server but not declared in the spec is an error).
As a result, the profile: true test cannot pass with the fetch fix alone — it surfaced two more discrepancies in
the same response, which are fixed in this PR as well:

2. ShardProfile is missing the network time fields

The server unconditionally writes inbound_network_time_in_millis and outbound_network_time_in_millis for every shard profile — see SearchProfileShardResults#toXContent
(present since OpenSearch 2.0.0, opensearch-project/OpenSearch#1360).

3. FetchProfileBreakdown does not match the server's FetchTimingType

Rewrote the schema to match FetchTimingType
(introduced with fetch profiling in OpenSearch 3.2.0, opensearch-project/OpenSearch#18664):

  • Added missing fields: build_sub_phase_processors, create_stored_fields_visitor, get_next_reader, set_next_reader (+ their _count counterparts)
  • Removed next_reader / next_reader_count, which the server never emits
  • Changed format: int32 to int64: breakdown values are Java longs (AbstractProfileBreakdown#toBreakdownMap returns Map<String, Long>), and the timing values are nanoseconds, which canexceed int32's ~2.1s range. This also matches the sibling QueryBreakdown / AggregationBreakdown schemas, which already use int64. Note that Fix type and schema mismatches across the spec #1110 touches the same schema with the same int64 direction, so a small merge conflict is possible depending on ordering.

Testing

Added tests/default/_core/search/profile.yaml (the second and third fixes above are prerequisites for this test to pass) and verified against a local OpenSearch 3.7.0 cluster:

  • Before this change: RESPONSE PAYLOAD SCHEMA fails with the mismatches above
  • After: PASSED, with no regressions across the rest of the _core/search test suite
  • npm run lint:spec passes

Since this spec is the source of truth for all generated clients, the fix benefits the other language clients as well.

Issues Resolved

Addresses opensearch-project/opensearch-java#1965 — the Java client fix will follow automatically via the weekly code generation workflow once this spec change is released.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

- Change ShardProfile's fetch from a single FetchProfile object to an
  array, matching FetchProfileShardResult#toXContent which always
  serializes it with startArray
- Add missing inbound_network_time_in_millis and
  outbound_network_time_in_millis fields to ShardProfile
- Rewrite FetchProfileBreakdown to match the server's FetchTimingType
  enum: add missing timing fields, remove next_reader which the server
  never emits, and use int64 since breakdown values are Java longs
- Add a spec test for search with profile enabled

Signed-off-by: Minseok Choi <minseok184@gmail.com>
Signed-off-by: Minseok Choi <minseok184@gmail.com>
- fetch: fetch phase profiling was introduced in OpenSearch 3.2.0
  (opensearch-project/OpenSearch#18664)
- inbound/outbound_network_time_in_millis: added in OpenSearch 2.0.0
  (opensearch-project/OpenSearch#1360)

Signed-off-by: Minseok Choi <minseok184@gmail.com>
@minseoky

Copy link
Copy Markdown
Author

Note on delivery: the weekly code generation in opensearch-java is currently failing against the latest spec (UnsupportedOperationException: Can not get type name for allOf on the aggregate buckets schemas — see the discussion in opensearch-project/opensearch-java#1957), so this fix won't reach the Java client until that is resolved.

The schemas in this PR are not affected by that limitation: regenerating with only this change applied on top of the spec snapshot committed in opensearch-java succeeds, changing just ShardProfile.java and FetchProfileBreakdown.java. I also verified end-to-end against a real OpenSearch 3.7.0 cluster — the released client (3.9.0) throws UnexpectedJsonEventException on profile: true, while the client regenerated from this spec deserializes the response correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant