Support spath with dynamic fields

[ykmr1224]

Description

• Extend spath command to support dynamic fields, where it no longer requires field selection.
  • It has limitations as it does not fully support field ordering of dynamic fields.
• This change is for step2-1 in [RFC] Support Schema-on-Read in PPL #4984
• The fields not explicitly referred in the query would be stored in _MAP(dynamic fields), and expanded when result is organized.
• It does not support partial wildcard like prefix*, *suffix, and raise error in case those are used.
• When result contains dynamic fields, OpenSearchExecutionEngine puts dynamic fields at the end of the fields when organizing result. (This is temporal behavior until field ordering is implemented)
  • To align this behavior, it raises error when fields command specify * in the middle of field list.
• Following commands will be adopted in later PR (command with multiple input requires change) : appendcol, multisearch, lookup

Related Issues

#4984

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • Auto extraction for SPATH that appends conflicting fields and expands dynamic fields into result columns; improved join behavior with configurable overwrite semantics

• Documentation

  • SPATH docs renamed/expanded to “Auto Extraction”; adds system column guidance, examples, limitations and performance notes

• Bug Fixes

  • Stronger wildcard/partial-wildcard validation, fillnull safeguards, and more consistent field-resolution across project/aggregation/join flows

• Tests

  • Large expansion of SPATH and dynamic-field unit and integration tests, including explain and join scenarios

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Walkthrough

Adds end-to-end dynamic-fields (auto extraction): SPATH/field-resolution updates, Calcite planning/merge changes to emit a runtime _MAP, utilities to align/merge dynamic maps during joins, runtime expansion of _MAP into STRING columns, many tests, and docs updates.

Changes

Cohort / File(s)	Summary
Debug util \common/src/main/java/org/opensearch/sql/common/utils/DebugUtils.java``	Removed Log4j logger; debug output now uses System.out.println.
Field resolution API \core/src/main/java/org/opensearch/sql/ast/analysis/FieldResolutionResult.java``	Switched constructors/inputs to Collection<String>, added accessors (getRegularFieldsUnmodifiable, hasWildcards, hasPartialWildcards, hasRegularFields), set ops (exclude, or, and), and helpers.
Field resolution visitor & SPATH \core/src/main/java/org/opensearch/sql/ast/analysis/FieldResolutionVisitor.java`, `ppl/src/test/java/org/opensearch/sql/ppl/parser/FieldResolutionVisitorTest.java``	Introduced ALL_FIELDS sentinel, changed requirement propagation, partial-wildcard checks, added ALL_FIELDS exclusion helpers, SPATH validations, and many SPATH-related tests.
Calcite planning & dynamic-fields helper \core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java`, `core/src/main/java/org/opensearch/sql/calcite/DynamicFieldsHelper.java``	Injects dynamic _MAP handling into planning (project/append/join), enforces wildcard placement when dynamic fields present, and provides utilities to detect/align static vs dynamic fields and build MAP_FROM_ARRAYS/MAP_REMOVE projections.
Join merge & overwrite semantics \core/src/main/java/org/opensearch/sql/calcite/utils/JoinAndLookupUtils.java`, `core/src/main/java/org/opensearch/sql/ast/tree/Join.java``	Added OverwriteMode enum and mergeDynamicFieldsAsNeeded to merge _MAP fields after joins; added Join.isOverwrite() accessor and updated rename logic to include DYNAMIC_FIELDS_MAP.
Runtime result expansion \core/src/main/java/org/opensearch/sql/calcite/utils/DynamicFieldsResultProcessor.java`, `opensearch/src/main/java/org/opensearch/sql/opensearch/executor/OpenSearchExecutionEngine.java``	New processor expands _MAP into concrete STRING columns and OpenSearchExecutionEngine now post-processes QueryResponse with it.
Constants, Rex/func helpers & functions \core/src/main/java/org/opensearch/sql/calcite/plan/DynamicFieldsConstants.java`, `core/src/main/java/org/opensearch/sql/calcite/ExtendedRexBuilder.java`, `core/src/main/java/org/opensearch/sql/expression/function/BuiltinFunctionName.java`, `core/src/main/java/org/opensearch/sql/expression/function/PPLFuncImpTable.java``	Added DYNAMIC_FIELDS_MAP constant, ExtendedRexBuilder helpers (itemCall, makeCall), registered MAP_FROM_ARRAYS and operator mapping.
Tests & expected plans \core/src/test/... `, `integ-test/src/test/... `, `integ-test/src/test/resources/expectedOutput/... ``	Many new/updated unit and integration tests covering SPATH, dynamic fields, joins, explains, and DynamicFieldsResultProcessor; new expected explain YAMLs.
Docs & minor tests \docs/user/ppl/cmd/spath.md`, `docs/user/ppl/general/identifiers.md`, `integ-test/src/test/java/org/opensearch/sql/util/MatcherUtils.java``	Renamed sections to "Auto Extraction", documented _MAP system column and wildcard exclusions, improved assertion messages.
Test infra & helpers \integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLSpathTestBase.java`, `integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLSpathWithJoinIT.java`, `integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLSpathCommandIT.java``	New test base and many integration tests for JSON-backed SPATH scenarios, joins, overwrite behaviors, and dynamic-field interactions.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant Client as User Query
    participant FRV as FieldResolutionVisitor
    participant CRV as CalciteRelNodeVisitor
    participant JoinUtil as JoinAndLookupUtils
    participant Exec as OpenSearchExecutionEngine
    participant RP as DynamicFieldsResultProcessor

    Client->>FRV: parse & analyze (detect ALL_FIELDS / wildcards)
    FRV->>CRV: emit Calcite plan with field requirements and `_MAP` markers
    CRV->>JoinUtil: on join -> adjust inputs and request dynamic-map merge
    JoinUtil-->>CRV: merged RelNode with `_MAP` (order per OverwriteMode)
    CRV->>Exec: execute physical plan -> QueryResponse (may include `_MAP`)
    Exec->>RP: expandDynamicFields(QueryResponse)
    RP-->>Client: return expanded QueryResponse (concrete STRING columns)

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• Pushdown join with max=n option to TopHits aggregation #4929 — overlaps CalciteRelNodeVisitor (join handling) and dynamic-field input alignment.
• Introduce logical dedup operators for PPL #5014 — related Calcite planning infra changes and metadata-field handling.

Suggested reviewers

• penghuo
• anirudha
• kavithacm
• derek-ho
• joshuali925
• GumpacG
• Swiddis
• qianheng-aws
• LantaoJin

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 21.21% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title 'Support spath with dynamic fields' clearly and concisely describes the main change—extending the spath command to support dynamic fields, which is the primary feature added throughout the changeset.
Description check	✅ Passed	The PR description is directly related to the changeset, detailing the spath dynamic fields feature, its limitations, scope (step 2-1 of issue #4984), and implementation specifics including the _MAP field, error handling, and future work.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[dai-chen]

Just one concern: I feel it's unclear what future PPL command developers need to understand or implement when adding commands that interact with dynamic fields, since current _MAP implementation seems leaks across multiple layers (field resolution, RelNode visitor, join handling, result post-processing, etc). We can consider improving document or consolidating the logic later if this is valid concern. Thanks!

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-5058-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 633d760a5d0eb5a516232037ef79cb1cdf3af75c
# Push it to GitHub
git push --set-upstream origin backport/backport-5058-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-5058-to-2.19-dev.