[Pluggable Dataformat] Adding support for FieldTypeCapabilities#21733
Conversation
PR Reviewer Guide 🔍(Review updated until commit 3295038)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 3295038 Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit e734c19
Suggestions up to commit b536cfb
Suggestions up to commit 9d26779
Suggestions up to commit b4f6c48
|
|
❌ Gradle check result for 3bde29b: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Persistent review updated to latest commit 022ce52 |
|
❌ Gradle check result for 022ce52: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit bb7cd74. 'Diff too large, requires skip by maintainers after manual review' Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
Bukhtawar
left a comment
There was a problem hiding this comment.
Thanks for the changes, overall lgtm
Signed-off-by: Sagar Darji <darsaga@amazon.com>
Signed-off-by: Sagar Darji <darsaga@amazon.com>
…es; add tests Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
…mats Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
…uggable data formats Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
|
Persistent review updated to latest commit b536cfb |
|
❌ Gradle check result for b536cfb: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Persistent review updated to latest commit e734c19 |
Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
|
Persistent review updated to latest commit 3295038 |
…search-project#21733) This PR introduces a capability-based routing system for field types in the Pluggable Dataformat architecture. Instead of each field mapper implementing a parseCreateFieldForPluggableFormat method (duplicating parsing logic per format), field types now declare their capabilities (e.g., POINT_RANGE, COLUMNAR_STORAGE, FULL_TEXT_SEARCH), and the data format infrastructure uses these declarations to determine which format handles which aspect of a field. Key changes: 1. Capability declaration — MappedFieldType gains a searchCapability() method. Each field mapper returns its appropriate capability (e.g., numeric types return POINT_RANGE, keyword returns FULL_TEXT_SEARCH). 2. FieldCapabilityAssigner — A new component that walks configured data formats in priority order and assigns capabilities to field types greedily (primary format claims first, secondary fills gaps). 3. Removed redundant pluggable format overrides — Mappers like SearchAsYouTypeFieldMapper, TokenCountFieldMapper, RankFeatureFieldMapper no longer need custom parseCreateFieldForPluggableFormat implementations — the capability system handles routing automatically. 4. Test infrastructure — New test utilities (DataFormatTestUtils, MockParquetDataFormatPlugin) for testing capability assignment and pluggable format behavior. The net effect: data format plugins (parquet, lucene) declare what capabilities they support per field type, and the system automatically routes each field's data to the appropriate format — without field mappers needing to know about specific data formats. --------- Signed-off-by: Sagar Darji <darsaga@amazon.com> Signed-off-by: Mohit Godwani <mgodwan@amazon.com> Co-authored-by: Sagar Darji <darsaga@amazon.com> Co-authored-by: Mohit Godwani <mgodwan@amazon.com>
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.