Fixes: Advanced Search Filter failing for Custom Properties at Database Schema level#27320
Fixes: Advanced Search Filter failing for Custom Properties at Database Schema level#27320
Conversation
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
When the user is on a specific entity type's explore tab (e.g., Database Schema), the AdvanceSearchProvider now correctly derives the entity type from the current searchIndex. This ensures that custom property extension field paths do not include the entity type namespace prefix (e.g., `extension.informationOwners.displayName.keyword` instead of `extension.databaseSchema.informationOwners.displayName.keyword`), matching the actual Elasticsearch document structure where extension data is stored without the entity type prefix. Key changes: - Add `effectiveEntityType` useMemo that derives the entity type from `searchIndex` when `entityType` prop is not explicitly provided - Add `customPropsEntityTypeRef` useRef to track cache validity by entity type context, enabling correct cache invalidation on tab switches - Use `effectiveEntityType` in `fetchCustomPropertyType` so extension subfields are built with the correct path context Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/a304e4a7-a4f1-4848-b390-7616f043352c Co-authored-by: aniketkatkar97 <51777795+aniketkatkar97@users.noreply.github.com>
Agent-Logs-Url: https://github.com/open-metadata/OpenMetadata/sessions/a304e4a7-a4f1-4848-b390-7616f043352c Co-authored-by: aniketkatkar97 <51777795+aniketkatkar97@users.noreply.github.com>
sonika-shah
added
the
safe to test
❌ Playwright Lint Check Failed — ESLint + Prettier + Organise ImportsThe following files have style issues that need to be fixed: Fix locally (fast — only for changed files in the branch): make ui-checkstyle-playwright-changedOr to fix all playwright files: make ui-checkstyle-playwright |
❌ I18n Sync Check FailedTranslation files are out of sync. Changed files: Fix locally: make i18n-sync-fix |
❌ Lint Check Failed — ESLint + Prettier + Organise Imports (src)The following files have style issues that need to be fixed: Fix locally (fast — only for changed files in the branch): make ui-checkstyle-src-changedOr to fix all files: make ui-checkstyle-src |
❌ App Docs Check FailedGenerated app docs are stale. Changed files: Fix locally: make generate-app-docs |
❌ Licence Header Check FailedThe following files are missing or have outdated licence headers: Fix locally: make license-header-fix |
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
There was a problem hiding this comment.
Pull request overview
Fixes Advanced Search filters for custom properties on Explore tabs (notably Database Schema) by ensuring extension field paths match how Elasticsearch flattened stores extension keys (without an entity-type namespace when appropriate).
Changes:
- Derives an
effectiveEntityTypefromsearchIndexwhen theentityTypeprop is not provided. - Uses a ref (
customPropsEntityTypeRef) to synchronously track which entity type context populated the cached custom properties. - Invalidates/refetches the custom properties cache when the effective entity type changes to rebuild extension subfields with the correct path context.
| // When entityType is not explicitly provided, derive it from the current searchIndex | ||
| // so that custom property extension field paths do not include the entity type namespace | ||
| // prefix. The Elasticsearch `extension` field uses the `flattened` type and stores data | ||
| // WITHOUT the entity type prefix (e.g., key is "informationOwners.displayName", not | ||
| // "databaseSchema.informationOwners.displayName"), so omitting the prefix is required | ||
| // for queries to match actual documents. | ||
| const effectiveEntityType = useMemo(() => { | ||
| if (entityType) { | ||
| return entityType; | ||
| } | ||
| if (isArray(searchIndex)) { | ||
| return undefined; | ||
| } | ||
| const mapping = searchClassBase.getSearchIndexEntityTypeMapping(); | ||
| const mapped = mapping[searchIndex]; | ||
| // EntityType.ALL represents a multi-entity context — preserve namespace prefixes | ||
| return mapped && mapped !== EntityType.ALL ? mapped : undefined; | ||
| }, [entityType, searchIndex]); |
There was a problem hiding this comment.
This change alters how custom property paths are generated (deriving an effective entity type from searchIndex and changing cache invalidation). There’s currently no active unit test covering the Explore-tab scenario where entityType is undefined and ensuring extension fields are built without the entity-type namespace. Please add/enable a test that asserts fetchCustomPropertyType/processEntityTypeFields are invoked with the derived entity type (e.g., DATABASE_SCHEMA → "databaseSchema") and that switching tabs causes the customProps cache to be refetched with the new context.
❌ UI Checkstyle Failed❌ ESLint + Prettier + Organise Imports (src)One or more source files have linting or formatting issues. Affected files
Fix locally (fast — only checks files changed in this branch): make ui-checkstyle-changed |
Jest test CoverageUI tests summary
|
Code Review ✅ ApprovedAdvanced Search Filter now works correctly for Custom Properties at the Database Schema level. No issues found. OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
| if (isArray(searchIndex)) { | ||
| return undefined; | ||
| } | ||
| const mapping = searchClassBase.getSearchIndexEntityTypeMapping(); | ||
| const mapped = mapping[searchIndex]; |
There was a problem hiding this comment.
effectiveEntityType returns undefined for any array searchIndex, even when the array contains a single index. In flows that pass a single selected entity as an array (e.g., onChangeSearchIndex([SearchIndex.TABLE])), this keeps the old extension namespace behavior and can reintroduce the same custom-property query mismatch. Consider deriving the entity type when searchIndex is an array of length 1 (and only treating it as multi-entity when length > 1).
| if (isArray(searchIndex)) { | |
| return undefined; | |
| } | |
| const mapping = searchClassBase.getSearchIndexEntityTypeMapping(); | |
| const mapped = mapping[searchIndex]; | |
| const normalizedSearchIndex = isArray(searchIndex) | |
| ? searchIndex.length === 1 | |
| ? searchIndex[0] | |
| : undefined | |
| : searchIndex; | |
| if (isNil(normalizedSearchIndex)) { | |
| return undefined; | |
| } | |
| const mapping = searchClassBase.getSearchIndexEntityTypeMapping(); | |
| const mapped = mapping[normalizedSearchIndex]; |
| let extensionSubField = | ||
| customPropsEntityTypeRef.current === effectiveEntityType | ||
| ? customProps | ||
| : null; | ||
| if (extensionSubField === null) { | ||
| extensionSubField = await fetchCustomPropertyType(); | ||
| setCustomProps(extensionSubField); | ||
| customPropsEntityTypeRef.current = effectiveEntityType; |
There was a problem hiding this comment.
This change causes getAllCustomProperties() to be re-fetched whenever effectiveEntityType changes (e.g., each Explore tab switch). Since the API response is global and only the derived subfields shape changes, consider caching the raw custom properties response once and rebuilding extensionSubField locally per entity type (or caching computed subfields per entity type) to avoid repeated network calls and recomputation.
| if (extensionSubField === null) { | ||
| extensionSubField = await fetchCustomPropertyType(); | ||
| setCustomProps(extensionSubField); | ||
| customPropsEntityTypeRef.current = effectiveEntityType; | ||
| } |
There was a problem hiding this comment.
loadData() can run concurrently when users switch tabs quickly (it’s triggered by searchIndex changes). Because it awaits fetchCustomPropertyType() and then updates customProps and customPropsEntityTypeRef, a slower previous request can complete after a newer one and overwrite the cache/ref for the current tab. Consider guarding with a monotonically increasing request id / "latest only" check (or abort/cancel) before applying setCustomProps, customPropsEntityTypeRef.current = ..., and setConfig.
|
🟡 Playwright Results — all passed (23 flaky)✅ 3687 passed · ❌ 0 failed · 🟡 23 flaky · ⏭️ 89 skipped
🟡 23 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
Advanced search filters for custom properties on the Database Schema explore tab return no results. The query builder generates paths like
extension.databaseSchema.informationOwners.displayName.keyword, but Elasticsearch'sflattenedtype stores extension data without the entity type prefix — the actual key isinformationOwners.displayName.Root Cause
AdvanceSearchProviderreceives noentityTypeprop on the Explore page, soprocessEntityTypeFieldsadds an entity type namespace to all extension field paths. This works for display purposes but produces ES queries that never match any document, regardless of entity type.The bug was most visible on Database Schema because it was the reported case — the same namespace mismatch affects any entity type tab when using the "all entities" mode custom property cache.
Changes —
AdvanceSearchProvider.component.tsxeffectiveEntityType(useMemo): Derives entity type fromsearchIndexviasearchClassBase.getSearchIndexEntityTypeMapping()when theentityTypeprop is absent.SearchIndex.DATABASE_SCHEMA→"databaseSchema", multi-entity indexes orEntityType.ALL→undefined(preserves existing namespace behavior for the all-entities tab).customPropsEntityTypeRef(useRef): Tracks which entity type was used to populate thecustomPropscache. Avoids relying on React state update timing for cache invalidation — the ref is checked synchronously insideloadDatawhen the user switches tabs.fetchCustomPropertyType: UseseffectiveEntityTypeinstead ofentityType, so on the Database Schema tab onlydatabaseSchemaproperties are included with direct paths (no namespace prefix).Cache invalidation in
loadData: ComparescustomPropsEntityTypeRef.current === effectiveEntityType; on mismatch, refetches and updates the ref.Before → After (Database Schema tab):
Type of change:
Checklist:
Fixes <issue-number>: <short explanation>