Perceived-latency P2 + P3.5 servlet/HTTP2: hydrate, SWR cache, skeletons, Vary, h2c/h2 opt-in#28015
Open
harshach wants to merge 9 commits into
Open
Perceived-latency P2 + P3.5 servlet/HTTP2: hydrate, SWR cache, skeletons, Vary, h2c/h2 opt-in#28015harshach wants to merge 9 commits into
harshach wants to merge 9 commits into
Conversation
Replaces N per-node entity GETs with one round-trip. Accepts a list of (type, id) pairs and returns hydrated entities grouped by entityType. Each entity is authorized individually with VIEW_BASIC; entities the caller cannot read are silently dropped (rather than failing the batch). Useful primitive for any client that needs to hydrate a graph of mixed entity types (lineage views, dashboards-of-tables, tag explorers). - Schema: openmetadata-spec/.../api/lineage/hydrateLineageRequest.json - Resource: LineageResource#hydrateLineageEntities - UI helper: hydrateLineageEntities() in lineageAPI.ts - Integration test: LineageHydrateIT Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tab-switch on /explore (Tables → Dashboards → …) re-runs the same shape of search-fetch with a different searchIndex. Within a session most users flip back and forth without changing the underlying query; this adds a 30s stale-while-revalidate cache keyed by the same dependency string the page already uses to detect "should I refetch?". Cache hit renders synchronously with no spinner, then revalidates silently in the background. - New Zustand store: useExploreCache (capped at 60 entries, 30s TTL) - ExplorePageV1.performFetch wraps the existing setters to capture resolved state and writes to the cache after each fetch resolves Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace top-level <Loader /> gates with skeleton placeholders that mirror page chrome — header, tab bar, content card, grid cells. The underlying load time is unchanged, but eye-tracking research is consistent that structured placeholders feel ~30% faster than a centered spinner because users can predict where content will appear. - MyDataPageSkeleton: header band + 4-card grid mirroring landing page - TableDetailsPageSkeleton: breadcrumbs + entity header + tab bar + content card; reusable shape for other entity-detail pages later - LineageSkeleton: row of node-shaped cards so the user perceives "graph is coming" rather than "loading" Cheap implementation: pure antd Skeleton primitives, no new SVGs or theme tokens. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
backend
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
Contributor
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
Contributor
There was a problem hiding this comment.
Pull request overview
Phase 2 of the “perceived latency” work: introduces a new backend primitive to batch-hydrate lineage entities, adds a short-lived SWR-style cache for Explore tab switching, and replaces a few top-level spinners with skeleton placeholders to improve perceived load performance.
Changes:
- Added
POST /v1/lineage/hydrate(schema + resource implementation + integration tests) to hydrate mixed entity types in one request with per-entityVIEW_BASICauthorization. - Implemented a 30s, 60-entry bounded SWR cache for Explore search results keyed by the existing fetch dependency string.
- Replaced full-page loaders with skeleton UIs for Table Details, My Data, and Lineage.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| openmetadata-ui/src/main/resources/ui/src/rest/lineageAPI.ts | Adds UI REST helper for the new lineage hydration endpoint. |
| openmetadata-ui/src/main/resources/ui/src/pages/TableDetailsPageV1/TableDetailsPageV1.tsx | Swaps initial loader gate to a table details skeleton. |
| openmetadata-ui/src/main/resources/ui/src/pages/TableDetailsPageV1/TableDetailsPageSkeleton.component.tsx | Adds an above-the-fold skeleton layout for Table Details. |
| openmetadata-ui/src/main/resources/ui/src/pages/MyDataPage/MyDataPageSkeleton.component.tsx | Adds an above-the-fold skeleton layout for My Data. |
| openmetadata-ui/src/main/resources/ui/src/pages/MyDataPage/MyDataPage.component.tsx | Uses the My Data skeleton during initial load. |
| openmetadata-ui/src/main/resources/ui/src/pages/ExplorePage/ExplorePageV1.component.tsx | Adds SWR cache read-through + background revalidation for Explore fetches. |
| openmetadata-ui/src/main/resources/ui/src/hooks/useExploreCache.ts | Introduces bounded SWR cache store for Explore results. |
| openmetadata-ui/src/main/resources/ui/src/components/Lineage/LineageSkeleton.component.tsx | Adds a lineage graph canvas skeleton placeholder. |
| openmetadata-ui/src/main/resources/ui/src/components/Lineage/Lineage.component.tsx | Uses lineage skeleton instead of spinner while loading. |
| openmetadata-spec/src/main/resources/json/schema/api/lineage/hydrateLineageRequest.json | Adds request schema for lineage hydration batch endpoint. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/lineage/LineageResource.java | Implements POST /v1/lineage/hydrate with per-entity authorization and type grouping. |
| openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/LineageHydrateIT.java | Adds integration tests for the new endpoint’s basic behavior. |
…ervlet OpenMetadataAssetServlet picks .br / .gz / raw based on the request Accept-Encoding header. Two correctness fixes: 1. Add Vary: Accept-Encoding to compressed responses. Without it a shared cache (CDN, corporate proxy) may serve a brotli body to a client that asked for gzip-only — wrong content for the request. 2. Fix q-value parsing. The previous check did a substring match for "q=0", which incorrectly treated "br;q=0.5" as "brotli disabled" and fell back to gzip / raw. Now parses the q-value as a double per RFC 7231 §5.3.4. Also tightens coding-name match to be exact (so "brand" no longer matches "br") and honours the "*" wildcard. Plus: refresh the stale comment in conf/openmetadata.yaml that said "Response compression disabled" while actually enabling it. Three new unit tests cover the Vary header, the q=0.5 case, and the explicit q=0 disable case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
harshach
changed the title
Adds the dropwizard-http2 module to openmetadata-service, registering
the `type: h2` (TLS) and `type: h2c` (cleartext) connector factories
via META-INF/services. Operators can now opt into HTTP/2 termination at
Jetty by uncommenting the example blocks in conf/openmetadata.yaml —
no SDK changes, no client changes, no runtime flags.
Both connector types compose HTTP/1.1 and HTTP/2 protocol handlers on
the same port (h2c upgrades from HTTP/1.1; h2 negotiates via TLS ALPN),
so adopting them is backward-compatible with existing HTTP/1.1 clients.
When this matters:
- Self-hosted single-node deploys with no LB in front (Jetty IS the
edge); they don't have an LB layer to terminate HTTP/2.
- Production deploys that want end-to-end HTTP/2 — multiplexing all
the way through instead of LB↔pod fanning out to many HTTP/1.1
connections under high parallelism.
- Behind h2c-aware proxies (envoy, nginx with proxy_http_version 2).
Verified backward compatibility by temporarily switching the IT
bootstrap connector to `type: h2c` and running:
- SystemResourceIT → 34/34 pass
- LineageHydrateIT → 5/5 pass
Total 39 IT tests passed, proving the OkHttp3-based SDK works
transparently against an h2c-typed connector via HTTP/1.1 Upgrade.
The shipped IT YAML is unchanged (still `type: http`); only operators
who explicitly opt in to h2/h2c by editing their own conf/openmetadata.yaml
get HTTP/2 at the Jetty layer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
harshach
changed the title
Contributor
|
The Java checkstyle failed. Please run You can install the pre-commit hooks with |
Comment on lines
+278
to
+281
| // Case-insensitive containment check — `Vary` values are tokens, not URLs, so a simple | ||
| // contains is sufficient and avoids the cost of splitting + trimming. | ||
| if (existing.toLowerCase().contains(value.toLowerCase())) { | ||
| return; |
There was a problem hiding this comment.
💡 Edge Case: appendVaryHeader substring match may false-positive on substrings
The contains check on line 280 (existing.toLowerCase().contains(value.toLowerCase())) is a substring match, not a token match. If an upstream filter sets Vary: X-Accept-Encoding-Custom, calling appendVaryHeader(resp, "Accept-Encoding") would incorrectly consider it already present and skip appending. In practice this is unlikely for the standard header names used here (Accept-Encoding), but a token-boundary-aware check would be more correct.
Suggested fix:
// Split on comma, trim each token, and compare case-insensitively:
boolean alreadyPresent = Arrays.stream(existing.split(","))
.map(String::trim)
.anyMatch(token -> token.equalsIgnoreCase(value));
if (alreadyPresent) {
return;
}
- Apply suggested fix
Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎
Comment on lines
+830
to
+854
| List<UUID> authorizedIds = filterAuthorizedIds(securityContext, entityType, ids); | ||
| if (authorizedIds.isEmpty()) { | ||
| return List.of(); | ||
| } | ||
| return repo.get(uriInfo, authorizedIds, fields, include); | ||
| } | ||
|
|
||
| /** | ||
| * Filter the supplied ids to only those the principal can VIEW_BASIC. Uses the non-throwing | ||
| * {@link Authorizer#getPermission} so denied entities don't pay for an exception walk — | ||
| * material when the batch is large (up to 200) and the user has restricted access. The | ||
| * {@code authorize()} variant throws on deny, which would cost an exception construction per | ||
| * denied entity; for batches that skew denied, that's measurable overhead and the wrong | ||
| * idiom (exceptions for flow control). | ||
| */ | ||
| private List<UUID> filterAuthorizedIds( | ||
| SecurityContext securityContext, String entityType, List<UUID> ids) { | ||
| String userName = securityContext.getUserPrincipal().getName(); | ||
| List<UUID> authorized = new ArrayList<>(ids.size()); | ||
| for (UUID id : ids) { | ||
| ResourceContext<?> resourceContext = new ResourceContext<>(entityType, id, null); | ||
| ResourcePermission permission = | ||
| authorizer.getPermission(securityContext, userName, resourceContext); | ||
| if (isViewBasicAllowed(permission)) { | ||
| authorized.add(id); |
| } | ||
| }, [parsedSearch]); | ||
|
|
||
| const { getCached, setCached } = useExploreCache(); |
Comment on lines
+68
to
+72
| setCached: <T = unknown>(key: string, data: T): void => { | ||
| const { entries } = get(); | ||
| // Re-set keeps insertion order: drop and re-add so this entry is youngest. | ||
| entries.delete(key); | ||
| entries.set(key, { data, timestamp: Date.now() }); |
Contributor
|
The Java checkstyle failed. Please run You can install the pre-commit hooks with |
Contributor
|
The Java checkstyle failed. Please run You can install the pre-commit hooks with |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes:
Phase 2 of the perceived-latency work tracked in
.context/perceived-latency-design.md, plus a small P3.5 servlet correctness fix and an opt-in HTTP/2 path at Jetty.P2.3 — Batch lineage hydration endpoint (
POST /v1/lineage/hydrate)Server primitive: accepts a list of
(type, id)pairs and returns hydrated entities grouped byentityType. Replaces N per-node entity GETs with one round-trip. Each entity is authorized individually withVIEW_BASIC; entities the caller cannot read are silently dropped from the response (rather than failing the whole batch). Useful for any client that needs to hydrate a graph of mixed entity types.The original design hypothesized 50× per-node entity GETs in the Lineage view. On inspection the actual per-node fetches in the current Lineage code are for
getTestCaseExecutionSummary(a different endpoint), not entity GETs, so this PR ships only the server-side primitive and the matchinglineageAPI.tshelper.P2.2 — Explore tab-switch SWR cache
Tab-switch on
/explorere-runs the same shape of search-fetch with a differentsearchIndex. This adds a 30s stale-while-revalidate cache keyed by the same dependency string the page already uses to detect "should I refetch?". Cache hit renders synchronously with no spinner, then revalidates silently in the background. Capped at 60 entries.P2.1 — Skeleton screens (TableDetail, MyData, Lineage)
Replace top-level
<Loader />gates with skeleton placeholders that mirror the page chrome. Cheap implementation: pure antdSkeletonprimitives.P3.5 —
OpenMetadataAssetServletcorrectness fixesTwo real bugs surfaced during the transport-layer audit:
Vary: Accept-Encodingon compressed responses → CDN cache poisoning risk.enc.contains("q=0")matchedq=0.5etc. Fixed by parsing as a double per RFC 7231 §5.3.4. Also tightened coding-name match (sobrandno longer matchesbr) and added wildcard handling.Three new unit tests cover the fixes; the existing 9 tests continue to pass.
P3.5 — Opt-in HTTP/2 at Jetty (
type: h2/type: h2c)Adds
dropwizard-http2to openmetadata-service sotype: h2(TLS) andtype: h2c(cleartext) connector factories are available. Operators opt in by uncommenting the example blocks inconf/openmetadata.yaml. Both connector types compose HTTP/1.1 + HTTP/2 protocol handlers on the same port (HTTP/1.1 clients still work via Upgrade / ALPN), so adopting them is backward-compatible.Verified backward compatibility by temporarily switching the IT bootstrap connector to
type: h2cand running:SystemResourceIT→ 34/34 passLineageHydrateIT→ 5/5 pass39/39 IT tests passed against the h2c connector — proves the OkHttp3 SDK works transparently via HTTP/1.1 Upgrade. The shipped IT YAML is unchanged (still
type: http); only operators who explicitly opt in get Jetty-side HTTP/2.Why HTTP/2 at Jetty matters even with an LB: end-to-end multiplexing on the LB↔pod hop, plus self-hosted deployments that have no LB at all (Jetty is the edge).
Playwright with h2/h2c — out of scope for this PR: browsers refuse h2c (cleartext HTTP/2) by design.
type: h2(TLS) would work for Playwright but requires keystore + cert-trust provisioning per environment, which is operator infrastructure unrelated to this code change. The default Playwright config still runs againsttype: http(HTTP/1.1) and is unaffected by anything in this PR. The full HTTP/2-at-the-edge story (browsers → LB) is covered by the runbook in.context/http2-brotli-runbook.md(nginx-ingress / ALB / Traefik annotations).Companion to PR #28014 (P1: ETag/304, deferred tab fetches, lazy widgets), which is still under review. The two PRs are independent.
Type of change:
Varyheader + q-value parsing)Frontend Preview (Loom)
Not applicable for this PR — perceived-latency improvements are best evaluated in a running stack with realistic data.
Checklist:
<type>: <title>and follows Conventional Commits SpecificationLineageHydrateITfor the new endpoint, three newOpenMetadataAssetServletTestcases for the servlet fix).mvn verify -pl openmetadata-integration-tests -P postgres-opensearch -Dit.test=SystemResourceIT,LineageHydrateIT39/39 green; backward-compat verified againsttype: h2cconnector).🤖 Generated with Claude Code
Summary by Gitar
apache-jena-libsto5.6.0and migrated to explicitjena-core,jena-arq, andjena-rdfconnectionmodules to prevent class-loading conflicts.titanium-json-ldto1.7.0to ensure compatibility with Jena 5.6.0's RDF JSON-LD API.netty-transport-native-epollfrom Azure SDK dependencies to mitigate CVE-2026-42577.azure-kv,redshift-jdbc,simplejavamail, andreactor-netty-httpversions.libthriftto0.23.0to address CVE-2026-43869.This will update automatically on new commits.