Skip to content

feat(query): built-in zoid/oid query operator#198

Merged
jensens merged 2 commits into
mainfrom
feat/zoid-query-operator
Jul 3, 2026
Merged

feat(query): built-in zoid/oid query operator#198
jensens merged 2 commits into
mainfrom
feat/zoid-query-operator

Conversation

@jensens

@jensens jensens commented Jul 3, 2026

Copy link
Copy Markdown
Member

Closes #197.

What

Adds a built-in zoid (and oid) query operator:

catalog(zoid=[123, 456, ...])   # ints (the object_state PK / catalog rid)
catalog(oid=[b"\x00...", ...])  # raw 8-byte ZODB oids -> int.from_bytes

Why

pgcatalog stores every record in object_state keyed by zoid (the BIGINT PK; the catalog rid is the zoid), but there was no way to query by it (getMetadataForRID is unsupported). That matters whenever you hold a set of ZODB object ids and want their security-filtered brains without waking the objects — the motivating case being zc.relation/intids resolution:

intid → getUtility(IIntIds).refs[intid].oid (wake-free) → int.from_bytes(..., "big") → zoid → catalog(zoid=[...])

On a real page (51 related objects) the wake path cost ~214 cold ZODB loads (more anonymously, walking the acquisition chain for View); a single catalog(zoid=[...]) returns the security-filtered brains directly. See #197 for the full analysis.

How

  • IndexType.ZOID pseudo-index (not a ZCatalog meta-type → never auto-registered) wired via _SPECIAL_BUILTIN_INDEX_TYPES, targeting the real zoid column rather than a JSONB idx key.
  • _handle_zoid emits zoid = ANY(%(p)s::bigint[]). Accepts int / digit-string / 8-byte oid bytes (_coerce_zoid); scalar or list; bool rejected; non-coercible values dropped (ZCatalog-lenient). Explicit ::bigint[] cast so an empty set matches nothing (= ANY('{}')) instead of a type-inference error — never the whole table. _MAX_ZOIDS bounds a hostile list.
  • No storage, no reindex — it filters the existing primary key. Composes with the normal query and with the allowedRolesAndUsers security filter injected by searchResults. Honored by strip_non_index_keys.

Tests

  • tests/test_query.py::TestZoidIndex — 12 unit tests (no PG): single/list ints, explicit spec, oid-bytes conversion, digit strings, bool rejection, non-coercible drop, empty-matches-nothing, _MAX_ZOIDS, AND-composition, strip_non_index_keys.
  • tests/test_query_integration.py::TestZoidIndexIntegration — 5 integration tests (real PG): zoid list/scalar, oid-bytes, empty-matches-nothing, composition with a FieldIndex.
  • Full query + columns suites green (197 unit, 35 query-integration). ruff clean.

Follow-up

Happy to add a Diataxis how-to ("resolve relations/intids to brains via zoid") if you want it in the same PR.

@jensens

jensens commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

Docs added (commit 3d7d3ba): a Diataxis how-to "Resolve relations to brains without waking objects" (docs/sources/how-to/resolve-relations-by-zoid.md) covering the intid → KeyReference.oid → zoid → catalog(zoid=[...]) recipe with a full related-items example and notes on security composition, no-reindex, empty-matches-nothing, and order preservation. Registered in the how-to toctree; sphinx-build clean (page builds, no new warnings). So this now includes code + tests + docs.

@jensens
jensens force-pushed the feat/zoid-query-operator branch 2 times, most recently from aa7e6b6 to b3e2d9d Compare July 3, 2026 07:51
jensens added 2 commits July 3, 2026 09:58
`catalog(zoid=[...])` / `catalog(oid=[...])` filters the object_state BIGINT
primary key directly (the catalog rid IS the zoid). Lets callers resolve a set
of ZODB object ids to their security-filtered brains WITHOUT waking the objects
— e.g. zc.relation/intids: intid -> KeyReference.oid -> int.from_bytes(...) ->
zoid — replacing an object-load N+1 in relation/back-reference resolution.

- `zoid` registered as a special built-in (IndexType.ZOID) targeting the real
  `zoid` column, not a JSONB idx key; `oid` is the same filter accepting raw
  8-byte ZODB oids (converted via int.from_bytes, matching obj_to_zoid).
- Accepts int / digit-str / bytes; scalar or list; bool rejected; non-coercible
  values dropped (ZCatalog-lenient). Explicit `::bigint[]` cast so an empty set
  is `= ANY('{}')` (matches nothing) rather than a type-inference error — never
  the whole table.
- No extra storage, no reindex (filters the existing PK). Composes with the
  normal query and the allowedRolesAndUsers security filter injected by
  searchResults. Honored by strip_non_index_keys.

Unit tests (12, no PG) + integration tests (5, real PG): list/scalar/oid-bytes,
bool rejection, empty-matches-nothing, AND-composition. Full query + columns
suites green (197 unit, 35 query-integration).
Diataxis how-to for the new zoid/oid query operator: the wake-free
intid -> KeyReference.oid -> zoid -> catalog(zoid=[...]) recipe for
related-items / back-reference resolution, with a full example and notes on
security composition, no-reindex, empty-matches-nothing, and order preservation.
Registered in the how-to toctree; adds zoid/oid/intid/IOBTree/… to the vale
vocabulary.
@jensens
jensens force-pushed the feat/zoid-query-operator branch from b3e2d9d to 6157aa8 Compare July 3, 2026 07:58
@jensens
jensens merged commit 165afa0 into main Jul 3, 2026
5 checks passed
@jensens
jensens deleted the feat/zoid-query-operator branch July 3, 2026 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Built-in zoid (and oid) query operator — resolve relations/intids to brains without waking objects

1 participant