perf(api): hot-first routing + filter-context for get_actions account polls#176
Conversation
…ontext get_actions always sorts by global_sequence (never _score), so scoring the account should-array, generic term/range filters, and the code:name filter was wasted work on every poll. Move them from bool.must / root-level should into bool.filter: identical result sets (account AND generics AND >=1 code-filter), minus BM25 scoring, and now cacheable. must_not is unchanged. Adds filter-context.test.ts pinning the placement.
…(default off) An unbounded newest-first account poll (e.g. account=eosio.token, desc, no time bound) only needs the most recent actions, which live in the newest action partition(s) -- yet today it fans out across every <chain>-action-* shard, including old/warm ones, which is a top search-CPU cost on history nodes. Add opt-in hot-first routing (api.hot_first_actions, default off; api.hot_first_window, default 2): search the newest partition(s) first and widen to the full set only if that window returns fewer than 'limit' hits. Heavy pollers never touch old shards; sparse/old-only accounts stay correct via the fallback. Eligible only for the default global_sequence-desc sort with skip=0 and no time bound -- bounded queries, pagination, asc, custom sortedBy, and explicit hot_only keep their existing path. resolveHotIndices() resolves the newest physical partitions via a TTL-cached, stampede-safe _cat/indices lookup and degrades to the <chain>-action-* wildcard on any error. Also repairs ?hot_only=true, which targeted a <chain>-action alias the codebase never creates (it would throw index_not_found); it now routes through the same resolver.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces "hot-first" routing for unbounded get_actions polls, optimizing performance by searching the newest index partitions first and only widening to the full wildcard set if necessary. It also refactors several Elasticsearch query builders to use filter context instead of scoring context to avoid unnecessary scoring overhead, and adds corresponding unit tests. Feedback highlights two key improvements: preventing a redundant fallback query when the hot index resolution degrades to the wildcard fallback, and safely checking if the Elasticsearch cat.indices response is an array to avoid potential runtime type errors.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if (esResults.hits.hits.length < size) { | ||
| // The account is sparse within the hot window — widen to the full set for correctness. | ||
| esResults = await fastify.elastic.search<any>(esOpts); | ||
| } else { | ||
| hotFirstUsed = true; | ||
| } |
There was a problem hiding this comment.
When resolveHotIndices degrades to the wildcard fallback (e.g., due to Elasticsearch errors or lack of physical indices), hotIndex will be identical to indexPattern (both being ${chain}-action-*). Under this condition, if the first search returns fewer than size hits, the fallback search is executed on the exact same wildcard pattern again.\n\nThis results in a redundant, heavy wildcard query on the entire cluster, doubling the load precisely when Elasticsearch might be struggling.\n\nWe should check if hotIndex !== indexPattern before executing the fallback query.
| if (esResults.hits.hits.length < size) { | |
| // The account is sparse within the hot window — widen to the full set for correctness. | |
| esResults = await fastify.elastic.search<any>(esOpts); | |
| } else { | |
| hotFirstUsed = true; | |
| } | |
| if (esResults.hits.hits.length < size && hotIndex !== indexPattern) { | |
| // The account is sparse within the hot window — widen to the full set for correctness. | |
| esResults = await fastify.elastic.search<any>(esOpts); | |
| } else { | |
| hotFirstUsed = hotIndex !== indexPattern; | |
| } |
| const names = (records as Array<{ index?: string }>) | ||
| .map(r => r.index) | ||
| .filter((n): n is string => typeof n === 'string' && n.length > 0) | ||
| .slice(0, win); |
There was a problem hiding this comment.
Depending on the Elasticsearch client version or configuration, cat.indices might return a wrapped response (e.g., { body: [...] } in v7) or a non-array structure under certain error/empty states. Casting records directly to an array and calling .map without checking if it is an array can lead to a runtime TypeError: records.map is not a function.\n\nUsing Array.isArray(records) ensures safe degradation to the fallback wildcard without throwing uncaught exceptions.
| const names = (records as Array<{ index?: string }>) | |
| .map(r => r.index) | |
| .filter((n): n is string => typeof n === 'string' && n.length > 0) | |
| .slice(0, win); | |
| const names = (Array.isArray(records) ? records : []) | |
| .map(r => r.index) | |
| .filter((n): n is string => typeof n === 'string' && n.length > 0) | |
| .slice(0, win); |
There was a problem hiding this comment.
Pull request overview
This PR reduces Elasticsearch CPU and shard fan-out for high-frequency, newest-first v2/get_actions account polls by moving non-scoring clauses into filter context and adding an opt-in “hot-first” index routing strategy that queries only the newest action partitions first.
Changes:
- Move
get_actionsaccount/generic/code:name clauses from scoring context (must/rootshould) intobool.filterto avoid unnecessary scoring and enable caching. - Add opt-in “hot-first” routing (
api.hot_first_actions,api.hot_first_window) using a TTL-cached, stampede-safe resolver of newest physical action indices, with safe wildcard fallback. - Add unit tests pinning filter-context placement and hot-index resolver behavior; update config schema/reference defaults.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/hot-index.test.ts | Adds unit coverage for hot-index resolution, caching, and wildcard fallback behavior. |
| tests/unit/filter-context.test.ts | Pins get_actions query clause placement in bool.filter to prevent regressions back to scoring context. |
| src/interfaces/hyperionConfig.ts | Adds config interface + Zod schema for hot_first_actions and hot_first_window. |
| src/api/routes/v2-history/get_actions/get_actions.ts | Implements hot-only repair and opt-in hot-first routing for eligible account polls. |
| src/api/routes/v2-history/get_actions/functions.ts | Moves generic/account/code-action clauses into filter context and preserves semantics with explicit minimum_should_match. |
| src/api/helpers/hot-index.ts | Introduces resolver for newest physical indices with TTL caching and stampede protection. |
| references/config.ref.json | Documents new config defaults (hot_first_actions, hot_first_window). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (hotFirstEligible) { | ||
| const hotIndex = await resolveHotIndices(fastify, 'action', hotWindow); | ||
| esResults = await fastify.elastic.search<any>({...esOpts, index: hotIndex}); | ||
| if (esResults.hits.hits.length < size) { | ||
| // The account is sparse within the hot window — widen to the full set for correctness. | ||
| esResults = await fastify.elastic.search<any>(esOpts); | ||
| } else { | ||
| hotFirstUsed = true; | ||
| } | ||
| } else { | ||
| esResults = await fastify.elastic.search<any>(esOpts); | ||
| } |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b258df1fac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| (queryStruct.bool.filter ??= []).push({ | ||
| match: _qObj | ||
| }); |
There was a problem hiding this comment.
When a request uses sortedBy=_score:desc (which addSortedBy still accepts), moving the @transfer.memo match query into bool.filter removes all scoring for that clause, so Elasticsearch returns matching actions with essentially arbitrary/tie-broken scores instead of the previous relevance order. This is especially visible for memo searches using fuzziness/operator options, because the result set remains the same but the requested _score ordering is no longer meaningful.
Useful? React with 👍 / 👎.
| const records = await fastify.elastic.cat.indices({ | ||
| index: fallback, | ||
| h: 'index', | ||
| s: 'index:desc', | ||
| format: 'json' |
There was a problem hiding this comment.
When a cluster contains multiple action index versions during/after a reindex, resolving <chain>-action-* by lexicographic index:desc can pick a higher version's low/old partition (for example chain-action-v2-000001) ahead of the live latest partition from another version (for example chain-action-v1-000999). With hot_first_actions enabled, if that selected window returns limit hits, the fallback full search never runs, so account polls can omit the true latest actions from the full wildcard result set.
Useful? React with 👍 / 👎.
- get_actions: when the resolver degrades to the wildcard (hotIndex === fullPattern), run the full search once instead of executing the same full-pattern query twice on shortfall, and don't set hot_first in that case (gemini/copilot). - hot-index: scope the _cat/indices lookup to the active index_version so a multi-version cluster can't surface a higher-version low partition ahead of the live latest one (codex). - hot-index: guard the _cat response with Array.isArray before .map (gemini). - functions: keep the @transfer.memo full-text match in scoring context so sortedBy=_score still ranks by relevance; only the high-cardinality keyword clauses move to filter (codex). - tests: cover version-scoped resolution, non-array _cat response, and memo-in-must.
|
Thanks for the reviews — all findings addressed in
New tests cover version-scoped resolution, a non-array |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0cdc4d2a68
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } else { | ||
| hotFirstUsed = true; |
There was a problem hiding this comment.
When api.hot_first_actions is enabled and the hot window returns at least limit hits for a busy account, this branch skips the full <chain>-action-* search and later exposes response.total from the hot-index-only result. For accounts with additional older matches, clients that rely on total (including track=true/numeric track, and even the default tracked count) now see only the hot-window count instead of the total for the requested query, even though the endpoint still advertises the normal full-history query semantics.
Useful? React with 👍 / 👎.
Problem
Unbounded, newest-first
get_actionsaccount polls — e.g.?account=eosio.token&limit=100(desc, no time bound) — are a top search-CPU cost on history nodes. The query fans out across every<chain>-action-*shard, including old/warm ones, even though the latest N actions all live in the newest partition. On busy contracts (eosio.tokenon WAX) this is hammered by external pollers and lights up warm nodes that should never be touched for "latest 100".Reported in the field by another operator monitoring inbound queries; the exact query is the standard
shouldovernotified/receipts.receiver/act.authorization.actorsorted byglobal_sequence desc.Changes
Tier 1 — filter context (always on, behavior-preserving)
get_actionsalways sorts byglobal_sequence(never_score), so scoring the account should-array, generic term/range filters, and the code:name filter was wasted work on every poll. Moved them frombool.must/ root-levelshouldintobool.filter:account AND generics AND (>=1 code-filter)— minus BM25 scoring, and now cacheable.must_notleft unchanged.Tier 2 — hot-first routing (opt-in, default off)
New
api.hot_first_actions(defaultfalse) +api.hot_first_window(default2). When enabled, an eligible poll searches the newestwindowpartitions first and widens to the full<chain>-action-*set only if that window returns fewer thanlimithits.eosio.token) fill the window → never fan out to old/warm shards.global_sequenceare provably in the newest partitions, so a full window = the true top N).global_sequence-desc sort withskip=0and noafter/before. Bounded queries, pagination,asc, customsortedBy, and explicithot_onlykeep their existing path.resolveHotIndices()resolves the newest physical partitions via a TTL-cached (30s), stampede-safe_cat/indiceslookup and degrades to the<chain>-action-*wildcard on any error — never fails a request.Bonus: repair
?hot_only=trueIt previously targeted a
<chain>-actionalias that nothing in the codebase ever creates (would throwindex_not_found). It now routes through the same resolver, so it actually works.Operator usage (no reindex required)
Caveat
When hot-first is satisfied, the response
totalreflects the hot window only. These polls already captotalat 10k, so impact is low — worth a changelog line.Testing
tsc --noEmitclean.bun test tests/unit— 118 pass (12 new):filter-context.test.tspins the filter-context placement;hot-index.test.tscovers resolver join order, caching/dedup, and wildcard fallback on error/empty.Scope / follow-ups
get_actionshas a separate older query path; out of scope here.hot_first_actions/hot_first_windowneed an entry ineosrio/hyperion-docswhen this ships.