perf: optimize enrichable members selection query#4389
Merged
Conversation
Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>
Contributor
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
Contributor
There was a problem hiding this comment.
Pull request overview
Optimizes member-enrichment selection by limiting ranked candidates before aggregating identities.
Changes:
- Uses an activity-ranked candidate CTE.
- Replaces epoch cache checks with
make_interval. - Preserves identity aggregation after candidate selection.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
services/libs/data-access-layer/src/old/apps/members_enrichment_worker/index.ts:162
- This inner join changes the eligible population.
membersGlobalActivityCountonly contains members represented inmemberSegmentsAgg, while the currently enabled ProgAI and LinkedIn-scraper filters do not require activity. The previousLEFT JOINreturned those members with an activity count of 0; now they can never be selected for enrichment. Preserve a fallback branch for enrichable members absent from the materialized view (after the indexed activity-ranked branch) so the optimization does not silently drop them.
FROM "membersGlobalActivityCount"
INNER JOIN members ON members.id = "membersGlobalActivityCount"."memberId"
services/libs/data-access-layer/src/old/apps/members_enrichment_worker/index.ts:198
enrichableBySqlis documented to allow references to themembersalias, and the SERP source does so. This outer query only definesc,mi, andmembersGlobalActivityCount, so using that valid source makes PostgreSQL fail withmissing FROM-clause entry for table "members"when this predicate is reapplied. Join the base member row for the limited candidate set before evaluating the identity filter.
WHERE ${enrichableBySqlJoined}
Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>
…ation/crowd.dev into perf/enrichable-members-query
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fetchMembersForEnrichmentwas selecting enrichable members by scanning a huge identity join, aggregating, then sorting — soLIMIT 500only applied at the end and the query regularly ran for tens of minutes.Changes
membersGlobalActivityCounthigh → low, apply the same enrichability / cache / bot filters, takeLIMITfirst, thenjson_aggidentities for those candidates onlyupdatedAt > now() - make_interval(...)(same meaning as the old epoch check)