perf: optimize enrichable members selection query

[skwowet]

Summary

fetchMembersForEnrichment was selecting enrichable members by scanning a huge identity join, aggregating, then sorting — so LIMIT 500 only applied at the end and the query regularly ran for tens of minutes.

Changes

• Rewrite the query to walk membersGlobalActivityCount high → low, apply the same enrichability / cache / bot filters, take LIMIT first, then json_agg identities for those candidates only
• Keep the same result shape and enrichable identity filtering used by the enrichment worker
• Express cache freshness as updatedAt > now() - make_interval(...) (same meaning as the old epoch check)

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[Copilot]

Pull request overview

Optimizes member-enrichment selection by limiting ranked candidates before aggregating identities.

Changes:

• Uses an activity-ranked candidate CTE.
• Replaces epoch cache checks with make_interval.
• Preserves identity aggregation after candidate selection.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

services/libs/data-access-layer/src/old/apps/members_enrichment_worker/index.ts:162

• This inner join changes the eligible population. membersGlobalActivityCount only contains members represented in memberSegmentsAgg, while the currently enabled ProgAI and LinkedIn-scraper filters do not require activity. The previous LEFT JOIN returned those members with an activity count of 0; now they can never be selected for enrichment. Preserve a fallback branch for enrichable members absent from the materialized view (after the indexed activity-ranked branch) so the optimization does not silently drop them.

      FROM "membersGlobalActivityCount"
           INNER JOIN members ON members.id = "membersGlobalActivityCount"."memberId"

services/libs/data-access-layer/src/old/apps/members_enrichment_worker/index.ts:198

• enrichableBySql is documented to allow references to the members alias, and the SERP source does so. This outer query only defines c, mi, and membersGlobalActivityCount, so using that valid source makes PostgreSQL fail with missing FROM-clause entry for table "members" when this predicate is reapplied. Join the base member row for the limited candidate set before evaluating the identity filter.

    WHERE ${enrichableBySqlJoined}

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.