Skip to content

feat: update to daily features#3962

Merged
capJavert merged 5 commits into
mainfrom
daily-highlights-updates
Jun 26, 2026
Merged

feat: update to daily features#3962
capJavert merged 5 commits into
mainfrom
daily-highlights-updates

Conversation

@capJavert

Copy link
Copy Markdown
Contributor
  • add MV for deducing tags per channel
  • upsert channels on dailyHeadlines call
  • add support to return brief from dailyHeadlines

@capJavert capJavert self-assigned this Jun 26, 2026
@pulumi

pulumi Bot commented Jun 26, 2026

Copy link
Copy Markdown

🍹 The Update (preview) for dailydotdev/api/prod (at 80d6303) was successful.

Resource Changes

    Name                                                       Type                           Operation
~   vpc-native-calculate-top-readers-cron                      kubernetes:batch/v1:CronJob    update
~   vpc-native-update-tags-str-cron                            kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-users-cron                         kubernetes:batch/v1:CronJob    update
~   vpc-native-daily-digest-cron                               kubernetes:batch/v1:CronJob    update
~   vpc-native-update-views-cron                               kubernetes:batch/v1:CronJob    update
-   vpc-native-api-clickhouse-migration-4810e472               kubernetes:batch/v1:Job        delete
~   vpc-native-sync-subscription-with-cio-cron                 kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-expired-better-auth-sessions-cron         kubernetes:batch/v1:CronJob    update
~   vpc-native-deployment                                      kubernetes:apps/v1:Deployment  update
~   vpc-native-rotate-daily-quests-cron                        kubernetes:batch/v1:CronJob    update
~   vpc-native-user-posts-analytics-refresh-cron               kubernetes:batch/v1:CronJob    update
~   vpc-native-post-analytics-achievements-cron                kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-channel-highlights-cron                   kubernetes:batch/v1:CronJob    update
~   vpc-native-update-current-streak-cron                      kubernetes:batch/v1:CronJob    update
+   vpc-native-api-clickhouse-migration-cee1bf93               kubernetes:batch/v1:Job        create
~   vpc-native-materialize-monthly-best-post-archives-cron     kubernetes:batch/v1:CronJob    update
~   vpc-native-post-lifecycle-state-clickhouse-cron            kubernetes:batch/v1:CronJob    update
~   vpc-native-update-achievement-rarity-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-user-profile-analytics-clickhouse-cron          kubernetes:batch/v1:CronJob    update
~   vpc-native-personalized-digest-deployment                  kubernetes:apps/v1:Deployment  update
~   vpc-native-personalized-digest-cron                        kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-stale-user-transactions-cron              kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-opportunities-cron                 kubernetes:batch/v1:CronJob    update
~   vpc-native-materialize-yearly-best-post-archives-cron      kubernetes:batch/v1:CronJob    update
~   vpc-native-generate-search-invites-cron                    kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-images-cron                        kubernetes:batch/v1:CronJob    update
~   vpc-native-bg-deployment                                   kubernetes:apps/v1:Deployment  update
~   vpc-native-update-tag-materialized-views-cron              kubernetes:batch/v1:CronJob    update
+   vpc-native-api-db-migration-cee1bf93                       kubernetes:batch/v1:Job        create
~   vpc-native-update-highlighted-views-cron                   kubernetes:batch/v1:CronJob    update
~   vpc-native-ws-deployment                                   kubernetes:apps/v1:Deployment  update
~   vpc-native-clean-old-notifications-cron                    kubernetes:batch/v1:CronJob    update
-   vpc-native-api-db-migration-4810e472                       kubernetes:batch/v1:Job        delete
~   vpc-native-temporal-deployment                             kubernetes:apps/v1:Deployment  update
~   vpc-native-validate-active-users-cron                      kubernetes:batch/v1:CronJob    update
~   vpc-native-user-profile-analytics-history-clickhouse-cron  kubernetes:batch/v1:CronJob    update
~   vpc-native-generic-referral-reminder-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-gifted-plus-cron                          kubernetes:batch/v1:CronJob    update
~   vpc-native-expire-super-agent-trial-cron                   kubernetes:batch/v1:CronJob    update
~   vpc-native-post-analytics-history-day-clickhouse-cron      kubernetes:batch/v1:CronJob    update
~   vpc-native-channel-digests-cron                            kubernetes:batch/v1:CronJob    update
... and 14 other changes

@capJavert

Copy link
Copy Markdown
Contributor Author

@claude review

@claude

claude Bot commented Jun 26, 2026

Copy link
Copy Markdown

Claude finished @capJavert's task in 2m 46s —— View job


PR Review: daily features update

I reviewed the channel-digest backfill, the keyword_channel MV, and the brief support in dailyHeadlines. Overall this is clean, idempotent, and well-tested. A few things worth confirming.

Questions / potential issues

1. The latest brief is not time-bounded — src/schema/highlights.ts:340
Channel digest posts are filtered to the last 24h (p."createdAt" >= :since), but the brief lookup has no recency filter:

queryRunner.manager.getRepository(Post).findOne({
  select: ['id'],
  where: { authorId: ctx.userId, type: PostType.Brief, visible: true, deleted: false },
  order: { createdAt: 'DESC' },
})

This means a user who hasn't generated a brief in weeks will still see a stale brief pinned to the top of dailyHeadlines, while the doc comment says "the latest digest post from the last 24 hours". Is the brief intentionally exempt from the 24h window? If not, it should get the same since guard. No test covers a brief older than 24h.

2. No cap on the number of seeded source follows — src/common/channelDigest/headlineFollows.ts (resolveHeadlineChannelSourcesForUser)
The seeding query has ORDER BY SUM(kc.posts) DESC but no LIMIT, so a user with many onboarding tags gets followed onto every qualifying channel-digest source at once. It's bounded by the number of enabled ChannelDigest rows (small today), so this is low-risk, but if that table grows this becomes a surprising bulk-follow. Consider an explicit top-N cap to make the intent clear.

3. Concurrent first calls can both seed — headlineFollows.ts (seedHeadlineChannelsForUser)
Two parallel dailyHeadlines requests for a not-yet-backfilled user both pass the alreadyBackfilled check and run the seed transaction. It's safe in practice (ContentPreferenceSource/FeedSource use orUpdate, markBackfilled uses orIgnore), but both can report seeded: true. Just flagging — the seedTagChipFeeds pattern reserves the slot before doing work specifically to make a concurrent call a guaranteed no-op. Low priority given the idempotent writes.

Nits

  • Consistency with followSourceheadlineFollows.ts builds the ContentPreferenceSource insert from a plain object, whereas the canonical followSource (src/common/contentPreference.ts:233) uses repository.create(...). It works here (TypeORM sets the child discriminator on .into(ContentPreferenceSource), and the test asserting follows.toHaveLength(1) would fail otherwise), but using .create() would keep the two follow paths aligned and future-proof against default columns.
  • Read replica usagegetUserOnboardingTags reads via the primary (con.getRepository(...)) while the rest of the file uses queryReadReplica. Minor; consider the replica for consistency since it's a pure read in the pre-write phase.
  • Lazy write inside a Query resolver — seeding writes during a GraphQL Query read. This mirrors the existing seedTagChipFeeds lazy-seed pattern and is correctly guarded with try/catch + logging, so it won't break the read. Noting for awareness, not a change request.

Looks good

  • MV unique index on (keyword, channel) matches the GROUP BY, so REFRESH ... CONCURRENTLY in updateTagMaterializedViews.ts is valid.
  • remoteConfig.vars is Partial, and the ?? defaultHeadlineChannelMinPosts fallback handles the unset case correctly.
  • Brief-first ordering (ORDER BY (id = :briefId) DESC) is deterministic and pagination keeps the brief on page 1 only.
  • Good test coverage: brief leading, no-channel brief, other-user brief exclusion, min-post threshold, already-backfilled, existing-preference, and no-tags no-op.
    · daily-highlights-updates

@capJavert

Copy link
Copy Markdown
Contributor Author

@claude review

@claude

claude Bot commented Jun 26, 2026

Copy link
Copy Markdown

Claude finished @capJavert's task in 1m 31s —— View job


PR Review: daily features update (re-review)

I re-read the diff against origin/main. The three substantive items from the previous round are all resolved:

  • Brief is now 24h-bounded...(recencyBounded && { createdAt: MoreThanOrEqual(since) }) (src/schema/highlights.ts:358), and there's now a test asserting a 30h-old brief is excluded (__tests__/highlights.ts:106).
  • Seed cap addedmaxSeededChannels = 50 with .limit(maxSeededChannels) (headlineFollows.ts:19,89).
  • Concurrent first-call seeding — claim-first via markBackfilled inside the transaction; the loser bails before writing follows (headlineFollows.ts:144-151). Matches the seedTagChipFeeds reserve-before-work pattern. The .create(...) consistency nit was also adopted (headlineFollows.ts:162).

This is solid, idempotent, and well-tested. A few smaller observations remain.

Observations

1. Users with tags but no qualifying channels are never marked backfilled — headlineFollows.ts:140
When resolveHeadlineChannelSourcesForUser returns empty (tags exist, but no channel clears minPosts in the 48h MV window), the function returns before markBackfilled. So every subsequent dailyHeadlines call for that user re-runs the full pre-seed sequence — exists + getChannelDigestSourceIds + hasDigestPreference + the MV aggregation query — indefinitely. This is intentional if the goal is "keep retrying until content exists to seed," but it's permanent per-request overhead for users whose onboarding tags never map to an active channel digest. Worth a comment documenting the intent, or marking backfilled with a TTL/retry bound if perpetual retry isn't desired.

2. No tie-breaker on the seed ordering — headlineFollows.ts:88
ORDER BY SUM(kc.posts) DESC + LIMIT 50 has no secondary sort key, so at the 50-row boundary (or any posts tie) which sources get followed is non-deterministic. Bounded by enabled ChannelDigest rows today so it rarely bites, but adding .addOrderBy('cd."sourceId"', 'ASC') makes the selection stable and reproducible.

3. getUserOnboardingTags reads via primary — headlineFollows.ts:47
Still con.getRepository(...) while the rest of the resolve path uses queryReadReplica. It's a pure pre-write read; the replica would be consistent with the others. Minor.

Nits

  • Migration index lacks IF NOT EXISTS / IF EXISTS1782457821133-KeywordChannelView.ts:20,27 uses bare CREATE UNIQUE INDEX / DROP INDEX, whereas CLAUDE.md asks for idempotent guards. It's generator output and the name is unique to the new MV, so re-run risk is low — flagging only for the documented convention.
  • MV REFRESH ... CONCURRENTLY prerequisites are met — the unique (keyword, channel) index matches the GROUP BY, and the migration creates the MV WITH DATA, so the first concurrent refresh in updateTagMaterializedViews.ts is valid. 👍

Looks good

  • Brief-first ordering (ORDER BY (id = :briefId) DESC) keeps the brief on page 1 only; pagination math is correct.
  • seeded ? primary : replica split gives read-after-write consistency right after seeding without forcing primary on the common path.
  • headlineChannelMinPosts is Partial in remote config with a ?? defaultHeadlineChannelMinPosts fallback, and the test mock sets it to 3.
  • Test coverage is thorough: brief leading, brief without channels, other-user brief exclusion, stale brief, min-post threshold, already-backfilled, existing preference, and no-tags no-op.
    · daily-highlights-updates

@capJavert capJavert enabled auto-merge (squash) June 26, 2026 08:40
@capJavert capJavert merged commit c5e8379 into main Jun 26, 2026
9 checks passed
@capJavert capJavert deleted the daily-highlights-updates branch June 26, 2026 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant