Skip to content

Commit bb14d0a

Browse files
authored
feat(reco): gate cold_start engines by nsessions (FFM-1161) (#248)
Diagnostic in FFM-1107 showed cold_start_adapt was serving users averaging 198 days old with 2 lifetime sessions — dormant returners, not fresh users. Continuation was only 5% despite a healthy 50% LR at position 1, because the engine optimizes for users with zero history while the population had stale preferences. Add an nsessions gate behind COLD_START_NSESSIONS_GATE_ENABLED: - pulls nsessions from user_stats into the cached user_info payload - restricts cold_start_explore / cold_start_adapt to nsessions <= 1 - dormant returners (nmemes_sent < 30, nsessions >= 2) fall through to the growing-user blender, which has stale-but-real signal to work with Feature flag defaults off so this lands behind a config toggle and stays disjoint from the recently_liked_blender_v2 read (FFM-1094, May 17-18).
1 parent e1a19b6 commit bb14d0a

5 files changed

Lines changed: 180 additions & 1 deletion

File tree

experiments/log.jsonl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,3 +50,4 @@
5050
{"timestamp":"2026-04-20T03:40:00Z","agent":"ceo","action":"experiment_created","status":"success","summary":"Created early-channel-popup experiment: move popup.telegram_channel from meme #50 to #5 with conversion tracking","details":{"experiment":"2026-04-20-early-channel-popup","reason":"75% of new users leave before meme #5, channel popup at #50 reaches almost nobody. Channel = retention anchor via TG feed. Approved by founder.","impact":"Expected: 10%+ subscribe rate (vs ~0% at #50), improved D7 retention for subscribers","cto_task":"FFM-590","analyst_task":"FFM-591"},"error":null}
5151
{"timestamp":"2026-04-20T09:30:00Z","agent":"ceo","action":"daily_review","status":"success","summary":"Weekly CEO review. Retro: 22 commits, 1.0k LOC, 8 PRs, 59% fix ratio, 5-day streak. WAU 589 (-13% WoW) is #1 concern. Goat experiment Day 7/14 all green (LR 41.9% vs 39.4% baseline). Early-channel-popup waiting on deploy. describe_memes structurally rate-limited (0.7% coverage). 31 stale QA issues need cleanup.","details":{"north_star":30,"wau":589,"wau_change":"-13%","active_experiments":["goat-recency-filter (Day 7/14, all green, conclude Apr 27)","early-channel-popup (code ready, FFM-590 in_review, not deployed)"],"retro":{"commits":22,"fix_ratio":"59%","test_ratio":"3%","streak_days":5,"ship_of_week":"goat recency filter PR #169 + describe_memes hardening (6 commits)"},"actions":["experiment_continue: goat-recency-filter — all metrics passing, Day 7/14","task_created: FFM-598 CTO — deploy early-channel-popup branch (HIGH)","task_created: FFM-599 CTO — structural fix for describe_memes rate limiting (HIGH)","task_created: FFM-600 CTO — batch close 31 stale QA issues (LOW)","todos_updated: marked goat recency filter as DONE in TODOS.md"],"priorities_next_week":["1. Deploy early-channel-popup (retention lever for WAU decline)","2. Fix describe_memes structurally (content quality)","3. Monitor goat experiment through Apr 27","4. Clean up stale issues"]},"error":null}
5252
{"timestamp":"2026-05-01T07:15:11Z","agent":"comms-manager","action":"daily_channel_post","status":"success","summary":"Published meme-of-April kulich highlight to @fastfoodmemes: 44% like rate, 60 likes from 133 sends.","details":{"task":"FFM-866","channel":"@fastfoodmemes","url":"https://t.me/fastfoodmemes/12590","telegram_message_id":12590,"editorial_post_id":1,"category":"E","entity_id":"meme:top_lr_10119534","note":"Fun meme findings may target the main RU meme channel; build-in-public/product/process posts target @ffmemes."},"error":null}
53+
{"timestamp":"2026-05-11T00:00:00Z","agent":"cto","action":"experiment_started","status":"success","summary":"FFM-1161: cold-start nsessions gate + feature flag shipped to PR. Adds nsessions to user_info, gates cold_start_explore/adapt to nsessions<=1 behind COLD_START_NSESSIONS_GATE_ENABLED. Dormant returners (nmemes_sent<30 but nsessions>=2) fall through to growing-user blender.","details":{"issue":"FFM-1161","experiment":"2026-05-11-cold-start-routing-fix","feature_flag":"COLD_START_NSESSIONS_GATE_ENABLED","files":["src/config.py","src/tgbot/service.py","src/recommendations/meme_queue.py","tests/recommendations/test_meme_queue.py"],"tests":"19/19 mocked tests green; DB integration tests require Docker","coordination":"Do not deploy until recently_liked_blender_v2 (FFM-1094) May 17-18 read is in flight — coordinate with Release Engineer."},"error":null}

src/config.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,11 @@ class Config(BaseSettings):
4949
DEEPSEEK_BASE_URL: str = "https://api.deepseek.com"
5050
CHAT_AGENT_ENABLED: bool = False
5151

52+
# FFM-1161: gate cold_start engines so they only serve genuinely-new users
53+
# (nsessions <= 1). Dormant returners with nmemes_sent < 30 but multiple
54+
# sessions fall through to the growing-user blender instead.
55+
COLD_START_NSESSIONS_GATE_ENABLED: bool = False
56+
5257
PREFECT_API_URL: str | None = None
5358
PREFECT_AUTH_STRING: str | None = None
5459

src/recommendations/meme_queue.py

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
from sqlalchemy import text
77

88
from src import redis
9+
from src.config import settings
910
from src.database import fetch_all
1011
from src.recommendations.blender import blend
1112
from src.recommendations.blender_experiments import (
@@ -95,6 +96,11 @@ async def generate_recommendations(
9596
if nmemes_sent is None:
9697
nmemes_sent = user_info["nmemes_sent"]
9798

99+
# FFM-1161: nsessions gate. cold_start engines were designed for first-session
100+
# users; the cached user_info may predate the gate (1h TTL) — treat missing as 0
101+
# so we don't accidentally route dormant returners into cold_start.
102+
nsessions = user_info.get("nsessions") or 0
103+
98104
queue_key = redis.get_meme_queue_key(user_id)
99105

100106
meme_ids_in_queue = []
@@ -149,10 +155,19 @@ async def get_candidates(user_id, limit, use_recently_liked_blender_v2: bool = T
149155
Phase 3 (16-30): Transition — blend adapt + growing engines
150156
151157
Fallback chain: phase engine -> lr_smoothed -> best_uploaded_memes
158+
159+
FFM-1161: when COLD_START_NSESSIONS_GATE_ENABLED, cold_start is only
160+
used for first-session users (nsessions <= 1). Dormant returners
161+
(nsessions >= 2, nmemes_sent < 30) fall through to the growing-user
162+
blender below — they have stale signal, not zero signal.
152163
"""
153164

165+
in_cold_start_window = nmemes_sent < 30
166+
if settings.COLD_START_NSESSIONS_GATE_ENABLED:
167+
in_cold_start_window = in_cold_start_window and nsessions <= 1
168+
154169
# Cold start: 3-phase adaptive
155-
if nmemes_sent < 30:
170+
if in_cold_start_window:
156171
if nmemes_sent < 6:
157172
# Phase 1: diverse exploration from top sources
158173
engine = "cold_start_explore"

src/tgbot/service.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -358,6 +358,7 @@ async def get_user_info(
358358
SELECT
359359
type,
360360
COALESCE(nmemes_sent, 0) nmemes_sent,
361+
COALESCE(nsessions, 0) nsessions,
361362
COALESCE(memes_watched_today, 0) memes_watched_today,
362363
UIL.interface_lang
363364
FROM "user" AS U

tests/recommendations/test_meme_queue.py

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,15 @@
1313
TEST_USER_ID = 99999
1414

1515

16+
def _patch_user_info(nsessions: int = 0, nmemes_sent: int = 0, **extra):
17+
user_info = defaultdict(int, {"nmemes_sent": nmemes_sent, "nsessions": nsessions, **extra})
18+
return patch(
19+
"src.recommendations.meme_queue.get_user_info",
20+
new_callable=AsyncMock,
21+
return_value=user_info,
22+
)
23+
24+
1625
@pytest.fixture(autouse=True)
1726
def mock_redis():
1827
"""Mock Redis and user_info calls — these tests validate blending logic, not Redis."""
@@ -488,3 +497,151 @@ class TestRetriever(CandidatesRetriever):
488497
for nmemes in [0, 3, 8, 12, 20, 25]:
489498
candidates = await generate_recommendations(TEST_USER_ID, 10, nmemes, TestRetriever())
490499
assert len(candidates) == 0, f"Expected empty at nmemes_sent={nmemes}"
500+
501+
502+
# ── FFM-1161: nsessions gate ──
503+
504+
505+
def _growing_retriever_class():
506+
"""Retriever covering both cold_start engines and the growing-user blender."""
507+
508+
async def cold_start_explore(self, user_id, limit=10, exclude_meme_ids=[], **kw):
509+
return [{"id": 101, "recommended_by": "cold_start_explore"}]
510+
511+
async def cold_start_adapt(self, user_id, limit=10, exclude_meme_ids=[], **kw):
512+
return [{"id": 201, "recommended_by": "cold_start_adapt"}]
513+
514+
async def lr_smoothed(self, user_id, limit=10, exclude_meme_ids=[], **kw):
515+
return [{"id": 301, "recommended_by": "lr_smoothed"}]
516+
517+
async def best_uploaded_memes(self, user_id, limit=10, exclude_meme_ids=[], **kw):
518+
return [{"id": 401, "recommended_by": "best_uploaded_memes"}]
519+
520+
async def like_spread_and_recent_memes(self, user_id, limit=10, exclude_meme_ids=[], **kw):
521+
return [{"id": 501, "recommended_by": "like_spread_and_recent_memes"}]
522+
523+
async def recently_liked(self, user_id, limit=10, exclude_meme_ids=[], **kw):
524+
return [{"id": 601, "recommended_by": "recently_liked"}]
525+
526+
async def goat(self, user_id, limit=10, exclude_meme_ids=[], **kw):
527+
return [{"id": 701, "recommended_by": "goat"}]
528+
529+
async def es_ranked(self, user_id, limit=10, exclude_meme_ids=[], **kw):
530+
return [{"id": 801, "recommended_by": "es_ranked"}]
531+
532+
class TestRetriever(CandidatesRetriever):
533+
engine_map = {
534+
"cold_start_explore": cold_start_explore,
535+
"cold_start_adapt": cold_start_adapt,
536+
"lr_smoothed": lr_smoothed,
537+
"best_uploaded_memes": best_uploaded_memes,
538+
"like_spread_and_recent_memes": like_spread_and_recent_memes,
539+
"recently_liked": recently_liked,
540+
"goat": goat,
541+
"es_ranked": es_ranked,
542+
}
543+
544+
return TestRetriever
545+
546+
547+
@pytest.mark.asyncio
548+
async def test_gate_off_dormant_returner_still_uses_cold_start():
549+
"""Default (gate disabled): nsessions is ignored — cold_start still routes by nmemes_sent."""
550+
retriever = _growing_retriever_class()()
551+
with (
552+
_patch_user_info(nsessions=5, nmemes_sent=8),
553+
patch("src.config.settings.COLD_START_NSESSIONS_GATE_ENABLED", False),
554+
):
555+
candidates = await generate_recommendations(
556+
TEST_USER_ID, 10, nmemes_sent=8, retriever=retriever
557+
)
558+
assert any(c["recommended_by"] == "cold_start_adapt" for c in candidates)
559+
560+
561+
@pytest.mark.asyncio
562+
async def test_gate_on_first_session_routes_to_cold_start_explore():
563+
"""Gate on + nsessions<=1 + nmemes_sent<6 → cold_start_explore (Phase 1)."""
564+
retriever = _growing_retriever_class()()
565+
with (
566+
_patch_user_info(nsessions=0, nmemes_sent=0),
567+
patch("src.config.settings.COLD_START_NSESSIONS_GATE_ENABLED", True),
568+
):
569+
candidates = await generate_recommendations(
570+
TEST_USER_ID, 10, nmemes_sent=0, retriever=retriever
571+
)
572+
assert len(candidates) == 1
573+
assert candidates[0]["recommended_by"] == "cold_start_explore"
574+
575+
576+
@pytest.mark.asyncio
577+
async def test_gate_on_first_session_phase2_routes_to_cold_start_adapt():
578+
"""Gate on + nsessions<=1 + 6<=nmemes_sent<16 → cold_start_adapt (Phase 2)."""
579+
retriever = _growing_retriever_class()()
580+
with (
581+
_patch_user_info(nsessions=1, nmemes_sent=8),
582+
patch("src.config.settings.COLD_START_NSESSIONS_GATE_ENABLED", True),
583+
):
584+
candidates = await generate_recommendations(
585+
TEST_USER_ID, 10, nmemes_sent=8, retriever=retriever
586+
)
587+
assert any(c["recommended_by"] == "cold_start_adapt" for c in candidates)
588+
589+
590+
@pytest.mark.asyncio
591+
async def test_gate_on_dormant_returner_falls_through_to_growing_blender():
592+
"""Gate on + nsessions>=2 + nmemes_sent<30 → growing-user blender, NO cold_start engines."""
593+
retriever = _growing_retriever_class()()
594+
with (
595+
_patch_user_info(nsessions=3, nmemes_sent=12),
596+
patch("src.config.settings.COLD_START_NSESSIONS_GATE_ENABLED", True),
597+
):
598+
candidates = await generate_recommendations(
599+
TEST_USER_ID, 10, nmemes_sent=12, retriever=retriever, random_seed=42
600+
)
601+
sources = {c["recommended_by"] for c in candidates}
602+
assert "cold_start_explore" not in sources
603+
assert "cold_start_adapt" not in sources
604+
# Growing blender is pinned at lr_smoothed in position 0
605+
assert candidates[0]["recommended_by"] == "lr_smoothed"
606+
607+
608+
@pytest.mark.asyncio
609+
async def test_gate_on_mature_user_unchanged():
610+
"""Gate on + mature user (nmemes_sent>=100) → blender_v2 path, untouched by gate."""
611+
retriever = _growing_retriever_class()()
612+
with (
613+
_patch_user_info(nsessions=5, nmemes_sent=120),
614+
patch("src.config.settings.COLD_START_NSESSIONS_GATE_ENABLED", True),
615+
patch(
616+
"src.recommendations.meme_queue.get_recently_liked_blender_v2_weights",
617+
new_callable=AsyncMock,
618+
return_value=MATURE_BLENDER_TREATMENT_WEIGHTS,
619+
) as get_weights,
620+
):
621+
candidates = await generate_recommendations(
622+
TEST_USER_ID, 10, nmemes_sent=120, retriever=retriever, random_seed=42
623+
)
624+
sources = {c["recommended_by"] for c in candidates}
625+
assert "cold_start_explore" not in sources
626+
assert "cold_start_adapt" not in sources
627+
get_weights.assert_awaited_once_with(TEST_USER_ID)
628+
629+
630+
@pytest.mark.asyncio
631+
async def test_gate_on_missing_nsessions_treated_as_zero():
632+
"""Stale cache without nsessions key → treated as 0, cold_start still applies."""
633+
retriever = _growing_retriever_class()()
634+
# user_info lacks 'nsessions' (defaultdict(int) returns 0)
635+
stale_info = defaultdict(int, {"nmemes_sent": 4})
636+
with (
637+
patch(
638+
"src.recommendations.meme_queue.get_user_info",
639+
new_callable=AsyncMock,
640+
return_value=stale_info,
641+
),
642+
patch("src.config.settings.COLD_START_NSESSIONS_GATE_ENABLED", True),
643+
):
644+
candidates = await generate_recommendations(
645+
TEST_USER_ID, 10, nmemes_sent=4, retriever=retriever
646+
)
647+
assert candidates[0]["recommended_by"] == "cold_start_explore"

0 commit comments

Comments
 (0)