[codex] Use LLM classification for job leads#378
Conversation
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_94ccb1b9-a6ab-4ace-90c2-b10c210bb1fb) |
📝 WalkthroughWalkthroughThis PR adds contractor-friendly job lead classification via a two-stage heuristic/LLM classifier for HN "Who is hiring?" scraping, threads classifier configuration through worker/runtime settings, exposes classification metadata and review summaries through shared helpers and API payloads, and updates admin dashboard and Discord bot UIs to display the new classification/summary fields instead of raw confidence percentages. ChangesJob lead contractor classification pipeline
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Scraper as scrape_job_leads
participant Source as HackerNewsWhoIsHiringLeadSource
participant Classifier as JobLeadClassifier
participant LLM as LLM API
participant API as Dashboard API
participant UI as Admin Dashboard/Discord Bot
Scraper->>Source: build_job_lead_source(classifier)
Source->>Classifier: classify(comment_text)
Classifier->>LLM: structured classification request
LLM-->>Classifier: classification response
Classifier-->>Source: JobLeadClassification
Source-->>Scraper: JobLeadInput with metadata
API->>API: job_lead_display_payload(lead)
API-->>UI: contractor_classification + review_summary
UI->>UI: render classification badge / review summary
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3d81dde852
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| classification = ( | ||
| classifier.classify(text) | ||
| if classifier is not None | ||
| else classify_contractor_lead_heuristic(text) | ||
| ) |
There was a problem hiding this comment.
When the LLM classifier is enabled, this branch now trusts the classifier before applying the existing _SEEKING_WORK_RE guard. In HN Who-is-hiring threads, applicant comments commonly start with SEEKING WORK and also contain terms like Freelance or Contract; if the LLM or an injected classifier returns a positive classification for that text, _lead_from_hn_comment will insert an applicant advert into job_leads, whereas the previous heuristic always rejected these comments. Apply the deterministic SEEKING WORK check before calling/trusting the classifier.
Useful? React with 👍 / 👎.
| except Exception: | ||
| pass | ||
|
|
||
| response = client.chat.completions.create( |
There was a problem hiding this comment.
If the structured parse call times out or the provider is down, this broad except falls through into a second chat.completions.create request before classify() can use the keyword heuristic. During an HN scrape this happens once per top-level comment, so with the default 8s classifier timeout a dead provider can cost roughly 16s per comment and block the scrape worker for many minutes before any leads are upserted. Treat timeout/provider failures as a signal to return the heuristic fallback instead of making another LLM call.
Useful? React with 👍 / 👎.
| summary = cls._truncate_job_lead_text(format_job_lead_review_summary(lead), 180) | ||
| return f"{index}. `{lead.id[:8]}` **{title}**\n{summary}\n{lead.source_url}" |
There was a problem hiding this comment.
With /list-job-leads limit:10, each row now includes up to a 180-character LLM/keyword summary, and the command concatenates all rows into one followup.send; a normal set of pending leads with HN URLs can exceed settings.discord_sendmsg_character_limit (2000), causing Discord to reject the response. Split or cap the total response when adding these summaries.
Useful? React with 👍 / 👎.
| fireworks_api_key=_clean(getattr(settings, "fireworks_api_key", None)), | ||
| fireworks_model=_clean(getattr(settings, "agent_fast_model", None)), |
There was a problem hiding this comment.
When only JOB_LEAD_CLASSIFIER_MODEL is set to a Fireworks model with FIREWORKS_API_KEY (and no OpenAI/Bifrost primary), this passes agent_fast_model as the Fireworks model instead of the classifier override. If AGENT_FAST_MODEL is unset, build_openai_compatible_provider_attempts() produces no Fireworks provider, so the enabled classifier silently never calls the LLM and all scrapes use the keyword fallback. Pass the resolved classifier model here for that direct-provider configuration.
Useful? React with 👍 / 👎.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_f716fe73-3398-4fd4-8492-1ebc491ea610) |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
apps/discord_bot/src/five08/discord_bot/cogs/jobs.py (1)
3437-3467: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueGuard against empty
leadsin_format_job_lead_review_message.If called with an empty list,
leads[0]in theif not lines:fallback branch raisesIndexError. Currently safe because the only caller (list_sourced_job_leads) checksif not leadsfirst, but the method itself has no such guard, making it a latent trap for future callers/tests.🛡️ Optional defensive guard
def _format_job_lead_review_message(cls, leads: list[JobLead]) -> str: """Format a bounded Discord response for pending lead review.""" + if not leads: + return "Pending job leads:\n\nNo pending job leads found." limit = settings.discord_sendmsg_character_limit🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/discord_bot/src/five08/discord_bot/cogs/jobs.py` around lines 3437 - 3467, The fallback path in _format_job_lead_review_message can still index leads[0] when the input list is empty, so add a defensive empty-list guard at the start of this method. Return an appropriate bounded empty-state message (or otherwise avoid the fallback branch) before any access to leads[0], and keep the existing formatting logic for non-empty inputs unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/shared/src/five08/job_lead_sources.py`:
- Around line 322-353: The structured-output branch in _classify_with_llm is
missing the deterministic sampling setting used by the JSON path. Update the
client.beta.chat.completions.parse call in _classify_with_llm to pass
temperature=0 as well, keeping both classification paths consistent and
deterministic while preserving the existing response parsing logic.
---
Nitpick comments:
In `@apps/discord_bot/src/five08/discord_bot/cogs/jobs.py`:
- Around line 3437-3467: The fallback path in _format_job_lead_review_message
can still index leads[0] when the input list is empty, so add a defensive
empty-list guard at the start of this method. Return an appropriate bounded
empty-state message (or otherwise avoid the fallback branch) before any access
to leads[0], and keep the existing formatting logic for non-empty inputs
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 116d7c7c-9d05-49da-8fca-227ba917857b
📒 Files selected for processing (16)
.env.exampleapps/admin_dashboard/src/main.tsxapps/api/src/five08/backend/api.pyapps/api/src/five08/backend/static/dashboard/.vite/manifest.jsonapps/api/src/five08/backend/static/dashboard/assets/index-DFyKzfnC.jsapps/api/src/five08/backend/static/dashboard/index.htmlapps/discord_bot/src/five08/discord_bot/cogs/jobs.pyapps/worker/src/five08/worker/config.pydocs/configuration.mdpackages/shared/src/five08/job_lead_sources.pypackages/shared/src/five08/job_leads.pypackages/shared/src/five08/runtime_config.pytests/integration/test_dashboard_playwright.pytests/unit/test_backend_api.pytests/unit/test_job_lead_sources.pytests/unit/test_jobs.py
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_6a65777d-c575-44b6-8ce9-5a078be3ed81) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_55d47667-d25e-4cca-9feb-6ee504bb3674) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0113dd0f2e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| openai_direct_model=_clean(getattr(settings, "openai_direct_model", None)) | ||
| or _clean(getattr(settings, "agent_fallback_model", None)), |
There was a problem hiding this comment.
When the worker is configured with only OPENAI_DIRECT_API_KEY (or uses a Bifrost primary with direct OpenAI fallback) and sets JOB_LEAD_CLASSIFIER_MODEL, these lines still pass agent_fallback_model as openai_direct_model; build_openai_compatible_provider_attempts() prefers that argument over the primary classifier model, so the classifier silently calls the generic fallback model instead of the configured classifier override. Use the resolved classifier model for this direct-provider path unless OPENAI_DIRECT_MODEL is explicitly intended to override it.
Useful? React with 👍 / 👎.
| explicit_fast = _clean(getattr(settings, "agent_fast_model", None)) | ||
| if explicit_fast: | ||
| return explicit_fast |
There was a problem hiding this comment.
When OPENROUTER_API_KEY is configured and AGENT_FAST_MODEL is an unqualified model such as gpt-4.1-mini, this returns that raw value as the OpenRouter direct model. That bypasses build_openai_compatible_provider_attempts()'s normal _openrouter_model_from_primary() conversion to openai/gpt-4.1-mini, so the direct OpenRouter fallback is called with the wrong model id and the classifier falls back to heuristics whenever the primary provider is unavailable.
Useful? React with 👍 / 👎.
| explicit_fast = _clean(getattr(settings, "agent_fast_model", None)) | ||
| if explicit_fast: | ||
| return explicit_fast |
There was a problem hiding this comment.
When FIREWORKS_API_KEY is set and AGENT_FAST_MODEL is an unqualified or OpenAI/OpenRouter model, this returns that model as the Fireworks direct model; build_openai_compatible_provider_attempts() then appends a fireworks-direct attempt with that invalid Fireworks model id. In configurations where the primary provider is unavailable or absent, the classifier tries Fireworks with the wrong model and falls back to heuristics instead of using a valid direct provider.
Useful? React with 👍 / 👎.
1 participant
Summary
review_summary/contractor_classificationfields through the dashboard API/list-job-leadsand the admin dashboard to show readable classification context instead of raw confidence percentagesValidation
uv run pytest tests/unit/test_job_lead_sources.py tests/unit/test_jobs.py tests/unit/test_backend_api.py -quv run ruff check packages/shared/src/five08/job_lead_sources.py packages/shared/src/five08/job_leads.py packages/shared/src/five08/runtime_config.py apps/api/src/five08/backend/api.py apps/discord_bot/src/five08/discord_bot/cogs/jobs.py apps/worker/src/five08/worker/config.py tests/unit/test_job_lead_sources.py tests/unit/test_jobs.py tests/unit/test_backend_api.py./scripts/pyrefly.shbun run typecheckbun run lintbun run testbun run buildNote
Medium Risk
Classification logic gates which sourced leads look contractor-friendly; incorrect LLM or heuristic results could affect review volume, but behavior degrades to keywords and does not touch auth or payments.
Overview
Adds LLM-first contractor-fit classification for Hacker News job-lead intake via
JobLeadClassifier, with keyword/heuristic fallback when no OpenAI-compatible provider is available or the model call fails. Classification metadata is stored on leads and normalized throughjob_lead_display_payload, exposingcontractor_classificationand a human-readablereview_summary.Dashboard API list and review job-lead handlers now return that display payload instead of raw lead rows. The admin dashboard types and list UI show method/confidence labels (e.g. LLM vs keyword fallback) and the review summary, replacing the previous confidence-percentage badge.
New
JOB_LEAD_CLASSIFIER_*env settings document enablement, model, and timeout for the classifier.Reviewed by Cursor Bugbot for commit 0113dd0. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation