Firecrawl Scraping

[ngoiyaeric]

Are you able to debug and ensure that firecrawl is correctly intergrated and functional for the website domain scraping and indexing into context for business users.

[ngoiyaeric]

@coderabbitai plan perhaps we need to connect an MCP https://docs.firecrawl.dev/mcp-server look through the docs for this functionality to work perfectly.

[coderabbitai]

Coding Plan

Summary

• Fix the single highest-impact bug first: replace the hardcoded 'anonymous' identity in app/actions.tsx so the business user's scraped/generated system prompt is actually injected into the agents' dynamicSystemPrompt.
• Harden the scraping worker and client factory: defensive API-key handling, type-safe scrape-result parsing with clear error messages, and prevention of post-timeout state corruption.
• Reconcile client/server domain validation and job-lifecycle/timeout handling so valid domains aren't blocked and timed-out jobs reach an accurate terminal state in both the UI and the database.
• Deliberately exclude the unrelated messages.embedding pgvector/AlphaEarth work and speculative additions (job cancellation, rate-limiting, DB enum constraints) as outside the scope of making Firecrawl scraping functional.

Implementation Steps

Phase 1: Restore System Prompt Delivery into AI Context

This phase fixes the core defect: scraped/generated business context never reaches the AI because the prompt is looked up under the wrong identity. The goal is to ensure the authenticated business user's saved prompt flows through to the agents that produce responses.

Task 1: Task 1.1: Use the authenticated user when fetching the system prompt

Resolve the hardcoded identity bug so saved prompts are actually retrieved.

• Update app/actions.tsx (around line 492) to resolve the real authenticated user via the existing lib/auth/get-current-user.ts helper instead of passing the literal 'anonymous' to getSystemPrompt(...).
• Ensure the resolved prompt is threaded into the dynamicSystemPrompt argument already consumed by lib/agents/researcher.tsx and lib/agents/writer.tsx.
• Preserve existing fallback behavior: when no user is resolved or the prompt is empty, the default geospatial/journalist prompt should still apply (no regression for anonymous/non-business users).

Task 2: Task 1.2: Verify end-to-end context propagation

Confirm the generated prompt reaches both agents that use it.

• Trace the value from getSystemPrompt(userId) in app/actions.tsx through to the dynamicSystemPrompt parameters in writer.tsx and researcher.tsx, confirming non-empty prompts override defaults and empty ones do not.
• Confirm the save path (saveSystemPrompt in lib/actions/chat.ts, invoked from components/settings/components/settings.tsx) writes to the same users.systemPrompt column being read back.

🤖 Prompt for AI agents

Fix the core identity bug in `app/actions.tsx` so the authenticated user's
system prompt is correctly fetched and delivered to AI agents.

- In `app/actions.tsx` around line 492, replace the hardcoded `'anonymous'`
string passed to `getSystemPrompt(...)` with the real authenticated user's ID,
resolved via the existing `lib/auth/get-current-user.ts` helper.
- Ensure the resolved prompt value is correctly passed into the
`dynamicSystemPrompt` argument consumed by both `lib/agents/researcher.tsx` and
`lib/agents/writer.tsx`.
- Preserve fallback behavior: if no user is resolved or the prompt is empty, the
default geospatial/journalist prompt must still apply — no regression for
anonymous or non-business users.
- Trace and verify the full data flow: `getSystemPrompt(userId)` in
`app/actions.tsx` → `dynamicSystemPrompt` in `writer.tsx` and `researcher.tsx`,
ensuring non-empty prompts override defaults and empty ones do not.
- Also confirm the save path: `saveSystemPrompt` in `lib/actions/chat.ts`
(called from `components/settings/components/settings.tsx`) writes to the same
`users.systemPrompt` column that `getSystemPrompt` reads from.

Phase 2: Harden the Firecrawl Scraping Pipeline

This phase makes the scraping worker robust and predictable so domain scraping succeeds reliably and fails cleanly with actionable messages. It addresses type-safety gaps, API-key handling, and post-timeout write races in the background worker.

Task 1: Task 2.1: Strengthen the Firecrawl client factory

Make client construction defensive and documented.

• Update lib/agents/tools/firecrawl.ts to treat whitespace-only FIRECRAWL_API_KEY as missing (trim before the empty check) and throw a clear, consistent error.
• Document FIRECRAWL_API_KEY in .env.local.example (and .env.embeddings.example if env vars are mirrored there) so deployments are configured correctly — this is a primary cause of "not functional" failures.

Task 2: Task 2.2: Make scrape-result handling type-safe and resilient

Remove the unsafe cast and produce clear failure messages.

• In lib/actions/system-prompt.ts runBackgroundWorker, replace the (scrapeResult as any)?.error handling with explicit, typed checks for the Firecrawl response shape (success flag, presence/absence of markdown, and any returned error text).
• Surface distinct, user-readable errorMessage values for the main failure modes (scrape failed, empty/too-short content, LLM generation failure) so the polling UI toast is meaningful.

Task 3: Task 2.3: Prevent post-timeout job corruption

Ensure the 2-minute worker timeout produces a single, consistent terminal state.

• In lib/actions/system-prompt.ts, guard updateJob writes in the worker so that once a terminal state (complete/error) is written, the still-running workPromise cannot overwrite it after the Promise.race timeout rejection (e.g., track a "settled" flag or re-check current job status before writing).
• Wrap the error-handler updateJob call so a DB failure inside the catch block cannot escape as an unhandled rejection.

🤖 Prompt for AI agents

Harden the Firecrawl scraping pipeline across three areas: API-key validation,
type-safe result handling, and post-timeout state protection.

**Firecrawl client factory (`lib/agents/tools/firecrawl.ts`):**
- Trim `FIRECRAWL_API_KEY` before checking if it is empty, so whitespace-only
values are treated as missing.
- Throw a clear, consistent error when the key is absent or whitespace-only.
- Add `FIRECRAWL_API_KEY` to `.env.local.example` (and `.env.embeddings.example`
if env vars are mirrored there) with a descriptive comment.

**Scrape-result handling (`lib/actions/system-prompt.ts` →
`runBackgroundWorker`):**
- Replace the `(scrapeResult as any)?.error` unsafe cast with explicit, typed
checks against the Firecrawl response shape: success flag, presence/absence of
`markdown`, and any returned error text.
- Surface distinct, user-readable `errorMessage` values for each failure mode:
scrape failed, empty/too-short content, and LLM generation failure.

**Post-timeout write race (`lib/actions/system-prompt.ts`):**
- Guard `updateJob` writes so that once a terminal state (`complete` or `error`)
has been written (e.g., via a "settled" flag or by re-checking current job
status before writing), the still-running `workPromise` after a `Promise.race`
timeout rejection cannot overwrite it.
- Wrap the error-handler `updateJob` call so any DB failure inside the catch
block cannot escape as an unhandled rejection.

Phase 3: Align Validation and Job Lifecycle Between Client and Server

This phase removes the input-validation mismatch that blocks valid submissions and reconciles timeout/staleness handling so job state is accurate in both the UI and the database.

Task 1: Task 3.1: Unify domain validation

Make client and server accept the same inputs.

• Reconcile the Zod rule in components/settings/components/settings.tsx (currently z.string().url()) with the server normalization in lib/actions/system-prompt.ts (which accepts bare domains like example.com by prepending https://).
• Choose a single accepted format — recommend accepting bare domains in both layers (validate/normalize identically client- and server-side) so users entering example.com are not blocked client-side while being accepted server-side.

Task 2: Task 3.2: Persist stale jobs and align timeouts

Ensure timed-out jobs reach a real terminal DB state and the client/server windows agree.

• Update getSystemPromptGenerationJob in lib/actions/system-prompt.ts so that when the staleness threshold is exceeded it also calls updateJob(... { status: 'error', errorMessage }), reconciling the DB row instead of only returning the error to the caller.
• Align the timing constants: the server worker timeout (2 min), the server staleness threshold (5 min), and the client poll cutoff (MAX_POLL_DURATION, 2 min) in system-prompt-form.tsx should be coherent so a near-2-minute job is not prematurely declared failed by the UI.

Task 3: Task 3.3: Harden UI polling robustness

Avoid lost results and unnecessary interval resets.

• In components/settings/components/system-prompt-form.tsx, stabilize the polling useEffect so a changing toast reference does not reset the interval mid-poll.
• Ensure error, timeout, and success branches consistently clear the interval and reset jobId/isGenerating state so the form never gets stuck in a generating state.

🤖 Prompt for AI agents

Align client/server domain validation, job lifecycle/timeout handling, and UI
polling robustness.

**Domain validation unification:**
- In `components/settings/components/settings.tsx`, update the Zod schema
(currently `z.string().url()`) to accept bare domains like `example.com`,
matching the server-side normalization in `lib/actions/system-prompt.ts` (which
prepends `https://`).
- Normalize/validate identically on both client and server so users entering
`example.com` are not blocked client-side while being accepted server-side.

**Stale job persistence and timeout alignment
(`lib/actions/system-prompt.ts`):**
- Update `getSystemPromptGenerationJob` so that when the staleness threshold is
exceeded, it calls `updateJob(... { status: 'error', errorMessage })` to write
the terminal state to the DB row, rather than only returning an error to the
caller.
- Align the timing constants — server worker timeout (2 min), server staleness
threshold (5 min), and client poll cutoff (`MAX_POLL_DURATION`, 2 min in
`system-prompt-form.tsx`) — so a near-2-minute job is not prematurely declared
failed by the UI.

**UI polling robustness
(`components/settings/components/system-prompt-form.tsx`):**
- Stabilize the polling `useEffect` so that a changing `toast` reference does
not reset the interval mid-poll.
- Ensure all terminal branches (error, timeout, and success) consistently clear
the polling interval and reset `jobId`/`isGenerating` state, preventing the form
from ever getting stuck in a generating state.

Research

Firecrawl is integrated as a focused feature for auto-generating a business "system prompt" from a company's website. A user enters a domain in the Settings → System Prompt tab (components/settings/components/system-prompt-form.tsx), which calls the startSystemPromptGeneration server action. That action creates a job row in prompt_generation_jobs, fires a fire-and-forget background worker (runBackgroundWorker in lib/actions/system-prompt.ts) that scrapes the domain via the Firecrawl client (lib/agents/tools/firecrawl.ts), feeds the markdown to an LLM, and stores the resulting prompt on the job. The UI polls getSystemPromptGenerationJob, then the saved prompt is persisted to users.systemPrompt. At response time the agents in lib/agents/writer.tsx and researcher.tsx accept a dynamicSystemPrompt that should originate from getSystemPrompt(userId) — but app/actions.tsx currently passes a hardcoded 'anonymous', breaking the context delivery entirely.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Fix the core identity bug in `app/actions.tsx` so the authenticated user's
system prompt is correctly fetched and delivered to AI agents.

- In `app/actions.tsx` around line 492, replace the hardcoded `'anonymous'`
string passed to `getSystemPrompt(...)` with the real authenticated user's ID,
resolved via the existing `lib/auth/get-current-user.ts` helper.
- Ensure the resolved prompt value is correctly passed into the
`dynamicSystemPrompt` argument consumed by both `lib/agents/researcher.tsx` and
`lib/agents/writer.tsx`.
- Preserve fallback behavior: if no user is resolved or the prompt is empty, the
default geospatial/journalist prompt must still apply — no regression for
anonymous or non-business users.
- Trace and verify the full data flow: `getSystemPrompt(userId)` in
`app/actions.tsx` → `dynamicSystemPrompt` in `writer.tsx` and `researcher.tsx`,
ensuring non-empty prompts override defaults and empty ones do not.
- Also confirm the save path: `saveSystemPrompt` in `lib/actions/chat.ts`
(called from `components/settings/components/settings.tsx`) writes to the same
`users.systemPrompt` column that `getSystemPrompt` reads from.
===============================================================================

Task: 2

Harden the Firecrawl scraping pipeline across three areas: API-key validation,
type-safe result handling, and post-timeout state protection.

**Firecrawl client factory (`lib/agents/tools/firecrawl.ts`):**
- Trim `FIRECRAWL_API_KEY` before checking if it is empty, so whitespace-only
values are treated as missing.
- Throw a clear, consistent error when the key is absent or whitespace-only.
- Add `FIRECRAWL_API_KEY` to `.env.local.example` (and `.env.embeddings.example`
if env vars are mirrored there) with a descriptive comment.

**Scrape-result handling (`lib/actions/system-prompt.ts` →
`runBackgroundWorker`):**
- Replace the `(scrapeResult as any)?.error` unsafe cast with explicit, typed
checks against the Firecrawl response shape: success flag, presence/absence of
`markdown`, and any returned error text.
- Surface distinct, user-readable `errorMessage` values for each failure mode:
scrape failed, empty/too-short content, and LLM generation failure.

**Post-timeout write race (`lib/actions/system-prompt.ts`):**
- Guard `updateJob` writes so that once a terminal state (`complete` or `error`)
has been written (e.g., via a "settled" flag or by re-checking current job
status before writing), the still-running `workPromise` after a `Promise.race`
timeout rejection cannot overwrite it.
- Wrap the error-handler `updateJob` call so any DB failure inside the catch
block cannot escape as an unhandled rejection.
===============================================================================

Task: 3

Align client/server domain validation, job lifecycle/timeout handling, and UI
polling robustness.

**Domain validation unification:**
- In `components/settings/components/settings.tsx`, update the Zod schema
(currently `z.string().url()`) to accept bare domains like `example.com`,
matching the server-side normalization in `lib/actions/system-prompt.ts` (which
prepends `https://`).
- Normalize/validate identically on both client and server so users entering
`example.com` are not blocked client-side while being accepted server-side.

**Stale job persistence and timeout alignment
(`lib/actions/system-prompt.ts`):**
- Update `getSystemPromptGenerationJob` so that when the staleness threshold is
exceeded, it calls `updateJob(... { status: 'error', errorMessage })` to write
the terminal state to the DB row, rather than only returning an error to the
caller.
- Align the timing constants — server worker timeout (2 min), server staleness
threshold (5 min), and client poll cutoff (`MAX_POLL_DURATION`, 2 min in
`system-prompt-form.tsx`) — so a near-2-minute job is not prematurely declared
failed by the UI.

**UI polling robustness
(`components/settings/components/system-prompt-form.tsx`):**
- Stabilize the polling `useEffect` so that a changing `toast` reference does
not reset the interval mid-poll.
- Ensure all terminal branches (error, timeout, and success) consistently clear
the polling interval and reset `jobId`/`isGenerating` state, preventing the form
from ever getting stuck in a generating state.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!