Skip to content

fix(Outreach): delete stale scraper_results.csv before each run#260

Open
octo-patch wants to merge 1 commit into
FujiwaraChoki:mainfrom
octo-patch:fix/issue-256-stale-scraper-results
Open

fix(Outreach): delete stale scraper_results.csv before each run#260
octo-patch wants to merge 1 commit into
FujiwaraChoki:mainfrom
octo-patch:fix/issue-256-stale-scraper-results

Conversation

@octo-patch
Copy link
Copy Markdown

Fixes #256

Problem

Outreach.start() runs the scraper and then checks os.path.exists(output_path) to determine if scraping succeeded. However, .mp/scraper_results.csv always points to a fixed path (get_results_cache_path()). If the file already exists from a previous successful run, this check passes even when the current scraper invocation fails or times out — causing the outreach flow to send emails based on stale lead data.

Solution

Delete scraper_results.csv before invoking the scraper. After the scraper finishes, the existing os.path.exists() check is now reliable: the file can only exist if the current run produced it.

Testing

  1. Run Outreach once so .mp/scraper_results.csv is created.
  2. Cause the next scraper invocation to fail (e.g. kill the process or pass an invalid niche).
  3. Before this fix: the stale CSV is found and outreach emails are sent to old leads.
  4. After this fix: the stale CSV is removed before the run; when the scraper fails no new file is written; os.path.exists() returns False and the error path is taken correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(Outreach): stale scraper_results.csv can be reused after scraper failure or timeout

1 participant