Scraper jobs for Devfolio and Unstop or other platform

[ShantKhatri]

Summary

Pull hackathon listings automatically via a cron job.

Contexts

Needs to respect rate limits from both platforms, called out explicitly in the spec.

Tasks

• Build scraper for Devfolio listings
• Build scraper for Unstop listings
• API fetch from the developer.events API
• Add rate limiting / backoff to avoid getting blocked
• Schedule as a recurring cron job

Acceptance Criteria

• Both scrapers run on schedule without hitting rate limits
• Scraped entries land correctly in the hackathon table
• Tests added
• docs updated

Area

backend

Difficulty

Advanced

[github-actions]

Hi @ShantKhatri,

Thanks for opening this issue.

Please reply with one of the following areas so the issue can be routed to the appropriate team member:

- /backend
- /web
- /mobile
- /devops

Once an area is selected, the corresponding label will be added automatically.

[Darko893]

On the rate limiting item, one thing that is easy to get wrong and expensive to learn: the damage is not from how many requests you make in total, it is from how many you send to one site in a short window.

I found that out the unpleasant way today. Testing against two retailers, a few hundred requests each over an afternoon, took one of them from reading 7 times in 10 to 0 in 6 through a residential pool that had been fine that morning. A different site the same pool had barely touched stayed at 6 out of 6. Nothing about the requests changed, only how many of them one host had seen.

For a scheduled job across Devfolio and Unstop that suggests three things:

• space requests to one host rather than capping total throughput. A shared queue per domain, a few a minute, and let everything else wait its turn.
• cache the fetched page for a few minutes. If two jobs want the same listing page, fetch once. That is the single biggest reduction in knocks on the door.
• back off per host on refusals. Three 403s in a row from one site and leave it alone for a while, rather than retrying into the same wall and teaching it your addresses.

Devfolio's listings read fine at the moment, for what it is worth. First one currently open:

{"name": "Push to Prod Hackathon: Building a...", "uuid": "e1465a24066e44509285410b254b39fb"}

Disclosure: I build an extraction API called Haunt (https://hauntapi.com?utm_source=github&utm_medium=comment&utm_campaign=issues-2026-07-26b), so I have an interest. The rate limiting advice is from measuring my own mistake this afternoon and applies whatever you build with.