Remove the Digger feature entirely#374
Merged
Merged
Conversation
Discogs deployed Cloudflare Bot Management across www.discogs.com, which hard-blocks (HTTP 403) every marketplace HTML request regardless of User-Agent — confirmed from the deployment host. The official Discogs API exposes no release→listings endpoint (only aggregate /marketplace/stats and per-seller /users/{username}/inventory), so the seller-bundling feature that was Digger's entire purpose cannot be restored without defeating the challenge. Rather than ship a permanently non-functional service, remove it. Removed surface area: - digger/ worker service (scraper + scheduler) and its Dockerfile - API: routers (digger, digger_agent, digger_proposals, digger_recommend, digger_reports, internal_digger), api/digger_agent/, api/digger_refresh/, digger queries, Digger* models, the service-token guard, and the wantlist-priority seeding hook in syncer - common/digger_optimizer/ and the DiggerConfig + digger/anthropic settings - schema-init/digger_schema.py and its wiring in postgres_schema.py - mcp-server digger_tools and MCP_API_TOKEN plumbing - Explore UI: Digger nav tab, pane, settings card, and JS (digger.js, api-client/settings methods, styles) - docker-compose (dev + prod) digger service, secrets, and anthropic key - CI (test/list-sub-projects/docker-compose-validate), justfile targets, pyproject workspace member/extra, perftest scenarios + auth machinery - docs: digger-agent/optimizer/scraping-policy pages and all references - all digger tests (worker, api, optimizer, eval, schema, perftest) The git history retains everything if the feature is ever resurrected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Contributor
E2E Coverage (chromium)
|
Contributor
E2E Coverage (webkit)
|
Contributor
E2E Coverage (firefox)
|
Contributor
E2E Coverage (webkit - iPhone 15)
|
Contributor
E2E Coverage (webkit - iPad Pro 11)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Reviewing production logs (
discogsography-digger) showed every Discogs marketplace scrape returning HTTP 403 — 5,896 requests, 100% blocked. Probing from the deployment host confirmed Discogs has deployed Cloudflare Bot Management (cf-mitigated: challenge) across all ofwww.discogs.com(the homepage 403s too). No plain HTTP client can pass it, regardless of User-Agent.A spike against the official API confirmed there's no viable replacement:
/marketplace/stats/{release_id}→ aggregate lowest-price + count only (no sellers)/users/{username}/inventory→ per-seller, but no way to discover which sellers hold a releaseDigger's entire purpose was per-seller bundle optimization, which depends on that mapping. Rather than keep a permanently non-functional service (and a scheduler hammering a hard-blocking endpoint), this removes the feature entirely. Git history retains everything if it's ever resurrected via a headless-browser/proxy approach.
What's removed
digger/(scraper + scheduler) + Dockerfileapi/digger_agent/,api/digger_refresh/, digger queries,Digger*models, theX-Service-Tokenguard, and the wantlist-priority seeding hook in the syncercommon/digger_optimizer/+DiggerConfig/digger/anthropic settingsschema-init/digger_schema.py+ its wiringdigger_tools+MCP_API_TOKENplumbingAlso drops now-unused deps (
pulp,bleach,selectolax) and a stray transitivecommonPyPI package from the lock.Verification
uv lock --checkconsistent · both compose files validate197 files changed, −31,606 lines.
🤖 Generated with Claude Code