feat: cache reader content images for offline access

[anpryl]

Summary

Adds a new contentImageWorker that downloads images referenced in extracted reader content HTML, saves them as CONTENT_IMAGE assets, and rewrites <img src> URLs to point to /api/assets/{assetId} — so reader view images survive even when the original source goes offline or blocks hotlinking.

Closes #1563
Closes #2363

Key design decisions

• Default OFF — gated behind CRAWLER_STORE_CONTENT_IMAGES=false env var so existing users aren't surprised by extra storage/bandwidth
• Separate worker — not inline in the crawler, for reliability isolation (a single bad image doesn't block the crawl)
• Sequential downloads per bookmark to reduce rate-limiting from origin servers
• Deterministic asset IDs — SHA-256(bookmarkId:sourceUrl).slice(0,32) enables skip-if-exists on retries and idempotent re-crawls
• Per-image exponential backoff — up to 10 retries, 1s initial / 30s max cap (~3 min worst case per image)
• Browser-like request headers — Chrome UA, Accept: image/*, Referer from source page
• Magic bytes detection — falls back to file signature detection when servers return wrong Content-Type
• Extended format support — JPEG, PNG, GIF, WebP + SVG, AVIF, APNG (worker-scoped CONTENT_IMAGE_ASSET_TYPES, does not alter global IMAGE_ASSET_TYPES)
• Content images are system-managed — hidden from AttachmentBox, isAllowedToAttach/Detach both return false
• Stale image cleanup on re-crawl — after successfully downloading new images and rewriting HTML, assets no longer referenced in the current HTML are deleted (both file and DB row). Only runs after at least one image was successfully cached so partial failures preserve old working images.

Changes

New files

• apps/workers/workers/contentImageWorker.ts — the worker (extraction, download, rewrite, stale cleanup)
• apps/workers/workers/contentImageWorker.test.ts — 91 unit tests
• apps/workers/workers/contentImageWorker.integration.test.ts — 5 integration tests with real HTTP server
• packages/db/drizzle/0081_add_content_image_status.sql — adds nullable contentImageStatus column to bookmarkLinks

Modified files

• packages/db/schema.ts — contentImageStatus column (pending/failure/success), CONTENT_IMAGE asset type
• packages/shared/config.ts — 5 new env vars for feature configuration
• packages/shared/assetdb.ts — SUPPORTED_CONTENT_IMAGE_TYPES set, optional supportedTypes param on saveAsset (both store implementations)
• packages/shared-server/src/queues.ts — ContentImageQueue definition
• packages/trpc/routers/admin.ts — content image stats, bulk recacheContentImages mutation, per-bookmark adminRecacheContentImagesBookmark mutation, contentImageStatus in debug info output
• packages/trpc/models/bookmarks.ts — contentImageStatus in buildDebugInfo response
• packages/trpc/stats.ts — content image queue stats
• packages/trpc/index.ts — export content image queue
• packages/trpc/lib/attachments.ts — CONTENT_IMAGE not user-attachable
• apps/workers/workers/crawlerWorker.ts — enqueue content image job after crawl completes
• apps/workers/index.ts — register content image worker
• apps/web/components/admin/BackgroundJobs.tsx — Content Image Jobs card (hidden when feature disabled) with pending/failed counters, bulk recache actions
• apps/web/components/admin/BookmarkDebugger.tsx — per-bookmark "Re-cache images" action button, Image Crawl Status badge in status section
• apps/web/components/dashboard/preview/AttachmentBox.tsx — filter CONTENT_IMAGE assets from user-facing attachment list
• apps/web/lib/i18n/locales/en/translation.json — i18n strings
• apps/web/lib/i18n/locales/en_US/translation.json — i18n strings
• docs/docs/03-configuration/01-environment-variables.md — new env var docs
• docs/docs/08-development/04-architecture.md — updated worker count

Image extraction capabilities

• Lazy-load attributes (12 total): data-src, data-actualsrc, data-srv, data-original, data-lazy, data-lazy-src, data-lazyload, data-img-src, data-url, data-hi-res-src, data-highres, data-full-src
• Srcset parsing: srcset, data-srcset, data-lazy-srcset — picks largest candidate by width/density
• SVG support: <image href> and <image xlink:href> extraction and rewriting
• Cleanup: strips srcset attrs, removes <source> inside <picture>, cleans lazy-load attrs after rewriting

Configuration

Env var	Default	Description
CRAWLER_STORE_CONTENT_IMAGES	false	Enable/disable content image caching
CRAWLER_CONTENT_IMAGE_MAX_COUNT	50	Max images to cache per bookmark
CRAWLER_CONTENT_IMAGE_MAX_SIZE_MB	5	Max size per image in MB
CONTENT_IMAGE_NUM_WORKERS	1	Number of content image worker instances
CONTENT_IMAGE_JOB_TIMEOUT_SEC	120	Job timeout in seconds

Test plan

• 91 unit tests covering extraction, rewriting, download, magic bytes, run pipeline, stale cleanup
• 5 integration tests with real HTTP server serving minimal valid images
• Manual test: enable feature, bookmark a page with images, verify images are cached and visible in reader view
• Manual test: take source page offline, verify cached images still render
• Manual test: re-crawl a bookmark, verify stale images are cleaned up and new images are cached
• Manual test: admin panel shows correct pending/failed counters and bulk recache works
• Manual test: bookmark debugger shows Image Crawl Status and per-bookmark Re-cache images button works

Known limitations

• Re-cache images without a preceding re-crawl is a no-op if images were already rewritten (HTML contains /api/assets/ URLs, not external URLs). Use Re-crawl first to fetch fresh HTML.
• maxCount truncation may cause images beyond the limit to never be cached
• Bookmark deletion cascades DB rows but not asset files (handled by existing tidyAssets maintenance)
• Video poster and CSS background-image not handled (intentionally scoped out)

Migration

DB migration 0081_add_content_image_status adds a nullable contentImageStatus column to bookmarkLinks. No data migration needed — existing rows default to NULL (not processed).

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c97fbae4-cf7c-407f-ab9b-4b28cc206ef1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}