Skip to content

Commit ac3423e

Browse files
ajslaterclaude
andauthored
v1.12.4 (#769)
* docker tag script better than the one in ci/ * update deps * far saner param initialization for browsers and opds start pages * update dpes & bump alpha version * v1.10.5a0 (#551) * bump news * cirlcle ci no longer handles pre-release * remove alpha scripts for circleci" * fix lint-ci * fix pre-relase gha * fix default last route on start pages * bump version * fix default params * alpha2 * regular version 1.10.5 * remove develop circleci builds * adjust variable name * bump version to 6 alpha 0 * fix default params for feed_views in opds 2 * debug logging for django crash * v1.10.6a0 (#553) * bump news * ignore ruff qa for debug build * ignore shellcheck" * lint * minor refactors of opds v2 feed * regular v1.10.6 version * fix docker-tag-latest cscript * ignore gh token file * try to log more request errors * reconfigure logging to hopefully be more verbose about request errors in production * update deps and bump to alpha version * bump news (#555) * v1.10.7 * bump news * update devenv * update devevn and deps * fix pm script * move django-check to test category * workflow build frontend and collect static for prodcution build. fix test upload * bump version and news 1.10.8 * fix dev-module script * fix news * explain news * fix opds clear search setting * fix clear search button * use a registry cache instead of gha cache for the dist-builder * gha use more env vars for image names. retain python dist for 2 days. * new quick deploy gha script. update deps & devenv. * silence watchfiles 5 second timeout debug message * consolidate null values const * make scope private * update devenv * update deps. migrate to unhead v3 * update deps * fix creating reader global settings * fix caching * rename codex build-dist to codex-ci * fix image name. make gha steps depend on each other more. * fix gha syntax errors * names for gha steps * use ghcr.io for python-debian base * update deps * format dockerfile * picopt treestamps * fix custom covers not importing. v1.10.11 * fix custom covers count in admin view * bump news for custom cover count fix * update deps * codex identification in server tag and opds generator tag * update deps * force no entries on opds start page * common opds start page mixin. emtpy group objects on start page * update deps. typechecking. * api change q to search * standardize search param as 'search' instead of 'q' or other variations * remove errant icecream * clear settings on backend * Squashed commit of the following: Fix clear settings null bug, add global settings clear button - clearComicSettings was setting book.settings to null, causing Object.entries() to throw TypeError downstream in getBookSettings - Add null guard in getBookSettings as defense-in-depth - Add null guard in isClearDisabled computed - Add clearGlobalSettings action and clear button to Default Settings panel - Compare against READER_DEFAULTS to determine if global clear is disabled * simplify settings class hierarchy * rename select-many store to browser-select-many * switch to bun. updated devenv * add a claude md * use frozenattrdict to speed up configuration * fix rename of browserSelectMany store * auth token help * fix sort-ignores to make deterministic across shells with different locales * fix crash on settings not being raw * another gaurd for getMetadta() * remove keys from unhead meta headers * fix unhead description for admin tabs" * fix overzealous lazy importer * fix lazyImportEnabled variable in metadata-activator * fix metadata activator from bad cherry pick * fix errant quote * fix typechecking * update devenv & deps * bump news * fix import bug linking folders * fix possible batching crashes. adjust import variables for throughput * batch comic updates * move INTERNAL_IPS setting to general django area * fix typechecking issue * update deps * bump version to v1.10.12 * fix redirect on OPDS alternate view with metadata * minor change to browser empty page for better first time experience * allow browsing to comics with any top group in opds * bring project up to speed with bun and no package-lock.json * fix browser paginator * bump news for browser paginaor fix * fix search combobox clearing * uppercase book close button * version v1.10.13. update deps * fix pdfs not displaying * fix csp for pdfs * fix OPDS FK constraint failure when session row is missing (#607) iOS Panels (and other Basic-Auth OPDS clients) intermittently hit sqlite3.IntegrityError: FOREIGN KEY constraint failed when settings or bookmarks were saved. Two interacting bugs caused it: - Janitor cleanup_sessions used `if not session.get_decoded():` to detect "corrupt" sessions. get_decoded() returns {} for both real decode failures and legitimate anonymous sessions with no stored data — exactly what Basic-Auth OPDS clients produce. The nightly task was wiping valid session rows. Replaced with a direct signing.loads() call so only genuine signature/decode failures are flagged. - _ensure_session_key returned the cookie's session_key without verifying the row still exists. With cached_db the session loads from cache without rechecking, so a stale cookie key would slip through and cause an FK violation when used as SettingsBrowser / SettingsReader.session_id. Now we verify existence and flush+save to cycle the key when the row is gone. Either fix alone closes the user-visible error; both together also stop the underlying churn that created the bad state. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * extend session-key validation to bookmark + reader-settings paths (#608) The same stale-session_key FK-violation pattern existed in two more places that also write rows whose session FK can be stale: - BookmarkAuthMixin.get_bookmark_auth_filter — feeds session_id into Bookmark.objects.bulk_create / bulk_update. - ReaderSettingsBaseView._get_bookmark_auth_filter — feeds session_id into SettingsReader.objects.create. Both used the old `if not session.session_key: save()` pattern that trusts the cookie. Hoist the validated _ensure_session_key helper from SettingsBaseView up to AuthMixin so every auth-aware view shares one implementation, and switch both call sites to it. BookmarkAuthMixin now extends AuthMixin to inherit the helper. BookmarkFilterMixin is unchanged — it's read-only (filter Q only) and a missing session correctly returns no rows. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * format * bump news * Stats: fix user_registered_count / auth_group_count always-zero bug (#611) The /admin/stats endpoint's user_registered_count and auth_group_count fields have been silently returning 0 since at least Sep 2024. The Stats tab in the admin UI shows 0 registered users even on installs with multiple accounts. Root cause: _add_config tried to rename the per-model count keys produced by _get_model_counts: config["user_registered_count"] = config.pop("users_count", 0) config["auth_group_count"] = config.pop("groups_count", 0) But _get_model_counts builds keys via ``snakecase(model.__name__) + "_count"``. For Django's ``django.contrib.auth.models.User`` / ``Group`` that produces ``user_count`` and ``group_count`` (singular). The pop()s with the plural names never matched, so the default ``0`` won every time — and the actual ``user_count`` / ``group_count`` keys were left orphaned in the dict, then dropped by the StatsConfigSerializer which only declares ``user_registered_count`` / ``auth_group_count``. Fix: pop the right source keys. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * defaults for dockerfile ARGs * format' * Frontend correctness: 10 bug fixes from sub-plan 01 (#647) * Reader page: fix load-progress spinner that never appeared Two stacked bugs in the same setTimeout: 1. Non-arrow callback lost ``this``. The function ran with the timer's context, not the component's, so ``this.loaded`` and the write below were both no-ops. 2. The write targeted ``this.loading``, which has never been a data field on this component. The template binds the spinner to ``showProgress`` (line 15: ``v-if="showProgress && !loaded"``). So even if the arrow had been there from the start, the spinner still wouldn't have rendered — both bugs had to land at once. Net: ``LoadingPage`` has been dead code for slow image loads. Switch to an arrow function and write ``showProgress`` instead of ``loading``. Stash the timer ID so ``beforeUnmount`` can clear it; a fast page swap mid-delay would otherwise fire the write on a torn-down component. Implements B1 of tasks/frontend-perf/01-correctness-bugs.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Reader store: fix arc-mtime fallback that itself 500'd ``loadMtimes`` builds an arcs list of ``{ group, pks }`` from ``this.arcs``; if the dict is empty the function previously fell back to ``arcs.push({ r: "0" })``. The comment noted that "No arcs is a 500 from the mtime api" — the fallback was added to dodge that 500 — but the wrong-shape fallback also produced a 500 because the API expects ``group``/``pks`` keys, not ``r``. Use the canonical shape so the fallback actually works. Implements B2 of tasks/frontend-perf/01-correctness-bugs.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * iOS PWA download: pass the object URL to revokeObjectURL, not the Blob ``URL.createObjectURL(blob)`` returns a ``blob:...`` URL string; ``URL.revokeObjectURL`` must receive that same string to free the mapping. The previous code passed ``response.data`` (the Blob itself), which silently no-op'd and leaked one object URL per download. On iOS PWAs this matters more than elsewhere because the leak accumulates across the user's session and can't be reclaimed short of reloading the app. Implements B6 of tasks/frontend-perf/01-correctness-bugs.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Browser API: stop mutating caller's settings in getGroupDownloadURL ``getGroupDownloadURL`` did ``delete settings.show`` on the caller's object before building the URL. Side-effect: any caller that re-used the settings dict after the download-URL build saw its ``show`` key silently vanish. This was probably fine when the function was first written but it's a footgun now that settings flow through a Pinia store. Destructure-and-spread to drop ``show`` without touching the input. Implements B8 of tasks/frontend-perf/01-correctness-bugs.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Auth store: make logout awaitable + clear state unconditionally Two changes to the logout action: 1. ``async`` so callers can ``await``. The current call site (``auth-menu.vue``) fires and forgets, but a future UX pass that wants to disable the button while logout is in flight needs the promise. 2. Clear ``this.user`` in ``finally`` rather than only on success. The user clicked "log out" — UI should reflect the logged-out state immediately, regardless of whether the server-side logout endpoint succeeded. Server-side cookies that survive the network failure will get cleaned up by the next 401. Implements B7 of tasks/frontend-perf/01-correctness-bugs.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Browser filter menu: stop double-rendering each filter row ``<v-list>`` was passed both ``:items="vuetifyItems"`` AND a default-slot ``v-for`` over the same list. Vuetify renders the items prop into ``v-list-item`` children directly, so every row was being built twice — once by the prop, once by the manual ``v-for``. Visible to users on filter menus with large choice lists (genres, characters, etc.); each row appeared duplicated and the DOM cost doubled. Drop the prop. Keep the ``v-for`` because it carries the custom ``#append`` slot for ``metronName`` rendering. ``:model-value`` / ``@update:selected`` still drive selection state via each list- item's ``:value`` prop. Implements B10 of tasks/frontend-perf/01-correctness-bugs.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Admin job-tab: remove API-fetching click handler from expanded panel The expanded status panel had ``@click="loadAllStatuses"`` on its container div. Any click inside the panel — including clicks on child elements that bubbled — refetched the entire status map. Probably copy-pasted as a "refresh on click" gesture, but it fired far too often: a user inspecting a long status list would trigger N API calls just from glancing around. The status data is pushed through the websocket already (socket.js fans librarian notifications into the admin store), so the panel is up-to-date without a manual refresh. Implements B11 of tasks/frontend-perf/01-correctness-bugs.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Reader store: handle bookmark-write errors instead of silently rejecting ``setRoutesAndBookmarkPage`` awaited ``_setBookmarkPage`` but didn't catch its errors. On a network blip the promise rejected, the bookmark didn't persist, and the failure became an unhandled rejection in the browser console — not visible to the user, not retried, just lost. Wrap in try/catch. The local page state stays where it is (the user is reading forward; the bookmark catches up on the next write), but the failure is logged so debugging surfaces. A proper user-visible toast + retry path is broader UX work tracked in the plan. Implements B3 of tasks/frontend-perf/01-correctness-bugs.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Metadata dialog: clear progress timer on unmount ``updateProgress`` chains itself via ``setTimeout`` until the metadata loads or progress reaches 100. The timer ID was never stashed, so closing the dialog mid-animation left the chain running — each tick fired on a torn-down component, writing ``this.progress`` and re-scheduling against now-null refs. Stash the timer ID and clear it in ``beforeUnmount`` so the chain stops cleanly when the dialog goes away. Implements B12 of tasks/frontend-perf/01-correctness-bugs.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Reader pager: key dynamic component on identity to force remount ``<component :is="...">`` without ``:key`` lets Vue reuse the existing instance across an ``is`` change when the components share enough surface (props, name). For the reader's vertical/horizontal pager swap that's wrong: scroll listeners attached by the previous mode persist, the new mode's ``mounted`` runs against stale internal state, and any abort/teardown logic in ``beforeUnmount`` never fires. Add ``:key="component.name"`` so the swap is a true unmount + remount — old listeners go away, new mode starts clean. Implements B13 of tasks/frontend-perf/01-correctness-bugs.md. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Metadata dialog: lint cleanup for the B12 fixup Vue option-order rule: ``beforeUnmount`` belongs above ``methods``. Block-comment style required for the multi-line explanation in ``updateProgress``. Both surfaced when running eslint on the prior commit; pure cleanup, no behavior change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * news for cherry pick * OPDS auth: send WWW-Authenticate + opds-authentication content-type on 401 (#652) Panels and other strict OPDS clients require the WWW-Authenticate header (per RFC 7235) to trigger the auth prompt, and the spec calls for application/opds-authentication+json on the auth document. The exception handler was already converting 403 to 401 with the auth doc body, but was bypassing DRF's natural 401 response and dropping both the header and the proper content type, which Panels read as a forbidden state. Also fix a stray `from re import DEBUG` in the auth view that always forced the absolute-URL path. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix typing inheritence with drf perms * bump version to v1.10.14 * update picopt treestamps * fix typing import error * fix vulture ignorelist * readd the mistakenly removed vulture_ignorelist * v1.10.15 fix show order bug. update deps * fix nightly job manual expansion * fix news version number * V1.11 performance (#714) * title for age rating tagged * rename unrestricted to adult for clarity * remove circleci code * update to new devenv scripts in frontend * fix sort-ignores to make deterministic across shells with different locales * batch comic updates * fix possible batching crashes. adjust import variables for throughput * bump news for v1.11.0 * Add features to readme * add saved view feature * Add browser-views perf measurement harness (Stage 0) (#574) Stand up the measurement infrastructure that gates the rest of the browser-views performance work: - Add django-silk 5.5 to the dev group; install + route it to a separate ``silky`` SQLite DB via a new ``SilkRouter`` so profiler traces never touch the live DB. - Wire SilkyMiddleware below ServeStaticMiddleware so it only wraps the API stack, not static-file responses. - Expose /silk/ under the existing DEBUG URL block. - Add ``tests/perf/run_baseline.py``: hits the live dev DB via ``django.test.Client``, runs three cold/warm flow pairs (root browse, filtered search, series metadata), and writes a JSON baseline artifact. Cachalot + page-cache are invalidated before each cold pass so the numbers reflect actual DB work. - Add ``make perf-baseline`` target. - Commit the per-view analysis docs and initial baseline capture under ``tasks/browser-views-perf/`` for reference during the follow-on cleanup stages. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * Browser views perf: Stage 1 - startup + annotation + filter micros (#575) Bundle of small, surgical wins observed on the slimlib baseline: - Cache `libraries_exist` with a 60s TTL in Django's default cache; invalidate on Library save/delete via lazy-imported signal handlers. Drops a redundant `Library.objects.filter(...).exists()` from every browser response. - Short-circuit the four `_save_browser_*` code paths in `codex/views/settings.py` when nothing actually changed. Avoids gratuitous `.save()` calls that flush cachalot on no-op PATCH writes. - Dedupe the `add_group_by(qs)` call in `codex/views/browser/browser.py`: group once in `_get_common_queryset` (no-op for Comic), drop the second call in `_get_group_queryset`. - Memoize `get_max_bookmark_updated_at_aggregate` per `(model, agg_func, default)` on the view instance — the three callers (group_mtime, order, bookmark) now share one Aggregate. - Move the `bmua_is_max` flag off the per-row `Value` annotation and read it from `self.context["view"]` in the browser serializer. - `@lru_cache(maxsize=256)` on `_preparse_search_query`: extract to a pure module-level helper keyed on `(text, path_allowed)`; returns frozen tokens. - `@lru_cache(maxsize=512)` on `get_field_query`: cache the Lark-parsed Q tree, `copy.deepcopy` on return because `_hoist_filters` mutates `child.negated` downstream. - Stash the `BaseDatabaseOperations(None)` singleton as `_DB_OPS` in `search/field/expression.py`; `prep_for_like_query` doesn't use the connection. - Pre-filter the field loop in `filters/field.py` to keys actually set in the request — saves ~20 no-op calls per browser. - Delete `search/field/optimize.py` (`like_qs_to_regex_q` and friends were already unused — grep confirms only self-references). Slimlib cold baseline (3 flows, stage1.json vs Stage 0 baseline.json): - flow_a_root_browse: 21→18 SQL, 189.6→182.3 ms - flow_b_filtered_search: 21→17 SQL, 187.4→178.0 ms - flow_c_series_metadata: 34→31 SQL, 251.1→226.9 ms Warm paths unchanged (0 SQL, ~2 ms). Full tests + ruff pass. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * fix type warnings * fix lint error * format * Run silk migrations on silky DB at startup (#576) Stage 0 wired a second SQLite DB (silky) for django-silk profiling traces and a DATABASE_ROUTERS entry that routes silk app models there. But ensure_db_schema only invoked `call_command("migrate")` without a database arg, which only migrates the default DB. The router then blocks silk migrations from running on default — so the silky DB was never populated, and the first request through SilkyMiddleware failed with "no such table: silk_request". Mirror the pattern from tests/perf/run_baseline.py: after the default migrate, also run `migrate silk --database=silky` when the silky DB is configured. Guarded on `"silky" in connections.databases` so production (DEBUG=False, no silky DB) is a no-op. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * Browser views perf: Stage 2 — triple COUNT + page mtime cache (#577) ## PR 2a — Eliminate triple COUNT on the paginate path Each browse request ran three COUNT queries per section (groups & books): 1. The outer grouped COUNT in `_get_common_queryset`. 2. Paginator's internal COUNT triggered by `paginator.page()`. 3. An explicit `.count()` on the paginated slice. The outer COUNT is needed (sizing the paginator). The other two are redundant — the page row count is bounded by `per_page` and derivable from `end_index - start_index + 1`. - Shadow `Paginator.count` (a `@cached_property`) with the pre-computed total to skip Paginator's internal COUNT. - Derive page row count arithmetically from `Page.start_index()` / `end_index()`. - Pass `book_count` through `paginate()` alongside `group_count`; drop `book_qs.count()` on the opds2 path. - `_paginate_section` returns `(qs, count)` directly. Short-circuits on `total_count == 0` (avoids Paginator instantiation on empty sections) and preserves the EmptyPage warning branch. ## PR 2b — Short-TTL page mtime cache `BrowserView._get_page_mtime()` calls `get_group_mtime(page_mtime=True)` on every browse request. The query is a filtered Max aggregate that cachalot caches — but any write to Comic / Bookmark invalidates it, so bookmark-heavy usage forces recomputation. Cold-path silk traces show this aggregate at ~26ms on flow_a — the second-slowest query in the request. Add a 5s TTL cache layer gated on page_mtime=True. Key includes user, model, group, pks, page, and a hash of filter-affecting params (filters, search, q, order_by, order_reverse). The polling MtimeView path (no page_mtime) is unaffected, so frontend change-detection stays live. ## Measurements tests/perf/run_baseline.py on the slimlib DB. Cold = full cache invalidation; warm = cachalot populated. | flow | stage1 cold | stage2 cold | |-------------------------|------------------------|------------------------| | flow_a root browse | 18 queries / 182.3 ms | 16 queries / 135.6 ms | | flow_b filtered search | 17 queries / 178.0 ms | 15 queries / 130.3 ms | | flow_c series metadata | 31 queries / 226.9 ms | 31 queries / 229.9 ms | flow_a / flow_b: -2 queries, ~26% cold wall-time reduction. flow_c unaffected (metadata doesn't traverse paginate). PR 2b's benefit is a dogpile guard after cachalot invalidation — doesn't show in the harness (cold = both caches empty; warm = cachalot wins first). Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * fix browser paginator * typecheck browser paginate * Browser views perf: Stage 3 — cover fan-out collapse (#578) * Stage 3 — cover fan-out: per-pk endpoint + pre-resolved cover_pk Before stage 3, every cover card triggered a full BrowserAnnotateOrderView pipeline to pick the representative comic pk. A default page does ~100 of these in parallel. This PR collapses that into one correlated Subquery on the browse response plus a thin per-pk cover endpoint. Components - Option A: annotate_order_aggregates(for_cover=True) drops the JsonGroupArray + page_count aggregates from the cover path — unused for picking a pk. - Option B.1: BrowserView group cards get cover_pk / cover_custom_pk via correlated Subquery that replicates CoverView.get_group_filter exactly (direct fk match when dynamic_covers/Volume/Folder; sort_name fuzzy match otherwise, correlated on _GROUP_BY columns so the same comic set as `ids` is picked — no JSON parsing, no peer aggregate). - New endpoints /api/v3/c/<pk>/cover.webp and /api/v3/cc/<pk>/cover.webp serve already-resolved pks with a cheap single-row ACL probe and the existing CoverPathMixin / CoverCreateThread pipeline. - Frontend getCoverSrc prefers the new per-pk URL when cover_pk / cover_custom_pk is on the card; falls back to the old group+pks URL otherwise, so OPDS and search-active browses keep working. - FTS skip: cover_pk annotation is skipped when params["search"] is set. MATCH inside a correlated subquery re-scans the FTS5 index per outer row (~900ms on a 100-group page). The old URL path still applies the search filter per cover — same behavior, parallelized over HTTP. Perf (slimlib dev DB, Flow D = browse + every card's cover): Flow Before cold After cold Delta A root browse 156.9ms / 15 q 180.5ms / 15 q +24ms / +0 q B search 148.3ms / 16 q 132.4ms / 16 q -16ms / +0 q C metadata 161.9ms / 31 q 165.0ms / 31 q +3ms / +0 q D browse+covers 2311ms / 1216q 1161ms / 815 q -50% / -33% Flow A takes a mild regression for the correlated cover subquery, but Flow D — the realistic user wall-clock — drops by half and 400 SQL queries. Subsequent cover fetches drop from ~90ms each (full pipeline) to ~5ms (disk read + 1 ACL probe). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Stage 3 follow-ups: custom_cover URL, FTS pre-materialization, OPDS thin covers Address review feedback on the cover fan-out collapse: - Rename /api/v3/cc/<pk>/cover.webp → /api/v3/custom_cover/<pk>/cover.webp. Descriptive over terse; matches the view name. - FTS pre-materialization replaces the FTS-skip fallback. A correlated MATCH re-scans FTS5 per outer row (~900ms on a 100-group page); the old path worked around this by skipping cover_pk annotation on search and sending every cover through the legacy pipeline once. We now pre-select the FTS match set as a non-correlated sub-SELECT in the outer cover subquery — SQLite materializes it once and each correlated cover row lookup becomes an indexed pk filter. Cover_pk is annotated on search responses, the thin endpoint handles each cover. - OPDS now emits thin-endpoint cover URLs (v1 and v2): - New OPDSComicCoverByPkView / OPDSCustomCoverByPkView wrap the browser thin views with OPDSAuthMixin's Basic Auth. - New opds:bin:cover_by_pk and opds:bin:custom_cover_by_pk URL names. - v1 _cover_link picks: cover_custom_pk → custom thin; group=='c' → own pk thin; cover_pk → thin; else legacy group+pks fallback. - v2 _thumb always uses the thin endpoint (publications are Comic rows so obj.pk IS the cover pk). - OPDS inherits annotate_cover() via BrowserView._get_group_and_books, so group rows already carry cover_pk / cover_custom_pk. - tests/perf/run_baseline.py gains a flow_e (search + covers) to measure the search path end-to-end. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * fix search combobox clearing * format * Stage 3 follow-ups: folder covers, endpoint rename, offline cover pipeline (#579) - Fix folder cover_pk picking comics not actually in a folder's subtree by using the recursive ``folders`` M2M (same relation the browse filter uses) for the per-card cover subquery instead of the direct ``parent_folder`` FK. - Delete the legacy thick cover endpoint. Rename cover_by_pk.py to cover.py and drop the "_by_pk" suffix from view class names (CoverView, CustomCoverView, OPDSCoverView, OPDSCustomCoverView). Update URL configs, OPDS wrappers, and the frontend client. - Move all cover generation off the HTTP path. Add CoverCreateTask and enqueue it from the importer after bulk_create so new comics get thumbnails pre-warmed offline. When a cached thumb is missing the per-pk endpoint enqueues the task and responds 202 Accepted with Retry-After plus the missing-cover placeholder instead of synthesizing the WebP inline under a worker thread. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * update devenv sort ignores * Force cover refresh after 202 so the placeholder doesn't stick (#580) Browsers don't honor Retry-After on <img> elements and happily cache the placeholder served with the 202 response, so even after the cover thread finishes writing the real thumb the img src keeps rendering the stale placeholder bytes. - Backend sends Cache-Control: no-store on the 202 placeholder so the response isn't cached at that URL. - BookCover now probes the cover URL with fetch() on mount and, if it sees 202, waits Retry-After and bumps a reactive `retry` counter that becomes a cache-busting query param on coverSrc. v-img re-fetches with the new URL and gets the real cover once the cover thread is done. Retries are capped at 5 and aborted on unmount via AbortController. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * Harden cover endpoint against concurrent read/write races (#581) Under a full cold-cache page load, 32 HTTP workers hitting the cover endpoint at the same time as the cover thread was writing thumbs returned 500s for a handful of covers. The endpoint used a three-step exists()/stat()/read_bytes() sequence that could race against an in-flight write, and save_cover_to_cache wrote directly to the target path so readers could observe a truncated file mid-write. - save_cover_to_cache now stages to a ``{name}.{pid}.tmp`` sibling and renames with Path.replace, so a read either sees the pre-existing state or the fully written file — never a partial one. Stale temps are unlinked on failure. - _get_cover_response collapses to a single read_bytes() that catches FileNotFoundError and logs OSError, then falls through to the 202 path. No race window between stat and read. - LIBRARIAN_QUEUE.put and the outer get() methods are wrapped in try/except that log and return the placeholder, so a surprise exception is never user-visible as a 500 and always leaves a log line. - cleanup_orphan_covers runs a dedicated ``*.tmp`` sweep over both cover roots before the orphan scan, so stale temps from crashed writers don't accumulate. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * fix custom cover generation * relegate watch for changes restart to a en env variable settings * await and finish watch task * update devenv * Fix metadata cover fetching pk=0 after Stage 3 cover fan-out (#583) Stage 3's follow-up commit (2ff08524) dropped the legacy `group+pks` fallback from the frontend `getCoverSrc()` helper, leaving it to only build per-pk URLs from `coverPk` / `coverCustomPk`. The browser card pipeline annotates those fields via `BrowserAnnotateCoverView`, but the metadata endpoint was never wired into that annotation, and `metadata-cover.vue` never passed the cover pks through to `BookCover`. As a result every metadata dialog tried to load `/api/v3/c/0/cover.webp` — pk 0 — and fell back to the missing-cover placeholder. Wire the annotation + serializer + component end-to-end: - `MetadataAnnotateView` inherits from `BrowserAnnotateCoverView` so `annotate_cover()` is available on the metadata queryset. - `MetadataView.get_object()` calls `annotate_cover(qs)` after `annotate_card_aggregates(qs)`, mirroring `browser.py`'s ordering. - `MetadataSerializer` exposes `cover_pk` and `cover_custom_pk` as `SerializerMethodField`s with the same fallback-to-`obj.pk` semantics that `BrowserCardSerializer` uses — so Comic metadata (`group=c`) works without annotation, falling back to its own pk. - `metadata-cover.vue` forwards `md.coverPk` / `md.coverCustomPk` to `BookCover`. Verified on /api/v3/{p,i,s,v,c,f}/*/metadata: `coverPk` now resolves to a real comic pk on every group, and the comic path falls back to its own pk. No SQL query delta — `annotate_cover` is a correlated Subquery that inlines into the outer SELECT. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * update deps * Browser views perf: Stage 4 — metadata annotation batch + FK/M2M hints (#582) * Browser views perf: Stage 4 — metadata annotation batch + FK/M2M hydration hints Fold the three `_intersection_annotate()` passes (value fields, conflict shadow fields, related-count fields) into a single batched call that collapses N distinct-count probes into one `aggregate()` and one `values_list()`, with unique synthetic keys so annotation groups don't collide. Extend `FK_QUERY_OPTIMIZERS` so intersecting FK hydration (`AgeRating`, `Character`, `Team`) carries `select_related` + `.only()` hints for the nested fields the metadata serializer actually reads (`AgeRating.metron`, `Character.identifier.url`). Without this, each nested access fired a follow-up query per instance. For the M2M intersection path, short-circuit empty-intersection fields with `Model.objects.none()` so we skip the optimizer setup (and, for the Comic self-reference path, pointless prefetch dispatches on already empty results). Add `flow_c2_comic_metadata` to the perf baseline harness so the comic-detail metadata path (the FK + M2M heavy flow) is tracked alongside the series flow. Measured: flow_c_series_metadata cold SQL drops from 31 → 28 (-3). flow_c2_comic_metadata baseline captured at 47 queries for future stages. Other flows unchanged within noise. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * WATCH for changes variable depends on debug --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * Fix folder + FTS crash and preserve rank-ordered cover selection (#584) * Fix folder browse + FTS crash on search_score ordering Browsing folders (or any group) with an FTS search and ``orderBy=search_score`` blew up with:: sqlite3.OperationalError: no such column: codex_comicfts.rank Root cause: Stage 3's cover fan-out collapse builds a correlated Comic subquery per group card to pick ``cover_pk``. The follow-up (#579) replaced the correlated ``comicfts__match`` filter with a non-correlated ``pk__in fts_sq`` pre-materialization — so the Comic subquery no longer joins ``codex_comicfts``. But ``annotate_order_aggregates`` still annotated ``search_score=ComicFTSRank()`` inside the subquery, and ``add_order_by`` issued ``ORDER BY "codex_comicfts"."rank" * -1``, which SQLite can't resolve from the subquery's FROM list. - ``_annotate_search_scores`` now takes ``for_cover`` and skips the annotation when set. ``annotate_order_aggregates`` threads the flag through (matching the existing ``for_cover`` pipeline-trims). - ``_cover_comic_subquery`` passes ``order_key="sort_name"`` to ``add_order_by`` whenever the user's order is ``search_score`` — the cover's tie-break isn't user-visible, and ``sort_name`` is a real indexed column on Comic. Verified: /api/v3/{r,f,a,p}/0/1?q=iron+man&orderBy=search_score all return 200 with a real ``coverPk`` (not 0). Lint + 20-test pytest suite pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Preserve FTS rank ordering in cover subquery The previous fix short-circuited to ``sort_name`` when ``order_by`` was ``search_score`` so a group card's cover could be picked without referencing ``codex_comicfts.rank``. That fixed the crash but meant the cover chosen for each card was the alphabetically-first matching comic, not the top-ranked FTS match — a UX regression on searched pages. Restore rank-ordered cover selection by: * Applying ``fts_q`` (``comicfts__match=...``) directly in the cover subquery so ``codex_comicfts`` is joined and ``rank`` is populated, while keeping the ``pk__in`` pre-materialization for cheap filtering. * Teaching ``ComicFTSRank`` to resolve the query-local alias for ``codex_comicfts`` at compile time. Its literal template worked for the top-level browse query but emitted an unresolvable column ref in nested subqueries where Django aliases the join as ``V4``/``U1``. * Skipping ``.group_by("id")`` in the cover-subquery search_score annotation. The custom force-group-by compiler emits a literal ``"codex_comic"."id"`` that breaks under nested aliasing, and the cover subquery is already ``.distinct() … LIMIT 1`` so dedup is redundant there. Verified against the dev DB: for each publisher card on a ``batman`` search, the annotated ``coverPk`` now matches the top-ranked comic returned by a direct ``ComicFTSRank`` ordering within that publisher. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * update for claude rules * Browser views perf: Stage 5a — server-side cache_page on cover endpoints (#585) Wrap /api/v3/c/<pk>/cover.webp, /api/v3/custom_cover/<pk>/cover.webp, and their OPDS counterparts in cache_page(COVER_MAX_AGE). Compose with cache_control + vary_on_cookie (API) or vary_on_headers("Cookie", "Authorization") (OPDS, which accepts Basic + Bearer + Session auth) so the Vary header is set before cache_page stores the response — the cache key is keyed per auth identity, no cross-user leakage. Also raise Django FileBasedCache MAX_ENTRIES from the default 300 to 10000. Cachalot query results + cache_page entries (browser + cover) exceed 300 during a single browse-with-covers pageload, triggering the 2/3 random cull that silently evicts just-populated cover entries before the next request can read them. Without this, Flow D warm only dropped to 743 (~50% cover-cache hit rate); with it, Flow D warm drops to 0. Perf impact (stage5a-after.json vs. stage4-after.json): Flow D — browse + 100 covers warm 802 → 0 queries Flow E — search + 46 covers warm 368 → 232 queries Flow E's residual 232 queries are 29 covers that return 202 Accepted (cover not yet generated; response has Cache-Control: no-store, which cache_page correctly skips). The 17 covers that returned 200 were 0 queries each. In production, 202s resolve within seconds as the cover thread catches up, so steady-state warm on Flow E is also 0. Cold query counts and Flow A/B/C are unchanged. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * Raise RLIMIT_NOFILE on startup to avoid macOS 256 FD cap (#586) Cold browser sessions loading a full 100-card grid intermittently crash the dev server with `OSError: [Errno 24] Too many open files`. Diagnosis: - macOS ships a 256 soft cap for RLIMIT_NOFILE (`ulimit -Sn 256`). - Each Django request thread keeps a sticky SQLite connection (`CONN_MAX_AGE=600`). - SQLite WAL mode opens 3 FDs per connection (main + `-wal` + `-shm`). - The thin cover endpoint dispatches to a `sync_to_async` threadpool, so a burst of 100 cold cover requests easily spawns ~100 threads → ~300 SQLite FDs, blowing past the 256 cap before page reads even open. Bumping the soft limit toward the hard cap (or 8192, whichever is lower) at process start is non-invasive and matches what production deployments typically achieve via shell `ulimit`. No-op on Linux (already high) and on platforms without the `resource` module. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * update deps, format json * Browser views perf: Stage 5b — annotation gating + m2m-aware distinct (#587) Three surgical correctness/perf changes that surface the right SQL for the right target. The Stage 5a work (cover endpoint cache_page) collapsed Flow D warm to 0; 5b cleans up cold-path waste the harness flows don't directly exercise. 5.2 — Skip ``updated_ats`` JsonGroupArray outside browser/metadata ``obj.updated_ats`` is consumed only in ``BrowserAggregateSerializerMixin.get_mtime`` (browser + metadata serializers). OPDS computes its own mtime from the bookmark aggregate; cover/download paths never read it. Skip the DISTINCT scalar aggregate for those targets. 5.3 — m2m-aware .distinct() in ``BrowserFilterView.get_filtered_queryset`` New ``comic_filter_uses_m2m`` cached property. Comic queries skip ``.distinct()`` unless a real m2m or m2m-through join is present (story_arc browse, folder browse on cover/choices/bookmark/download TARGETs, or any m2m field filter). Non-Comic queries still always ``.distinct()`` because the ACL alone traverses ``comic__`` (one-to-many). 5.5 — Tie ``search_score`` ``group_by("id")`` to the same flag ``annotate/order.py:_annotate_search_scores`` only emits the GROUP BY when fan-out actually exists. Cover path stays gated by ``for_cover``. 5.4 — Skipped intentionally ``codex/views/auth.py`` already caches the three scalar inputs to ``get_acl_filter`` per request. The remaining cost is composing two Q objects from those scalars — microseconds, not queries. Wrapping a cached_property keyed on ``(model, user_id)`` would be cosmetic. 20 standalone cases of ``comic_filter_uses_m2m`` pass (every default-target group, every m2m-folder TARGET, every BROWSER_FILTER_KEY by category, mixed filters, ``pks=(0,)`` early-out). stage5b-after.json: existing perf flows preserved (query counts identical; wall times within run-to-run noise). The wins are on cold paths the harness doesn't exercise — browsing a Series's comics, default-TARGET folder browse, OPDS feeds — which a follow-up should add. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * Browser views perf: Stage 5c — batched choices_available (#588) Replaces the per-field probe loop in BrowserChoicesAvailableView with a single batched EXISTS annotate. FK fields keep "any non-null exists" semantics; m2m fields decompose into (has_rel, has_null) booleans plus a lazy distinct-count probe for the rare has_rel ∧ ¬has_null corner. The natural EXISTS(SELECT DISTINCT rel ... LIMIT 1 OFFSET 1) form is broken on SQLite — EXISTS short-circuits on the first row from the underlying join, before DISTINCT collapses or OFFSET skips — so the m2m path uses two cheap booleans + a Python-side cap-at-2 distinct probe. Perf: flow_f_choices_available cold drops 34 → 11 queries (−68%) and 121 → 53 ms (−56%) on the dev DB. Other flows are noise-level unchanged. tests/perf/run_baseline.py picks up three new flows (choices_available, m2m field, FK field) so the changed code path is visible in the artifact. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * run frontend with bun * fix syntax error in filter sub menu * Browser views perf: Stage 5d — has_metadata boolean cast + series-browse perf flow (#589) Cleanup bundle (5.7) shrinks against post-5c code: - #26: switch ``has_metadata`` annotation from ``F("metadata_mtime")`` to ``ExpressionWrapper(Q(metadata_mtime__isnull=False), BooleanField())`` in both ``codex/views/browser/annotate/card.py`` and ``codex/views/reader/books.py``. Matches the consumer serializer's ``BooleanField`` and trims the SELECT projection to one byte per row. - Harness: add ``flow_a2_series_browse`` so the harness covers the Comic-queryset / no-m2m-filter path that Stage 5b's distinct + group_by skip wins on. Headline measurement: 6 cold / 4 warm queries, ~13 ms cold. Items #18 (already absorbed in 5c), #25 (parent-aware reduction already in place; trimming further would change ORDER BY semantics), #30 (already coalesced inside ``BookmarkUpdateMixin.update_bookmarks``), and #31 (``zipstream-ng`` already streams via ``FileResponse``) were re-investigated and skipped with reasons documented in §10 of ``05-replan.md``. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * Browser views perf: Stage 5.9 — FTS demote-joins regression guard + 5e handoff (#590) Locks in the conditional ``codex_comicfts`` demote inside ``BrowserFilterView.force_inner_joins`` so future perf work that strips the FTS table from the demote set fails loudly instead of re-introducing ``OperationalError: unable to use function MATCH in the requested context`` on every FTS-enabled browse. Tests in ``tests/test_search_fts.py``: * ``test_left_joined_fts_match_raises_operational_error`` — canonical failure mode. Uses ``Query.promote_joins`` (the documented inverse of ``demote_joins``) to flip the auto-promoted FTS join back to LEFT OUTER, then proves SQLite refuses the MATCH. * ``test_force_inner_joins_demotes_comicfts_when_fts_mode_true`` — positive contract. ``Comic.objects.values("comicfts__pk")`` joins the FTS table LEFT OUTER without a non-null filter (so Django's optimizer leaves it alone), and ``force_inner_joins(fts_mode=True)`` flips it to INNER. * ``test_force_inner_joins_skips_comicfts_when_fts_mode_false`` — negative contract. Same carrier, ``fts_mode=False`` leaves the join LEFT OUTER. * ``test_force_inner_joins_unblocks_match_on_left_joined_query`` — end-to-end repair. The promoted-LEFT-OUTER carrier funneled through ``force_inner_joins(fts_mode=True)`` returns the matching comic. Also lands the Stage 5e handoff doc that scopes the deferred R3 serializer audit (``stage5e-handoff-serializer-audit.md``) and updates ``05-replan.md`` §5 / §10 to reflect Stage 5.9 landing. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * Browser views perf: 99-summary status column + OPDS perf plan (#591) * Browser views perf: Stage 5.9 — FTS demote-joins regression guard + 5e handoff Locks in the conditional ``codex_comicfts`` demote inside ``BrowserFilterView.force_inner_joins`` so future perf work that strips the FTS table from the demote set fails loudly instead of re-introducing ``OperationalError: unable to use function MATCH in the requested context`` on every FTS-enabled browse. Tests in ``tests/test_search_fts.py``: * ``test_left_joined_fts_match_raises_operational_error`` — canonical failure mode. Uses ``Query.promote_joins`` (the documented inverse of ``demote_joins``) to flip the auto-promoted FTS join back to LEFT OUTER, then proves SQLite refuses the MATCH. * ``test_force_inner_joins_demotes_comicfts_when_fts_mode_true`` — positive contract. ``Comic.objects.values("comicfts__pk")`` joins the FTS table LEFT OUTER without a non-null filter (so Django's optimizer leaves it alone), and ``force_inner_joins(fts_mode=True)`` flips it to INNER. * ``test_force_inner_joins_skips_comicfts_when_fts_mode_false`` — negative contract. Same carrier, ``fts_mode=False`` leaves the join LEFT OUTER. * ``test_force_inner_joins_unblocks_match_on_left_joined_query`` — end-to-end repair. The promoted-LEFT-OUTER carrier funneled through ``force_inner_joins(fts_mode=True)`` returns the matching comic. Also lands the Stage 5e handoff doc that scopes the deferred R3 serializer audit (``stage5e-handoff-serializer-audit.md``) and updates ``05-replan.md`` §5 / §10 to reflect Stage 5.9 landing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Browser views perf: backlog status column on 99-summary Stage 5 final exit criterion. Adds a Status column to every tier table in tasks/browser-views-perf/99-summary.md so the backlog reads as a closed ledger rather than an open plan. Markers: - Tier 1 (4): all four landed (Stage 2, 3, 4, 5a) - Tier 2 (6): all six landed (Stage 1 / 5b) - Tier 3 (7): four landed (Stage 2, 5b, 5c), one skipped (#16 subsumed by GroupACLMixin per-request scalar caches), three open (#11, #12 absorbed by Stage 4 hints; #15 not on a hot Flow A-H path) - Tier 4 (14): seven landed (Stage 1 / 5c / 5d), four re-investigated in Stage 5d and confirmed already done (#25, #27, #30, #31), three open (#22, #24, #29) - Tier 5 (3): R1 ✅ Stage 5.9 (FTS demote regression test), R2 ❌ skipped (current code is correct), R3 ⏭️ deferred via the stage5e-handoff-serializer-audit.md brief Pure documentation update — no code or test changes. Closes the last unchecked exit criterion on the Stage 5 plan. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * OPDS views perf: initial planning pass Mirrors the structure of tasks/browser-views-perf/ for the OPDS surface. No code changes — produces a ranked backlog so the same per-stage rigor can be applied to OPDS once browser-views perf wraps. Files: - 00-meta-plan.md methodology, scope, sub-plan layout, exit criteria - 01-routes-and-cache.md OPDS_TIMEOUT=0 disables cache_page on every feed - 02-feed-pipeline.md v1 + v2 main feeds, BrowserView reuse, preview reruns - 03-entry-serialization-v1.md v1 entries, lazy_metadata Comicbox open, M2M fan-out - 04-publications-v2.md is_allowed static-method bypass of admin_flags cache - 05-manifest.md credit fan-out 11→1, story_arcs N+1, subjects 7→1 - 06-progression-binary-aux.md PUT conflict pre-check + dead expr; binary inheritance - 99-summary.md 5-tier ranked backlog, phasing A–F, cross-cutting guidance Top findings (in landing order): 1. OPDS_TIMEOUT = 0 in codex/urls/const.py — every feed wraps cache_page(OPDS_TIMEOUT) and gets nothing. Single-line config flip is the highest-leverage win and gates Phase A. 2. Manifest credit fan-out (v2/manifest.py:194-199) — 11 separate Credit.objects.filter queries because _MD_CREDIT_MAP is iterated. Collapsible to one query + Python partition. 3. is_allowed static method (v2/feed/publications.py:35-56) bypasses the request-cached admin_flags MappingProxyType; called per link spec on start-page render. 4. Manifest M2M subjects — 7 queries via get_m2m_objects loop. UNION or prefetch collapses. 5. get_publications_preview — full BrowserView pipeline rerun per preview link spec on start page. 6. lazy_metadata() Comicbox open in v1 stream-link path — synchronous file I/O on the request thread for partially-imported books. 7. Progression PUT conflict pre-check — two queries per PUT, foldable into one conditional UPDATE. 8. Story arcs N+1 in manifest — .only("story_arc", "number") defers FK; per-row StoryArc.objects.get. Replace with select_related or .values(). Plan only. No source files touched. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * ignore tasks for prettier * OPDS views perf: Stage 0 — baseline harness + Phase B low-risk wins (#592) Closes Phase A (R1 OPDS_TIMEOUT=0 rationale, R2 perf harness) and four of five Phase B items from `tasks/opds-views-perf/99-summary.md`: - #3 Fix story-arc N+1 in `_publication_belongs_to_story_arcs` — swap `.only("story_arc", "number")` for `.select_related("story_arc").only("number", "story_arc__name")` so `story_arc.name` access doesn't fire one query per row. - #6 Convert `OPDS2PublicationBaseView.is_allowed` from `@staticmethod` to instance method reading `self.admin_flags.get("folder_view")`, and the parallel `OPDS1FacetsView._facet_group` anti-pattern (`AdminFlag.objects.get` inside a per-facet loop) — both now use the request-cached MappingProxyType from `SearchFilterView`. - #13 Extract `_obj_ts(obj)` helper for the `floor(datetime.timestamp(obj.updated_at))` expression repeated at six sites across `v2/feed/publications.py` and `v2/manifest.py`. - #15 Remove dead expression `max(position - 1, 0)` (no assignment) at `v2/progression.py:226`. Phase B #12 (`_update_feed_modified` rescan) deferred — it overlaps with the preview-pipeline pass in Phase D. The harness lives at `tests/perf/run_opds_baseline.py` with eleven flows mirroring `tests/perf/run_baseline.py` shape. Cold + warm captures via django-silk; cold pass invalidates cachalot + django_cache. Captures `baseline.json` (pre-edits) and `stage0-after.json` (post-edits) alongside the harness. See `tasks/opds-views-perf/stage0.md` for the full writeup including the harness reproducibility note. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * OPDS views perf: Stage 1 — manifest credit + subject batching (#593) Closes Phase C from `tasks/opds-views-perf/99-summary.md`. Two manifest items land: - Tier 1 #2 — `_publication_credits` collapses 11 per-role `Credit.objects.filter` calls into a single query with `select_related("person", "role")`. Eliminates the 11-query loop AND the lazy `credit.person` FK fan-out triggered by `_add_tag_link` (7 queries on the dev DB's busiest comic). The role-set partition runs in Python. - Tier 2 #5 — `_publication_subject` collapses 7 per-model M2M queries into a single `UNION ALL` over `(pk, name, _kind)` tuples. Reconstructs `SimpleNamespace` rows so `_add_tag_link` and the downstream `OPDS2SubjectSerializer` continue to work. Drive-by: drop the now-unused `get_credits` helper from `codex/views/opds/metadata.py` (no remaining callers). `v2_manifest`: 47 → 24 cold queries (-23, ~49%), 113 → 87 ms cold. Captured `stage1-before.json` + `stage1-after.json` alongside `stage1.md` for the writeup including the surfaced (but not fixed here) `peniciller` typo in `OPDS2PublicationMetadataSerializer`. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * update deps * speling * OPDS views perf: Stage 2 — re-enable route caching (#595) Closes Tier 1 #1 from `tasks/opds-views-perf/99-summary.md` — the highest-impact item in the entire plan. - `OPDS_TIMEOUT` flips from 0 (cache_page no-op) to 60 s. Long enough to amortize a full feed pipeline run across a tab refresh / reader-app re-fetch, short enough that bookmark-position changes show up before the next poll. The disable rationale is reconstructed in stage0.md § R1. - New `codex/urls/opds/__init__.py:opds_cached` helper composes `cache_page(OPDS_TIMEOUT)` with `vary_on_headers("Cookie", "Authorization")` so the cache key scopes per-user / per-auth-scheme. Mirrors the binary cover-route shape at `codex/urls/opds/binary.py`. Applied uniformly across v1.py, v2.py, and the no-trailing-slash `/opds/v2.0` start in root.py (which previously bypassed `cache_page` entirely). - Progression route (`v2/<group>/<pk>/position`) explicitly NOT wrapped — a PUT mutates the bookmark, and a GET within the cache window would return stale position. Multi-device sync is the worst-case (device A PUTs page 100, device B GETs within 60 s and resumes at the wrong page). The ~9-query / 14 ms cold cost is small compared to the freshness cost. Warm-pass measurements collapse to 0 queries / ~1.5–2.7 ms across every cacheable route (the cache returns the response without entering the view layer): v2_manifest 62 → 2.4 ms warm v2_start 59 → 1.8 ms warm v2_root_browse 28 → 2.4 ms warm v1_root_browse 40 → 1.7 ms warm v1_series_acquisition 24 → 2.7 ms warm Cold-pass numbers unchanged — the view runs the full pipeline on a cache miss. Real-world OPDS traffic is dominated by warm hits. Cross-user isolation verified manually: User A and User B (different ACL) hit the same URL and receive different payloads (16 223 B vs 15 217 B); each user's warm response matches their own cold response; `Vary: Accept, Cookie, Authorization, origin` confirmed on every response. Captured `stage2-before.json` + `stage2-after.json` alongside `stage2.md` for the writeup. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * disable discord notification steps in gha * OPDS views perf: Stage 3 — preview pipeline cache sharing + book queryset joins (#596) Closes Tier 1 #4 (preview-pipeline re-runs) and the related #17 / sub-plan 02 #3 (`select_related` shortfall on the OPDS book queryset). - `OPDS2FeedLinksView.get_book_qs` overrides the parent `BrowserView.get_book_qs` to add `select_related("volume", "language")`. The base method joins `series` only, with an explicit comment that "OPDS doesn't need volume" — but `Comic.get_title(volume=True)` reads `obj.volume.name` / `obj.volume.number_to` per publication, and `_publication_metadata` reads `obj.language.name`. Without these joins, every publication iteration fired one lazy `Volume.objects.get` and one lazy `Language.objects.get`. v1 already does the same join in `v1/facets.py:64`; this brings v2 to parity. - `_get_publications_preview_feed_view` shares `_admin_flags` and `_cached_visible_library_pks` with the parent view. Both are request-scoped (depend on user, not on params/kwargs), so it's safe to skip the per-preview re-fetch. Cuts the visible-library ACL lookup from 1-per-preview to 1-per-request. `v2_start`: 53 → 29 cold queries (-24, ~45% reduction). Pre-fix breakdown: 15 codex_volume (N+1) + 12 codex_library (4 per preview × 3) dominated. Post-fix: 0 codex_volume + 3 codex_library. Out-of-scope hotspots still visible in the trace (documented in stage3.md): per-preview age-rating-with-metadata join (4 queries, needs the bigger UNION-batch rewrite) and the 9 codex_comic filter / annotation queries (preserved as the legitimate per-preview pipeline cost). Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * OPDS views perf: Stage 4 — v1 acquisition M2M batching + progression PUT conflict detection (#597) Closes Phase E from `tasks/opds-views-perf/99-summary.md`. #7 — Per-page batching of v1 entry M2M fan-out ============================================== Three new helpers in `codex/views/opds/metadata.py`: - `get_credit_people_by_comic` — one query for all credits across all comics on the page; partitions by comic_id in Python. - `get_m2m_objects_by_comic` — UNION ALL across the 7 `OPDS_M2M_MODELS` tables, partitioned by `(comic_id, kind)`. `OPDS1EntryData` gains `authors_by_pk` / `contributors_by_pk` / `category_groups_by_pk` optional dicts. `OPDS1FeedView._get_entries_section` populates them when `metadata=True` and `key=="books"`. Per-entry properties read from the dicts when present, fall back to the legacy single-comic helpers otherwise (so facet entries / single-comic feeds still work). Result: 9 queries per entry × N entries collapses to 3 queries per page. On the harness's "All Batman" series with `?opdsMetadata=1` (106 comics), `v1_acquisition_with_metadata` drops from 817 cold queries / 1585 ms to **20 cold queries / 154 ms** (~40× / ~10×) — verified in a controlled full-feed-state run. #8 — Progression PUT atomic conditional UPDATE ============================================== Two changes: 1. `OPDS2ProgressionSerializer.modified` flips from `read_only=True` to `required=False`. Previously the field was silently dropped from PUT validated_data, making the conflict pre-check at `view.py:207-217` unreachable (zero progression-related queries fired on PUT today). 2. `OPDS2ProgressionView.put` replaces the dead pre-check (which was `_get_bookmark_query() + qs.first()` — never executed) with a single atomic conditional UPDATE keyed on `updated_at__lte=new_modified`. If the UPDATE matches a row, write succeeds in one query. If 0 rows match AND a bookmark exists, the DB has a fresher row → 409. If 0 rows match AND no bookmark exists, fall through to the existing async `update_bookmark` path (first-time write). Behavior change: clients that previously sent stale `modified` and got silent 200s now correctly receive 409 per the OPDS v2 progression spec. Functional verification: - PUT no `modified` (no bookmark) → 200 (liberal accept, async create) - PUT with stale `modified` → 409 (atomic conflict detection) - PUT with fresh `modified` → 200 (atomic UPDATE) Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * bump news for progression bugfix * OPDS views perf: Stage 5 — Tier 3-4 cleanups (#598) Closes Phase F from `tasks/opds-views-perf/99-summary.md`. Three items land; two are intentionally skipped after audit. #16 — `select_related("parent_folder")` on manifest queryset `OPDS2ManifestView.get_object` adds the join; eliminates the lazy `Folder.objects.get` per request when folder_view is on. v2_manifest cold drops 23 → 22 queries. #11 — Memoize filters JSON via `self.params["filters"]` `_subtitle_filters` previously re-parsed `request.GET["filters"]` inline (urllib.parse.unquote + json.loads). The same JSON is already parsed by BrowserSettingsFilterInputSerializer; reading from `self.params` skips the third parse. Sub-ms per request, but on a hot client refresh cumulative. #10 — Resolve `opds:bin:page` URL once for `_publication_reading_order` Replace per-page `self.href()` (which fires `reverse()` each call) with a single sentinel-page resolution + `str.format` substitution. Saves N-1 `reverse()` calls per manifest hit. Invisible on the harness's 1-page comic_pk=10785; visible on high-page-count PDFs. #12 (audited won't-fix): the rescan is functionally necessary — preview-group mtimes aren't covered by `_get_group_and_books`'s mtime; removing the rescan would lose preview mtime tracking. #18 (audited won't-fix): Stage 4 already provides the cleaner SimpleNamespace pattern; legacy `_add_url_to_obj` fires only on cold fallback paths where the cachalot reuse risk doesn't apply (materialized list, not a queryset reused for caching). After Stage 5 the OPDS perf project has reached the point where remaining open items either need production telemetry (R3, #19), medium-risk correctness verification (#9), or are negligible wins versus framework cost (#14). Recommending pause until production data points to a specific path. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * Reader views perf plan: methodology + 3 sub-plans + ranked backlog (#599) Audit of `codex/views/reader/` (~850 LOC across 6 source files) mirroring the OPDS / browser-views perf plan structure. Identifies 15 ranked items + 5 research questions across three view families: - Reader view chain (`c/<pk>` GET) — params/arcs/books/reader.py - Reader settings (`c/settings`, `c/<pk>/settings`) — settings.py - Reader page binary (`c/<pk>/<page>/page.jpg`) — page.py Top three findings: 1. Comicbox archive open on every page request (sub-plan 03 #1) — 200-page comic = 200 archive opens per read-through. The biggest single hotspot; needs an LRU process-cache of open archives. 2. `get_book_collection` materializes the entire arc to find prev/curr/next (sub-plan 01 #1) — 100-issue series = up to 100 rows materialized per reader open. Two targeted LIMIT-1 queries collapse to 3 rows independent of arc size. 3. No server-side caching on the reader / page / settings routes — every request re-runs the full pipeline. Suggested landing order matches the OPDS / browser shape: Phase A measure (build a perf harness mirroring `tests/perf/run_opds_baseline.py`), Phase B low-risk per-request wins, Phase C `get_book_collection` rewrite, Phase D route caching, Phase E page-endpoint optimization (blocked on production traffic data), Phase F cleanups. Plans written but no implementation yet — Phase A is the next step. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * Reader views perf: Stage 0 — baseline harness + Phase B low-risk wins (#600) Closes Phase A (R1: harness, R2: baseline) and Phase B (#5, #9, #10, #13, #14) of `tasks/reader-views-perf/99-summary.md`. Phase A — measure first ======================= Built `tests/perf/run_reader_baseline.py` (mirrors `run_opds_baseline.py`). Seven flows covering the three reader view families: - reader_open + reader_open_large_arc — `c/<pk>` GET - settings_global + settings_multiscope — `c/<pk>/settings` GET - page_first + page_middle + page_no_bookmark — `c/<pk>/<page>/page.jpg` Cold-then-warm methodology via django-silk. Captured `stage0-before.json` and `stage0-after.json` artifacts. Phase B — low-risk wins ======================= #5 — `_get_field_names` hoists settings + admin-flag reads. - Hoisted the per-iteration `get_from_settings("show", browser=True)` call outside the loop. Was 2 SettingsBrowser queries per call; now 1. - The `AdminFlag.objects.get(folder_view)` lookup is now a local `@cached_property`. The reader chain doesn't inherit SearchFilterView (the OPDS / browser `self.admin_flags` source); a local cached_property is the smallest equivalent. Plan misjudgment corrected — see stage0.md. #9 — `Model.objects.get_or_create` over the manual two-query pattern in `_get_global_settings` and `_get_or_create_scoped_settings`. Atomic + race-aware. #10 — `get_reader_default_params` decorated with `@classmethod` + `@cache`. Pure model metadata; doesn't change at runtime. #13 — `_set_selected_arc` pre-builds `frozenset(requested_arc_ids)` outside the inner loop. Drive-by correctness fix: the original loop variable leaked the last iterated value as `arc_ids` even on no-match, so the fallback `if not arc_ids:` only triggered on empty `all_arc_ids`. Iteration now uses a separate `candidate` variable so the fallback fires correctly on no-match. #14 — Drop redundant `.distinct()` on the page-endpoint queryset. The next call is `.get(pk=…)` LIMIT 1, which collapses any ACL JOIN duplicates anyway. Headline numbers ================ reader_open …
1 parent 3c4bffa commit ac3423e

21 files changed

Lines changed: 565 additions & 110 deletions

NEWS.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,13 @@ width: 128px;
66
border-radius: 128px;
77
" />
88

9+
## v1.12.4
10+
11+
- Fixes
12+
- Crash on Cleanup Favorites Janitor Job.
13+
- Features
14+
- Force Update Tags by individual group.
15+
916
## v1.12.3
1017

1118
- Fixes
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
"""Force a metadata re-import for an explicit set of comic pks."""
2+
3+
from collections import defaultdict
4+
5+
from codex.librarian.scribe.importer.tasks import ImportTask
6+
from codex.librarian.scribe.tasks import ForceUpdateComicsTask
7+
from codex.librarian.worker import WorkerBase
8+
from codex.models.comic import Comic
9+
10+
11+
class ForceUpdater(WorkerBase):
12+
"""Dispatch ImportTasks scoped to a caller-supplied set of comic pks."""
13+
14+
def force_update(self, task: ForceUpdateComicsTask) -> None:
15+
"""Group comic pks by library and dispatch a forced ImportTask per library."""
16+
if not task.comic_pks:
17+
self.log.debug("Force update called with no comic pks.")
18+
return
19+
20+
comics = Comic.objects.filter(pk__in=task.comic_pks).only("path", "library_id")
21+
library_path_map: defaultdict[int, set[str]] = defaultdict(set)
22+
for comic in comics:
23+
library_path_map[comic.library_id].add(comic.path) # pyright: ignore[reportAttributeAccessIssue]
24+
25+
total = 0
26+
for library_id, paths in library_path_map.items():
27+
files_modified = frozenset(paths)
28+
if not files_modified:
29+
continue
30+
import_task = ImportTask(
31+
library_id=library_id,
32+
files_modified=files_modified,
33+
force_import_metadata=True,
34+
check_metadata_mtime=False,
35+
)
36+
self.librarian_queue.put(import_task)
37+
total += len(files_modified)
38+
self.log.info(f"Force update queued for {total} comics.")

codex/librarian/scribe/priority.py

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
JanitorCleanCoversTask,
1212
JanitorCleanFKsTask,
1313
JanitorCleanupBookmarksTask,
14+
JanitorCleanupFavoritesTask,
1415
JanitorCleanupSessionsTask,
1516
JanitorCleanupSettingsTask,
1617
JanitorCodexUpdateTask,
@@ -29,6 +30,7 @@
2930
SearchIndexSyncTask,
3031
)
3132
from codex.librarian.scribe.tasks import (
33+
ForceUpdateComicsTask,
3234
ImportAbortTask,
3335
LazyImportComicsTask,
3436
ScribeTask,
@@ -47,14 +49,16 @@
4749
JanitorFTSIntegrityCheckTask,
4850
JanitorFTSRebuildTask,
4951
JanitorImportForceAllFailedTask,
52+
ForceUpdateComicsTask,
5053
ImportTask,
5154
LazyImportComicsTask,
5255
UpdateGroupsTask,
5356
JanitorCleanFKsTask,
5457
JanitorCleanCoversTask,
5558
JanitorCleanupSessionsTask,
56-
JanitorCleanupSettingsTask,
5759
JanitorCleanupBookmarksTask,
60+
JanitorCleanupSettingsTask,
61+
JanitorCleanupFavoritesTask,
5862
SearchIndexClearTask,
5963
SearchIndexCleanStaleTask,
6064
SearchIndexSyncTask,

codex/librarian/scribe/scribed.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
from queue import PriorityQueue
55
from typing import Final, override
66

7+
from codex.librarian.scribe.force_updater import ForceUpdater
78
from codex.librarian.scribe.importer.importer import ComicImporter
89
from codex.librarian.scribe.importer.tasks import ImportTask
910
from codex.librarian.scribe.janitor.adopt_folders import OrphanFolderAdopter
@@ -22,6 +23,7 @@
2223
)
2324
from codex.librarian.scribe.tasks import (
2425
CleanupAbortTask,
26+
ForceUpdateComicsTask,
2527
ImportAbortTask,
2628
LazyImportComicsTask,
2729
SearchIndexSyncAbortTask,
@@ -74,6 +76,11 @@ def process_item(self, item) -> None:
7476
self.log, self.librarian_queue, self.db_write_lock
7577
)
7678
worker.lazy_import(task)
79+
case ForceUpdateComicsTask():
80+
worker = ForceUpdater(
81+
self.log, self.librarian_queue, self.db_write_lock
82+
)
83+
worker.force_update(task)
7784
case UpdateGroupsTask():
7885
worker = TimestampUpdater(
7986
self.log, self.librarian_queue, self.db_write_lock

codex/librarian/scribe/tasks.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,13 @@ class LazyImportComicsTask(ScribeTask):
2525
pks: frozenset[int]
2626

2727

28+
@dataclass
29+
class ForceUpdateComicsTask(ScribeTask):
30+
"""Force a metadata re-import for a specific set of comic pks."""
31+
32+
comic_pks: frozenset[int]
33+
34+
2835
class ImportAbortTask(ScribeTask):
2936
"""Abort Import."""
3037

codex/urls/api/browser.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
from codex.views.browser.browser import BrowserView
99
from codex.views.browser.choices import BrowserChoicesAvailableView, BrowserChoicesView
1010
from codex.views.browser.download import GroupDownloadView
11+
from codex.views.browser.force_update import ForceUpdateView
1112
from codex.views.browser.metadata import MetadataView
1213
from codex.views.browser.saved_settings import (
1314
SavedBrowserSettingsListView,
@@ -81,4 +82,12 @@
8182
LazyImportView.as_view(),
8283
name="import",
8384
),
85+
#
86+
#
87+
# Force Update Tags
88+
path(
89+
"<int_list:pks>/force_update",
90+
ForceUpdateView.as_view(),
91+
name="force_update",
92+
),
8493
]

codex/views/browser/filters/filter.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@
3535
# ``comic__folders`` (download) rather than ``parent_folder`` (FK).
3636
# Mirrors the branches in :meth:`GroupFilterView._get_rel_for_pks`.
3737
_M2M_FOLDER_GROUP_TARGETS: Final[frozenset[str]] = frozenset(
38-
{"cover", "choices", "bookmark", "download"}
38+
{"cover", "choices", "bookmark", "download", "force_update"}
3939
)
4040

4141
# Group codes whose transitive-favorite clauses introduce m2m JOINs on

codex/views/browser/filters/group.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
GROUP_RELATION,
1212
)
1313

14-
_GROUP_REL_TARGETS: Final = frozenset({"cover", "choices", "bookmark"})
14+
_GROUP_REL_TARGETS: Final = frozenset({"cover", "choices", "bookmark", "force_update"})
1515
_PK_REL_TARGETS: Final = frozenset({"metadata", "mtime"})
1616

1717

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
"""Force update browser view: re-import metadata for a filtered comic set."""
2+
3+
from collections.abc import Sequence
4+
from types import MappingProxyType
5+
from typing import override
6+
7+
from drf_spectacular.utils import extend_schema
8+
from rest_framework.permissions import BasePermission, IsAdminUser
9+
from rest_framework.response import Response
10+
11+
from codex.librarian.mp_queue import LIBRARIAN_QUEUE
12+
from codex.librarian.scribe.tasks import ForceUpdateComicsTask
13+
from codex.models.comic import Comic
14+
from codex.serializers.mixins import OKSerializer
15+
from codex.views.browser.filters.filter import BrowserFilterView
16+
17+
18+
class ForceUpdateView(BrowserFilterView):
19+
"""Force a metadata re-import for every comic under a group + filters."""
20+
21+
permission_classes: Sequence[type[BasePermission]] = (IsAdminUser,)
22+
serializer_class = OKSerializer
23+
TARGET: str = "force_update"
24+
25+
def __init__(self, *args, **kwargs) -> None:
26+
"""Init acl properties."""
27+
super().__init__(*args, **kwargs)
28+
self.init_group_acl()
29+
30+
@property
31+
@override
32+
def params(self):
33+
"""Retrieve params from the request without saving them to settings."""
34+
if self._params is None:
35+
params = self.load_params_from_settings()
36+
self._params = MappingProxyType(params)
37+
return self._params
38+
39+
@extend_schema(request=None, responses=serializer_class)
40+
def post(self, *_args, **_kwargs) -> Response:
41+
"""Enqueue a force-update task for every comic matching the filters."""
42+
group = self.kwargs.get("group")
43+
pks = self.kwargs.get("pks")
44+
comic_pks = frozenset(
45+
self.get_filtered_queryset(Comic, group=group, pks=pks).values_list(
46+
"pk", flat=True
47+
)
48+
)
49+
if comic_pks:
50+
task = ForceUpdateComicsTask(comic_pks=comic_pks)
51+
LIBRARIAN_QUEUE.put(task)
52+
return Response()

eslint.config.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ export default defineConfig([
4545
ignoreContent: [
4646
"notify_groups_changed",
4747
"notify_failed_imports_changed",
48+
"ForceUpdateConfirmDialog",
4849
],
4950
},
5051
],

0 commit comments

Comments
 (0)