Skip to content

Add FastFiltersLib, SQLiteFastFilters, and PostgreSQLFastFilters addons#1

Open
dsblank wants to merge 8 commits into
maintenance/gramps62from
add-fast-filters-addons
Open

Add FastFiltersLib, SQLiteFastFilters, and PostgreSQLFastFilters addons#1
dsblank wants to merge 8 commits into
maintenance/gramps62from
add-fast-filters-addons

Conversation

@dsblank

@dsblank dsblank commented Jun 6, 2026

Copy link
Copy Markdown
Owner

Summary

  • FastFiltersLib — shared `GENERAL` plugin (`load_on_reg=True`) providing SQL-accelerated `RuleOverride` subclasses and a `SQLCompat` dialect abstraction layer for SQLite and PostgreSQL
  • SQLiteFastFilters — `DATABASE` backend extending the built-in `SQLite` class; registers the overrides on open
  • PostgreSQLFastFilters — self-contained `DATABASE` backend for PostgreSQL (no dependency on the existing PostgreSQL addon); registers the same overrides on open

Initial filter rules accelerated: `IsMale`, `IsFemale`, `Disconnected`, `HasAlternateName`.

Design notes

  • `fastfilterslib.sql_compat.SQLCompat` abstracts placeholder style, `REGEXP`/`~`, JSON access (`json_extract` vs `->>`), and boolean literals so rule code is dialect-agnostic
  • `FastFiltersLib` uses `load_on_reg=True` so its package (`fastfilterslib.*`) is cached in `sys.modules` before any `DATABASE` plugin activates — cross-addon imports work without path manipulation
  • Both `SQLiteFastFilters` and `PostgreSQLFastFilters` declare `depends_on=["fastfilterslib"]`
  • Rules that require `json_data` (Disconnected, HasAlternateName) fall back gracefully to the original Python implementation when the database is in blob-data mode

Test plan

  • `FastFiltersLib/test/` — 45 tests: 8 registration/fallback tests (mock DB only) + 37 `SQLCompat` unit tests (including live SQLite syntax checks)
  • `SQLiteFastFilters/test/` — 19 integration tests against a small hand-built DB and the `example.gramps` fixture (known counts: 1168 male, 940 female, 71 disconnected, 2 with alternate names)
  • PostgreSQL integration tests pending a real server setup (shared base class in `FastFiltersLib/test/filter_overrides_base.py` is ready)

Run with:
```
cd FastFiltersLib && python -m pytest test/ -q
cd SQLiteFastFilters && python -m pytest test/ -q
```

🤖 Generated with Claude Code

dsblank and others added 8 commits June 6, 2026 09:22
@dsblank @claude
Add FastFiltersLib, SQLiteFastFilters, and PostgreSQLFastFilters addons
8c6ca9c 
FastFiltersLib is a shared GENERAL plugin (load_on_reg=True) that provides
SQL-accelerated filter rule overrides via RuleOverride.  SQLiteFastFilters
and PostgreSQLFastFilters are DATABASE backends that register those overrides
on open.  Initial rules: IsMale, IsFemale, Disconnected, HasAlternateName.
Dialect differences (placeholders, JSON access, regex) are abstracted in
fastfilterslib.sql_compat.SQLCompat.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dsblank @claude
FastFiltersLib: add 19 SQL-accelerated filter overrides
7deaaac 
Expands FastFiltersLib from 4 to 19 registered rule overrides:

Person overrides (secondary columns, no json_data):
- HasUnknownGender, HasOtherGender (gender = 2/3)

Person overrides (json_data, fallback to original for blob-data DBs):
- NeverMarried (family_list length = 0)
- MultipleMarriages (family_list length > 1)

Private/public overrides across all nine object types
(private is a secondary INTEGER column on every table):
- PeoplePrivate, PeoplePublic (person)
- FamilyPrivate, EventPrivate, PlacePrivate
- CitationPrivate, SourcePrivate, NotePrivate
- MediaPrivate, RepoPrivate

Place overrides:
- HasNoLatOrLon (TRIM(COALESCE(lat/long,''))='')

Uses _IsPrivateOverride base class with SQLCompat.true()/false()
for dialect-correct boolean comparisons (SQLite: 1/0, PostgreSQL: TRUE/FALSE).

Adds PrivateFilterTestsMixin covering all object types, a generalised
_commit() helper, and a shared _apply(db, rule, namespace) function.
SQLiteFastFilters grows from 35 to 49 passing tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dsblank @claude
FastFiltersLib: add HasNickname override, remove blob-data fallback code
50c31b8 
Add HasNicknameOverride (Tier 2) that queries primary_name.nick via
json_extract; nicks in alternate names or NICKNAME attributes are a
documented approximation (false negatives very rare in practice).

Remove all blob-data fallback branches (_needs_json_data, the guard
checks in prepare(), and the hasattr sentinel in apply_to_one()) since
blob databases are no longer supported.  Drop the TestJsonDataFallback
test class.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dsblank @claude
FastFiltersLib: add Tier 3 SQL overrides (MissingParent, HaveChildren…
aa12f49 
…, NoBirthdate, NoDeathdate)

Add four new SQL-accelerated filter overrides:

- MissingParentOverride: persons with no parent families OR a parent family
  missing a father/mother; uses json_each over parent_family_list joined to
  the family table's secondary father_handle/mother_handle columns.

- HaveChildrenOverride: persons in a family whose child_ref_list is non-empty;
  uses json_each over family_list joined to family.json_data child_ref_list
  length.

- NoBirthdateOverride / NoDeathdateOverride: persons whose birth/death event
  has date.sortval == 0 or no event at all; uses a dynamic array-index path
  via birth_ref_index/death_ref_index secondary columns and a LEFT JOIN to
  event.  A CASE WHEN guard prevents SQLite from evaluating the JSON path when
  the index is the -1 sentinel (SQLite rejects negative path indices).

Add three new SQLCompat helpers: json_each_text(), json_dynamic_array_field(),
and json_extract_int().  Add Tier3TestsMixin with setup and correctness tests,
plus count-match tests against example.gramps.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dsblank @claude
FastFiltersLib: add Tier 3+ overrides (HaveAltFamilies, IncompleteNames)
67ccc14 
HaveAltFamiliesOverride: pure SQL using nested EXISTS — outer json_each over
parent_family_list joined to family, inner json_each over child_ref_list
filtered to this person's handle, checking frel.value or mrel.value = 2
(ADOPTED).

IncompleteNamesOverride: SQL + Python hybrid — SQL covers all primary name
cases (blank first_name, empty surname_list, or blank surname entry via
json_each); Python post-checks alternate names in apply_to_one(), which is
the rare case and avoids a doubly-nested json_each over alternate_names →
surname_list.

Add json_each_json() to SQLCompat for iterating arrays of JSON objects (uses
jsonb_array_elements in PostgreSQL, json_each in SQLite).

Add Tier3PlusTestsMixin with correctness tests against a hand-built DB, plus
exact-match tests for both rules against example.gramps (2 adopted people;
76 people with incomplete names).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dsblank @claude
FastFiltersLib: use rule.selected_handles in all pure set-membership …
b84db4e 
…overrides

Sets self.rule.selected_handles instead of self.handles in prepare() for every
override whose apply_to_one() does nothing but check set membership.  The Gramps
filter optimizer detects this attribute on the rule and skips the per-object loop
entirely, reducing a filtered scan to a single SQL query + set intersection.

IncompleteNamesOverride is intentionally excluded because its apply_to_one()
performs additional Python checks on alternate names beyond set membership.

Update SQLPathTestsMixin to assert rule.selected_handles rather than
override.handles, matching the optimizer protocol.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dsblank @claude
FastFiltersLib: add Tier 4 and Tier 5 filter override roadmap
cbbd011 
Documents all remaining overrideable rules:
- Tier 4: ChangedSince and HasTag for all 9 object types
- Tier 4a: seven array-count rule families (HasGallery, HasNote, HasLDS,
  HasAssociation, HasAddress, HasSourceCount, HasRepository)
- Tier 4b: type-equality and secondary-column text match (HasRelType,
  NoteHasType, MatchesTitleSubstringOf)
- Tier 5: single JOIN or nested json_each (HasNameType, IsWitness,
  PersonWithIncompleteEvent, SearchFatherName/Mother/Child, InLatLonNeighborhood)
- Tier 5+: not-feasible table with reasons for each rule

Updates summary table and implementation order to cover all 27 new rules.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dsblank @claude
FastFiltersLib: convert IncompleteNames to pure SQL (cover alternate …
f22c547 
…names)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dsblank dsblank mentioned this pull request Jun 22, 2026
Merged
@eduralph

eduralph commented Jun 22, 2026

Copy link
Copy Markdown

@dsblank

  1. Discovery/runner convention (cosmetic). Add CI/CD pipeline with container-based testing (Feature Request ID 9393 in Mantis) gramps-project/addons-source#820's CI discovers /tests/test_*.py (plural dir, test_ prefix) and runs them with stdlib unittest. Yours use test/ (singular) + *_test.py (suffix) + pytest (conftest.py), so CI won't find or run them as-is.

  2. Inter-addon test dependencies (the real one). Your SQLite/PostgreSQL conftest.py put FastFiltersLib/ on sys.path so the dependents can import the shared lib and its test base classes — "simulates load_on_reg pre-loading." That's a genuine addon→addon dependency, and Add CI/CD pipeline with container-based testing (Feature Request ID 9393 in Mantis) gramps-project/addons-source#820's test model doesn't cover it: the runner puts only each addon's own directory on the path, not its sibling-addon dependencies. So even if the filenames matched, from fastfilterslib… import / from filter_overrides_base import would still fail in CI. The real question isn't naming — it's how CI should test an addon that depends on another addon. Any thoughts?

@dsblank

dsblank commented Jun 23, 2026

Copy link
Copy Markdown
Owner Author

how CI should test an addon that depends on another addon. Any thoughts?

A simple copy of the dependency to an installation path?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dsblank @eduralph