Add a self-contained v8 search schema revision (PP-4659) (#3522)

## Description

Adds a self-contained v8 OpenSearch schema revision (`SearchV8`) and
registers it in the revision directory.

Unlike earlier revisions, which chained off one another (`SearchV7` ->
`SearchV6` -> `SearchV5`), v8 subclasses `SearchSchemaRevision` directly
and defines its complete mapping (analyzers, filters, and fields)
inline. Breaking the chain means each revision stands alone, so an old
revision's module can simply be deleted once nothing in production is
using it.

Schema-wise v8 is equivalent to v7 (the full v5 mapping plus v6's
`lane_priority_level` and v7's `licensepools.last_updated`). The mapping
is unchanged; the differences are all in the index settings, which v8
now pins explicitly so a newly created index is fully deterministic
rather than relying on inherited cluster defaults:

- **`number_of_shards = 1`** — earlier indexes inherited a
5-primary-shard count from the original Elasticsearch 6.x defaults,
carried forward through every reindex because nothing set it explicitly.
Per-library indexes are well under a gigabyte, far below the per-shard
target for search workloads, so a single primary shard is correct. This
setting is immutable after index creation, so it must be set up front.
- **`number_of_replicas = 1`** — matches what every production index
already runs and is also the OpenSearch default, so this is
operationally a no-op. It is a dynamic setting and can still be retuned
at runtime for a larger topology; it is pinned only so indexes are
created with a known replica count.
- **`index.search.slowlog.threshold.*`** — seeds slow-query-log
thresholds on every index this revision creates, so slow query- and
fetch-phases are written to the cluster slow log. Paired with the
`SEARCH_SLOW_LOGS` publishing wired up in the hosting playbook, these
entries reach the domain's CloudWatch log group. These are dynamic
settings, so they establish a baseline at index creation that can still
be retuned on a live index without a reindex.

## Motivation and Context

The revision chain meant no old schema version could be removed without
breaking the revisions built on top of it, so deprecated versions
accumulated indefinitely. Making each revision self-contained lets old
versions be deleted once production no longer uses them.

Pinning the index settings replaces inherited, undocumented cluster
defaults with an explicit, deterministic configuration — most
importantly the shard count, which is immutable after index creation and
had been silently carried over from the Elasticsearch era.

JIRA: PP-4659

## How Has This Been Tested?

Added `tests/manager/search/test_search_v8.py`, covering the version
number, registration in the revision directory, the pinned shard/replica
counts, and the seeded slow-query-log thresholds in the generated
mapping document. Ran the search test suite under the docker tox
environment.

## Checklist

- [x] I have updated the documentation accordingly.
- [x] All new and existing tests passed.

Author: jonathangreen

src/palace/manager/search/revision_directory.py

@@ -5,8 +5,9 @@
 from palace.manager.search.v5 import SearchV5
 from palace.manager.search.v6 import SearchV6
 from palace.manager.search.v7 import SearchV7
+from palace.manager.search.v8 import SearchV8
 
-REVISIONS = [SearchV5(), SearchV6(), SearchV7()]
+REVISIONS = [SearchV5(), SearchV6(), SearchV7(), SearchV8()]
 
 
 class SearchRevisionDirectory:

src/palace/manager/search/v8.py

@@ -0,0 +1,296 @@
+from palace.manager.search.document import (
+    BASIC_TEXT,
+    BOOLEAN,
+    FILTERABLE_TEXT,
+    FLOAT,
+    INTEGER,
+    LONG,
+    SearchMappingDocument,
+    SearchMappingFieldType,
+    icu_collation_keyword,
+    keyword,
+    nested,
+    sort_author_keyword,
+)
+from palace.manager.search.revision import SearchSchemaRevision
+
+
+class SearchV8(SearchSchemaRevision):
+    """The version 8 search schema revision.
+
+    This revision is intentionally **self-contained**: it does not inherit from
+    any previous revision and defines its complete mapping (analyzers, filters,
+    and fields) directly. This is how new schema revisions are meant to be defined.
+
+    We formed a bad habit for several old schema revisions where they just chained
+    off one another (``SearchV7`` -> ``SearchV6`` -> ``SearchV5``) which meant
+    no old revision could be removed without breaking the ones build on top of
+    it.
+
+    From now on every revision should stand alone so that, once nothing in
+    production is using an older version, that version's module can simply be deleted.
+
+    Changes in v8:
+      - The shard and replica counts are set explicitly.
+      - sets search slow-query-log thresholds
+    """
+
+    @property
+    def version(self) -> int:
+        return 8
+
+    # The number of primary shards for indexes created by this revision.
+    # This setting is immutable once an index is created.
+    NUMBER_OF_SHARDS = 1
+
+    # The number of replicas for indexes created by this revision.
+    NUMBER_OF_REPLICAS = 1
+
+    # Slow-query-log thresholds applied to every index this revision creates.
+    #  Keys are relative to "index".
+    SEARCH_SLOWLOG_THRESHOLDS = {
+        "search.slowlog.threshold.query.warn": "3s",
+        "search.slowlog.threshold.query.info": "1s",
+        "search.slowlog.threshold.fetch.warn": "3s",
+        "search.slowlog.threshold.fetch.info": "1s",
+    }
+
+    # Use regular expressions to normalized values in sortable fields.
+    # These regexes are applied in order; that way "H. G. Wells"
+    # becomes "H G Wells" becomes "HG Wells".
+    CHAR_FILTERS = {
+        "remove_apostrophes": dict(
+            type="pattern_replace",
+            pattern="'",
+            replacement="",
+        )
+    }
+
+    AUTHOR_CHAR_FILTER_NAMES = []
+    for name, pattern, replacement in [
+        # The special author name "[Unknown]" should sort after everything
+        # else. REPLACEMENT CHARACTER is the final valid Unicode character.
+        ("unknown_author", r"\[Unknown\]", "\N{REPLACEMENT CHARACTER}"),
+        # Works by a given primary author should be secondarily sorted
+        # by title, not by the other contributors.
+        ("primary_author_only", r"\s+;.*", ""),
+        # Remove parentheticals (e.g. the full name of someone who
+        # goes by initials).
+        ("strip_parentheticals", r"\s+\([^)]+\)", ""),
+        # Remove periods from consideration.
+        ("strip_periods", r"\.", ""),
+        # Collapse spaces for people whose sort names end with initials.
+        ("collapse_three_initials", r" ([A-Z]) ([A-Z]) ([A-Z])$", " $1$2$3"),
+        ("collapse_two_initials", r" ([A-Z]) ([A-Z])$", " $1$2"),
+    ]:
+        normalizer = dict(
+            type="pattern_replace", pattern=pattern, replacement=replacement
+        )
+        CHAR_FILTERS[name] = normalizer
+        AUTHOR_CHAR_FILTER_NAMES.append(name)
+
+    def __init__(self) -> None:
+        super().__init__()
+
+        self._normalizers = {}
+        self._filters = {}
+        self._analyzers = {}
+
+        # Set up character filters.
+        #
+        self._char_filters = self.CHAR_FILTERS
+
+        # This normalizer is used on freeform strings that
+        # will be used as tokens in filters. This way we can,
+        # e.g. ignore capitalization when considering whether
+        # two books belong to the same series or whether two
+        # author names are the same.
+        self._normalizers["filterable_string"] = dict(
+            type="custom", filter=["lowercase", "asciifolding"]
+        )
+
+        # Set up analyzers.
+        #
+
+        # We use three analyzers:
+        #
+        # 1. An analyzer based on Opensearch's default English
+        #    analyzer, with a normal stemmer -- used as the default
+        #    view of a text field such as 'description'.
+        #
+        # 2. An analyzer that's exactly the same as #1 but with a less
+        #    aggressive stemmer -- used as the 'minimal' view of a
+        #    text field such as 'description.minimal'.
+        #
+        # 3. An analyzer that's exactly the same as #2 but with
+        #    English stopwords left in place instead of filtered out --
+        #    used as the 'with_stopwords' view of a text field such as
+        #    'title.with_stopwords'.
+        #
+        # The analyzers are identical except for the end of the filter
+        # chain.
+        #
+        # All three analyzers are based on Opensearch's default English
+        # analyzer, defined here:
+        # https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html#english-analyzer
+
+        # First, recreate the filters from the default English
+        # analyzer. We'll be using these to build our own analyzers.
+
+        # Filter out English stopwords.
+        self._filters["english_stop"] = dict(type="stop", stopwords=["_english_"])
+        # The default English stemmer, used in the en_default analyzer.
+        self._filters["english_stemmer"] = dict(type="stemmer", language="english")
+        # A less aggressive English stemmer, used in the en_minimal analyzer.
+        self._filters["minimal_english_stemmer"] = dict(
+            type="stemmer", language="minimal_english"
+        )
+        # A filter that removes English posessives such as "'s"
+        self._filters["english_posessive_stemmer"] = dict(
+            type="stemmer", language="possessive_english"
+        )
+
+        # Some potentially useful filters that are currently not used:
+        #
+        # * keyword_marker -- Exempt certain keywords from stemming
+        # * synonym -- Introduce synonyms for words
+        #   (but probably better to use synonym_graph during the search
+        #    -- it's more flexible).
+
+        # Here's the common analyzer configuration. The comment NEW
+        # means this is something we added on top of Opensearch's
+        # default configuration for the English analyzer.
+        common_text_analyzer = dict(
+            type="custom",
+            char_filter=["html_strip", "remove_apostrophes"],  # NEW
+            tokenizer="standard",
+        )
+        common_filter = [
+            "lowercase",
+            "asciifolding",  # NEW
+        ]
+
+        # The default_text_analyzer uses Opensearch's standard
+        # English stemmer and removes stopwords.
+        self._analyzers["en_default_text_analyzer"] = dict(common_text_analyzer)
+        self._analyzers["en_default_text_analyzer"]["filter"] = common_filter + [
+            "english_stop",
+            "english_stemmer",
+        ]
+
+        # The minimal_text_analyzer uses a less aggressive English
+        # stemmer, and removes stopwords.
+        self._analyzers["en_minimal_text_analyzer"] = dict(common_text_analyzer)
+        self._analyzers["en_minimal_text_analyzer"]["filter"] = common_filter + [
+            "english_stop",
+            "minimal_english_stemmer",
+        ]
+
+        # The en_with_stopwords_text_analyzer uses the less aggressive
+        # stemmer and does not remove stopwords.
+        self._analyzers["en_with_stopwords_text_analyzer"] = dict(common_text_analyzer)
+        self._analyzers["en_with_stopwords_text_analyzer"]["filter"] = common_filter + [
+            "minimal_english_stemmer"
+        ]
+
+        # Now we need to define a special analyzer used only by the
+        # 'sort_author' property.
+
+        # Here's a special filter used only by that analyzer. It
+        # duplicates the filter used by the icu_collation_keyword data
+        # type.
+        self._filters["en_sortable_filter"] = dict(
+            type="icu_collation", language="en", country="US"
+        )
+
+        # Here's the analyzer used by the 'sort_author' property.
+        # It's the same as icu_collation_keyword, but it has some
+        # extra character filters -- regexes that do things like
+        # convert "Tolkien, J. R. R." to "Tolkien, JRR".
+        #
+        # This is necessary because normal icu_collation_keyword
+        # fields can't specify char_filter.
+        self._analyzers["en_sort_author_analyzer"] = dict(
+            tokenizer="keyword",
+            filter=["en_sortable_filter"],
+            char_filter=self.AUTHOR_CHAR_FILTER_NAMES,
+        )
+
+        self._fields: dict[str, SearchMappingFieldType] = {
+            "summary": BASIC_TEXT,
+            "title": FILTERABLE_TEXT,
+            "subtitle": FILTERABLE_TEXT,
+            "series": FILTERABLE_TEXT,
+            "classifications.term": FILTERABLE_TEXT,
+            "author": FILTERABLE_TEXT,
+            "publisher": FILTERABLE_TEXT,
+            "imprint": FILTERABLE_TEXT,
+            "presentation_ready": BOOLEAN,
+            "sort_title": icu_collation_keyword(),
+            "sort_author": sort_author_keyword(),
+            "series_position": INTEGER,
+            "work_id": INTEGER,
+            "last_update_time": LONG,
+            "published": LONG,
+            "audience": keyword(),
+            "language": keyword(),
+            # Added in v6.
+            "lane_priority_level": INTEGER,
+        }
+
+        contributors = nested()
+        contributors.add_property("display_name", FILTERABLE_TEXT)
+        contributors.add_property("sort_name", FILTERABLE_TEXT)
+        contributors.add_property("family_name", FILTERABLE_TEXT)
+        contributors.add_property("role", keyword())
+        contributors.add_property("lc", keyword())
+        contributors.add_property("viaf", keyword())
+        self._fields["contributors"] = contributors
+
+        licensepools = nested()
+        licensepools.add_property("collection_id", INTEGER)
+        licensepools.add_property("data_source_id", INTEGER)
+        licensepools.add_property("availability_time", LONG)
+        licensepools.add_property("available", BOOLEAN)
+        licensepools.add_property("open_access", BOOLEAN)
+        licensepools.add_property("suppressed", BOOLEAN)
+        licensepools.add_property("licensed", BOOLEAN)
+        licensepools.add_property("medium", keyword())
+        # Added in v7.
+        licensepools.add_property("last_updated", LONG)
+        self._fields["licensepools"] = licensepools
+
+        identifiers = nested()
+        identifiers.add_property("type", keyword())
+        identifiers.add_property("identifier", keyword())
+        self._fields["identifiers"] = identifiers
+
+        genres = nested()
+        genres.add_property("scheme", keyword())
+        genres.add_property("name", keyword())
+        genres.add_property("term", keyword())
+        genres.add_property("weight", FLOAT)
+        self._fields["genres"] = genres
+
+        customlists = nested()
+        customlists.add_property("list_id", INTEGER)
+        customlists.add_property("first_appearance", LONG)
+        customlists.add_property("featured", BOOLEAN)
+        self._fields["customlists"] = customlists
+
+    def mapping_document(self) -> SearchMappingDocument:
+        document = SearchMappingDocument()
+        document.settings["analysis"] = dict(
+            filter=dict(self._filters),
+            char_filter=dict(self._char_filters),
+            normalizer=dict(self._normalizers),
+            analyzer=dict(self._analyzers),
+        )
+        # Index settings
+        document.settings["index"] = {
+            "number_of_shards": self.NUMBER_OF_SHARDS,
+            "number_of_replicas": self.NUMBER_OF_REPLICAS,
+            **self.SEARCH_SLOWLOG_THRESHOLDS,
+        }
+        document.properties = self._fields
+        return document

tests/manager/search/test_search_v8.py

@@ -0,0 +1,39 @@
+from palace.manager.search.revision import SearchSchemaRevision
+from palace.manager.search.v7 import SearchV7
+from palace.manager.search.v8 import SearchV8
+
+
+class TestSearchV8:
+    def test_version(self):
+        assert SearchV8().version == 8
+
+    def test_self_contained(self):
+        """v8 must not chain off any previous revision, so old revisions can be
+        deleted once nothing in production uses them."""
+        assert SearchV8.__bases__ == (SearchSchemaRevision,)
+
+    def test_pins_index_settings(self):
+        """v8 sets the expected index settings."""
+        index = SearchV8().mapping_document().settings["index"]
+        for setting in [
+            "number_of_shards",
+            "number_of_replicas",
+            "search.slowlog.threshold.query.warn",
+            "search.slowlog.threshold.query.info",
+            "search.slowlog.threshold.fetch.warn",
+            "search.slowlog.threshold.fetch.info",
+        ]:
+            assert setting in index
+
+    def test_mapping_matches_v7(self):
+        """v8 is a faithful, self-contained copy of the v7 schema: same fields
+        and same analysis settings. Only the index settings differ (v8 pins the
+        shard count)."""
+        v8_document = SearchV8().mapping_document()
+        v7_document = SearchV7().mapping_document()
+
+        assert v8_document.serialize_properties() == v7_document.serialize_properties()
+        assert v8_document.settings["analysis"] == v7_document.settings["analysis"]
+        # v7 left the shard count to an inherited default; v8 sets it explicitly.
+        assert "index" not in v7_document.settings
+        assert "index" in v8_document.settings