Baseline test improvements: pretty-print snapshots, unified-diff failures, docstrings, local data (#140)

Sets a cleaner baseline for the test suite so future data and code PRs
are easier to develop, review, and triage. All changes confined to
`tests/` (plus a one-character typo fix in `database/etfs.csv`).
**All 32 tests pass.**

## Snapshot diffs that actually tell you what changed

The `test_show_options*.json` snapshots are JSON arrays of category labels (countries, sectors, currencies, …) — single-line strings of 1-2 KB. When a database update added/removed one entry, the GitHub diff was one giant string change with no way to see what differed.

```diff
- string_value = json.dumps(data, **kwargs)
+ string_value = json.dumps(data, indent=2, ensure_ascii=False, **kwargs)
```

`ensure_ascii=False` also means non-ASCII characters render natively (`Côte d'Ivoire` instead of escape sequences).

## Unified-diff on snapshot mismatch

`Recorder.assert_equal()` previously raised a `AssertionError: Change detected` with two truncated 500-char strings, often appearing identical in the visible part even when they actually differed later. We hit this exact issue while developing #141 and #143.

Now `assert_equal()` produces a proper `difflib.unified_diff` showing which line(s) changed; `assert_in_list` gained a proper assertion message.

## Tests now read the PR branch's data, not main

The library defaults to fetching `compression/*` from the GitHub `main` branch over HTTP. This meant tests on a PR validated the data on `main`, not the data in the PR — silently passing data-breaking PRs and failing on data-fixing PRs.

Two cooperating fixes:
- Every `tests/test_*.py` now instantiates with `use_local_location=True` so the library reads the local checkout.
- `tests/conftest.py` regenerates `compression/*.bz2` and `compression/categories/*.gzip` from the checked-out `database/*.csv` at import time, mirroring the production `database_update.yml` workflow. This must run *before* pytest collects the test modules (test-module imports instantiate `fd.X(use_local_location=True)`), so it is a top-level statement, not a fixture.

The compression artifacts themselves are not committed in this PR — they are deterministically derived from the CSVs and the test suite regenerates them on every run.

## Test module quality (incidental cleanup)

Since the regen touched every test file, took the opportunity to:
- Fix wrong module docstrings (`test_equities.py` and `test_moneymarkets.py` both started with `"""Currencies Test Module"""`).
- Replace `# pylint: disable=missing-function-docstring` with real docstrings on every test function in the 7 asset-class files and in `test_sec_enrichment_controller.py`.
- Add type hints `(recorder: Recorder) -> None`; `Recorder` import under `TYPE_CHECKING`, guarded by `from __future__ import annotations` — zero runtime cost.

## conftest.py cleanup

- Removed `# pylint: skip-file` so static analysis is no longer suppressed.
- Cast `request.config.getoption("--rewrite-expected")` return through `bool(...)` to satisfy the declared `-> bool` annotation (previously `Any | None`).

## Test infrastructure config (pyproject.toml)

- Dropped `pytest-recording` from dev deps — unused; the real dep used by `conftest.py` is `pytest-recorder` (which provides the `record_mode` / `disable_recording` fixtures).
- Declared `[tool.pytest.ini_options]` `testpaths = ["tests"]`, `addopts = "--strict-markers"`, and the `record_stdout` marker so future PRs can't silently introduce undeclared markers.

## Side fix: ASYMshsare → ASYMshares (etfs.csv + tests)

While running the new local-data tests we noticed `ASPY` (Leverage Shares ASYMmetric 500 ETF) had `family = "ASYMshsare"` — a typo in `database/etfs.csv`. Corrected to `ASYMshares` along with the references in `tests/test_etfs.py` and the affected fixture snapshots. The maintainer's `Update Compression Files` workflow will regenerate the binary artifacts on `main` post-merge.

## What this PR explicitly does NOT do

Deferred to focused follow-ups (out of scope here):
- `@pytest.mark.parametrize` / class-based grouping — would change pytest node IDs and break `Recorder.capture()` positional snapshot naming (renames 90+ files in one go).
- `pytest-xdist` parallel execution — adds a new dev dep; pure perf win.
- Wiring `pytest-cov` (already in dev deps) into CI for coverage reporting.
- Filling the 22% coverage gap on `financedatabase/helpers.py` (`FinanceFrame.to_toolkit`, `show_options` URL error path, `case_sensitive=True`).

Author: dokson

database/etfs.csv

@@ -2275,7 +2275,7 @@ ASIL-GB.AQ,MULTI UNITS LUXEMB,GBP,"MULTI UNITS LUXEMB is an investment fund inco
 ASIL.L,Multi Units Luxembourg - Lyxor China Enterprise (HSCEI) UCITS ETF,GBP,"Multi Units Luxembourg - Lyxor China Enterprise (HSCEI) UCITS ETF is an exchange-traded fund designed to track the performance of the Hang Seng China Enterprises Index (HSCEI). The ETF provides exposure to the largest companies in China that are listed on the Hong Kong Stock Exchange. Lyxor Asset Management, a subsidiary of Societe Generale, manages the ETF, which is domiciled in Luxembourg.",Equities,,Lyxor International Asset Management,LSE,
 ASIU.L,Lyxor China Enterprise (HSCEI) UCITS ETF - Acc-EUR,USD,"The investment objective of the MULTI UNITS LUXEMBOURG — Lyxor China Enterprise (HSCEI) UCITS ETF (the ""Sub-Fund"") is to track both the upward and the downward evolution of the Hang Seng China Enterprises Index Net Total Return index (the ""Index"") denominated in Hong Kong dollars (HKD), and representative of the major Chinese equities known as ""H-shares, while minimizing the volatility of the difference between the return of the Sub-Fund and the return of the Index (the ""Tracking Error"").",Equities,,Lyxor International Asset Management,LSE,LU1900068914
 ASPY,ASYMshares ASYMmetric 500 ETF,USD,"The investment seeks to track the total return performance, before fees and expenses, of the ASYMmetric 500 Index.
- The index is a rules-based, quantitative long/short hedging strategy that seeks to provide protection against bear market losses and to capture the majority of bull market gains with respect to exposure to the 500 largest capitalized equity securities publicly traded in the United States, which is referred to as the index's ""market"". Under normal market conditions, the fund will invest at least 80% of its total assets in securities and cash included in the index's Long Book.",Equities,Equities,ASYMshsare,PCX,
+ The index is a rules-based, quantitative long/short hedging strategy that seeks to provide protection against bear market losses and to capture the majority of bull market gains with respect to exposure to the 500 largest capitalized equity securities publicly traded in the United States, which is referred to as the index's ""market"". Under normal market conditions, the fund will invest at least 80% of its total assets in securities and cash included in the index's Long Book.",Equities,Equities,ASYMshares,PCX,
 ASR0.BE,BNPP ENERGY TRAN. CLCAP,EUR,"BNPP ENERGY TRAN. CLCAP is a fund managed by BNP Paribas Asset Management focused on investing in companies involved in the energy transition. The ""CLCAP"" indicates that it is an accumulation share class, meaning dividends are reinvested. Further information is required to understand the fund's specific investment strategy and the sectors within the energy transition it targets.",Energy,,BNP Paribas Asset Management,BER,
 ASR0.MU,Parvest SICAV - Energy Innovators,EUR,"Parvest SICAV - Energy Innovators is an investment fund focused on companies involved in the energy transition, including renewable energy, energy efficiency, and related technologies. The fund aims to provide long-term capital appreciation by investing in companies that are contributing to a more sustainable energy future. BNP Paribas Asset Management manages the fund, which is part of the Parvest SICAV fund range domiciled in Luxembourg.",Energy,,BNP Paribas Asset Management,MUN,
 ASR1.BE,BNPP GER.MF EQ. CL.CAP,EUR,"BNPP GER.MF EQ. CL.CAP is a fund managed by BNP Paribas Asset Management that invests in German equities (EQ). ""MF"" suggests it might be a multi-factor fund. ""CL.CAP"" indicates an capitalization-weighted accumulation share class, where dividends are reinvested.",Equities,,BNP Paribas Asset Management,BER,

poetry.lock

@@ -91,4 +91,9 @@ disable_error_code = "misc, valid-type, attr-defined, index"
 [tool.pytest.ini_options]
 filterwarnings = [
     "ignore::pytest.PytestAssertRewriteWarning:",
+]
+testpaths = ["tests"]
+addopts = "--strict-markers"
+markers = [
+    "record_stdout: capture and snapshot stdout output for the marked test",
 ]

pyproject.toml

@@ -1,3 +1,4 @@
+import difflib
 import json
 import os
 import pathlib
@@ -28,7 +29,88 @@
     "txt": [str],
 }
 
-# pylint: skip-file
+REPO_ROOT = pathlib.Path(__file__).resolve().parents[1]
+ASSET_CSVS = {
+    "cryptos": "summary",
+    "currencies": "summary",
+    "equities": "summary",
+    "etfs": "summary",
+    "funds": "summary",
+    "indices": "summary",
+    "moneymarkets": "summary",
+}
+
+
+def _regenerate_compression_artifacts() -> None:
+    """Mirror the Database-Update workflow so tests run against compression
+    artifacts derived from the *checked-out* CSV files.
+
+    The financedatabase library reads `compression/<asset>.bz2` and
+    `compression/categories/<asset>_categories.gzip`; without this step those
+    artifacts lag the `database/*.csv` on a PR branch and tests silently
+    validate against `main` instead of the PR change.
+    """
+    compression_dir = REPO_ROOT / "compression"
+    categories_dir = compression_dir / "categories"
+    for asset, name_col in ASSET_CSVS.items():
+        csv_path = REPO_ROOT / "database" / f"{asset}.csv"
+        if not csv_path.exists():
+            continue
+        df = pd.read_csv(csv_path)
+        df.to_csv(compression_dir / f"{asset}.bz2", index=False, compression="bz2")
+        indexed = pd.read_csv(csv_path, index_col=0)
+        categories: dict[str, Any] = {}
+        skip_cols = {"name", name_col, "manager_name", "manager_bio"}
+        for column in indexed.columns:
+            if column in skip_cols:
+                continue
+            vals = indexed[column].dropna().unique()
+            vals.sort()
+            categories[column] = vals
+        cat_df = pd.DataFrame.from_dict(categories, orient="index").reset_index()
+        cat_df.to_csv(
+            categories_dir / f"{asset}_categories.gzip",
+            index=False,
+            compression="gzip",
+        )
+
+
+def _snapshot_compression_artifacts() -> dict[pathlib.Path, bytes]:
+    """Capture current bytes of compression artifacts so they can be restored."""
+    compression_dir = REPO_ROOT / "compression"
+    snapshot: dict[pathlib.Path, bytes] = {}
+    for asset in ASSET_CSVS:
+        for path in (
+            compression_dir / f"{asset}.bz2",
+            compression_dir / "categories" / f"{asset}_categories.gzip",
+        ):
+            if path.exists():
+                snapshot[path] = path.read_bytes()
+    return snapshot
+
+
+def _restore_compression_artifacts(snapshot: dict[pathlib.Path, bytes]) -> None:
+    """Restore compression artifacts to their pre-test contents."""
+    for path, data in snapshot.items():
+        try:
+            path.write_bytes(data)
+        except Exception:
+            pass
+
+
+# Regenerate compression artifacts BEFORE pytest collects test modules. The
+# asset-class test modules instantiate `fd.X(use_local_location=True)` at
+# import time, so the artifacts must already be in sync with the checked-out
+# CSVs by then — a session-scoped fixture runs too late. A snapshot of the
+# original bytes is taken first; `pytest_sessionfinish` restores them so the
+# working tree stays clean for contributors.
+_compression_snapshot = _snapshot_compression_artifacts()
+_regenerate_compression_artifacts()
+
+
+def pytest_sessionfinish(session, exitstatus) -> None:  # noqa: ARG001
+    """Restore compression artifacts after the pytest session ends."""
+    _restore_compression_artifacts(_compression_snapshot)
 
 
 class Record:
@@ -43,7 +125,7 @@ def extract_string(data: Any, **kwargs) -> str:
                 **kwargs,
             )
         elif isinstance(data, tuple(EXTENSIONS_MATCHING["json"])):
-            string_value = json.dumps(data, **kwargs)
+            string_value = json.dumps(data, indent=2, ensure_ascii=False, **kwargs)
         else:
             raise AttributeError(f"Unsupported type : {type(data)}")
 
@@ -225,19 +307,39 @@ def assert_equal(self):
 
         for record in record_list:
             if record.record_changed:
+                expected = (record.recorded or "").splitlines()
+                actual = (record.captured or "").splitlines()
+                diff = "\n".join(
+                    difflib.unified_diff(
+                        expected,
+                        actual,
+                        fromfile="expected",
+                        tofile="actual",
+                        lineterm="",
+                        n=3,
+                    )
+                )
+                if not diff:
+                    diff = (
+                        f"(line-level diff empty — content differs only in trailing whitespace or EOL)\n"
+                        f"expected length={len(record.recorded or '')}, "
+                        f"actual length={len(record.captured or '')}"
+                    )
                 raise AssertionError(
-                    "Change detected\n"
-                    f"Record    : {record.record_path}\n"
-                    f"Expected  : {record.recorded[:self.display_limit]}\n"
-                    f"Actual    : {record.captured[:self.display_limit]}\n"
+                    f"Snapshot mismatch: {record.record_path}\n"
+                    f"Run pytest with --rewrite-expected to regenerate, or inspect the diff below.\n\n"
+                    f"{diff}"
                 )
 
     def assert_in_list(self, in_list: list[str]):
         record_list = self.__record_list
 
         for record in record_list:
             for string_value in in_list:
-                assert string_value in record.captured  # noqa
+                assert string_value in record.captured, (  # noqa: S101
+                    f"Expected substring {string_value!r} not found in "
+                    f"{record.record_path}"
+                )
 
     def persist(self):
         record_list = self.__record_list
@@ -345,7 +447,7 @@ def pytest_addoption(parser: Parser):
 @pytest.fixture(scope="session")  # type: ignore
 def rewrite_expected(request: SubRequest) -> bool:
     """Force rewriting of all expected data by : `record_stdout` and `recorder`."""
-    return request.config.getoption("--rewrite-expected")
+    return bool(request.config.getoption("--rewrite-expected"))
 
 
 @pytest.fixture

tests/conftest.py

@@ -1,3 +1,3 @@
 symbol,name,currency,summary,category_group,category,family,exchange,isin
 ASPY,ASYMshares ASYMmetric 500 ETF,USD,"The investment seeks to track the total return performance, before fees and expenses, of the ASYMmetric 500 Index.
- The index is a rules-based, quantitative long/short hedging strategy that seeks to provide protection against bear market losses and to capture the majority of bull market gains with respect to exposure to the 500 largest capitalized equity securities publicly traded in the United States, which is referred to as the index's ""market"". Under normal market conditions, the fund will invest at least 80% of its total assets in securities and cash included in the index's Long Book.",Equities,Equities,ASYMshsare,PCX,
+ The index is a rules-based, quantitative long/short hedging strategy that seeks to provide protection against bear market losses and to capture the majority of bull market gains with respect to exposure to the 500 largest capitalized equity securities publicly traded in the United States, which is referred to as the index's ""market"". Under normal market conditions, the fund will invest at least 80% of its total assets in securities and cash included in the index's Long Book.",Equities,Equities,ASYMshares,PCX,

tests/csv/test_etfs/test_search_4.csv

@@ -1,3 +1,3 @@
 symbol,name,currency,summary,category_group,category,family,exchange,isin
 ASPY,ASYMshares ASYMmetric 500 ETF,USD,"The investment seeks to track the total return performance, before fees and expenses, of the ASYMmetric 500 Index.
- The index is a rules-based, quantitative long/short hedging strategy that seeks to provide protection against bear market losses and to capture the majority of bull market gains with respect to exposure to the 500 largest capitalized equity securities publicly traded in the United States, which is referred to as the index's ""market"". Under normal market conditions, the fund will invest at least 80% of its total assets in securities and cash included in the index's Long Book.",Equities,Equities,ASYMshsare,PCX,
+ The index is a rules-based, quantitative long/short hedging strategy that seeks to provide protection against bear market losses and to capture the majority of bull market gains with respect to exposure to the 500 largest capitalized equity securities publicly traded in the United States, which is referred to as the index's ""market"". Under normal market conditions, the fund will invest at least 80% of its total assets in securities and cash included in the index's Long Book.",Equities,Equities,ASYMshares,PCX,

tests/csv/test_etfs/test_select_3.csv

@@ -1 +1,4 @@
-["cryptocurrency", "currency"]
+[
+  "cryptocurrency",
+  "currency"
+]