Add country column to etfs.csv and funds.csv + API + invariant tests + automated README stats

Closes the long-standing data-shape gap where ETFs and Funds had no
country field, plus a series of related data-quality fixes surfaced by
the new cross-asset invariant test.

## Schema change: new `country` column on etfs.csv and funds.csv

Derived deterministically by chaining `exchange -> country` from
equities.csv (the source of truth for that mapping), with three small
manual overrides for fund-only exchange codes that don't appear in
equities.csv (`NAS`/`NYM`/`CME` -> United States) plus `NIM` for ETF
NextShares (also US). A second-pass ticker-suffix fallback handles
rows whose `exchange` was missing or corrupted.

Coverage after the fill:

| File | Rows | country populated |
|---|---:|---|
| `etfs.csv` | 36,485 | 100.00% |
| `funds.csv` | 57,853 | 100.00% |

Top 5 countries:

- **ETFs**: United States (17,723), Canada (6,688), Germany (5,598),
  United Kingdom (2,137), Switzerland (1,463) — 33 distinct countries
- **Funds**: United States (47,643), Spain (5,383), Canada (1,898),
  United Kingdom (1,769), India (496) — 24 distinct countries

## Data-quality fixes surfaced by the new invariant tests

- **Removed 14 non-ETF rows** from `etfs.csv` (^REIT plus 13 US-equity
  duplicates that existed correctly in `equities.csv` already:
  `BHF`, `BHFAN`, `BHFAO`, `BHFAP`, `DTB`, `DTE`, `DTP`, `HTGC`, `PBC`,
  `PSEC`, `RGA`, `RZA`, `TPVG`).
- **Removed 56 cross-asset symbol collisions** between `equities.csv`
  and `etfs.csv` — all 56 were corporate bonds / senior notes / equity
  share-class rows that had ended up in `etfs.csv` (Brighthouse,
  Corvus Gold, Great Ajax Corp. notes, Argo notes, CMS Energy junior
  subordinated notes, Conifer Holdings senior notes, Qwest Corp notes,
  DTE Energy variants, ASGI = Aberdeen Global Infrastructure Income
  Fund, etc.).
- **Fixed 29 ETF rows with corrupted `exchange` values** (issuer name
  written into the exchange column instead of a real exchange code:
  `Xtrackers`, `Fundlogic`, `Purpose Investments`, `CI Investments`,
  `Horizons ETFs Management`, `Harvest Portfolios Group`,
  `IA Clarington Investments`, `National Bank Investments`,
  `Caldwell Investment Management`, `Developed Markets`,
  `Emerging Markets`, `High Yield Bonds`). Re-derived from the ticker
  suffix.
- **Completed FSST** (Fidelity Sustainable U.S. Equity ETF) — the row
  previously had every field NaN. Filled name, currency, summary,
  category_group, category, family, exchange (PCX), country, isin.

After this round: 0 cross-asset symbol collisions, 0 corrupted
exchange values flagged by the new invariants.

## API additions

`ETFs.select()`, `ETFs.show_options()`, `Funds.select()`, and
`Funds.show_options()` now accept a `country` parameter, validated
against the new column. Calling with an unknown country raises
`ValueError` matching the pattern used by the existing filters.

## New tests

In `tests/test_etfs.py` and `tests/test_funds.py`:

- `test_exchange_country_one_to_one` — asserts that every `exchange`
  code on the asset maps to exactly one `country` value (the same
  invariant introduced for equities `exchange -> market` in JerBouma#143).
- `test_select_with_invalid_value_raises` now also exercises the
  `country` filter ValueError path.

New file `tests/test_invariants.py`:

- `test_no_symbol_collisions_across_asset_classes` — asserts that a
  given `symbol` belongs to at most one of `equities.csv`,
  `etfs.csv`, `funds.csv`. Catches the kind of drift fixed by the
  56-row cleanup above before it can land on `main` again.

## Automated README statistics

`.github/workflows/database_update.yml` gains a new
`Update-README-Statistics` job that runs after the existing
Add-New-Ticker / Update-Compression / Update-Categorization jobs. It
recomputes both statistics tables from the on-disk CSVs and rewrites
README.md in-place, committing the result. The Check-GICS job now
also depends on it.

Replaced the meaningless `Countries` numbers for ETFs/Funds (which
were not backed by any column) with the now-real value derived from
the new `country` column. Other numbers also refreshed to current
state:

| Product           | Quantity   | Sectors    | Industries    | Countries | Exchanges |
|-------------------|-----------:|-----------:|--------------:|----------:|----------:|
| Equities          | 160.113    | 11         | 62            | 113       | 84        |
| ETFs              | 36.485     | 320        | 51            | 33        | 51        |
| Funds             | 57.853     | 1.540      | 74            | 24        | 33        |

| Product           | Quantity  | Category              |
|-------------------|----------:|-----------------------|
| Currencies        | 2.556     | 175 Currencies        |
| Cryptocurrencies  | 3.367     | 351 Cryptocurrencies  |
| Indices           | 91.178    | 63 Exchanges          |
| Money Markets     | 1.367     | 2 Exchanges           |

For ETFs/Funds the `Sectors` column = `family` count and the
`Industries` column = `category` count, which is how those numbers
were always interpreted in the README — now backed by real columns.

## Test plan

- [ ] `pytest tests/` — 52 tests pass (was 49 before this PR)
- [ ] `test_exchange_country_one_to_one` passes for ETFs and Funds
- [ ] `test_no_symbol_collisions_across_asset_classes` passes
- [ ] CI Update-README-Statistics job rewrites README correctly
- [ ] `black --check tests/ financedatabase/` clean
- [ ] Spot-check: `fd.ETFs(use_local_location=True).show_options(selection="country")` returns 33 countries
- [ ] Spot-check: `fd.Funds(use_local_location=True).select(country="Spain")` returns ~5,383 rows

Author: dokson

.github/workflows/database_update.yml

@@ -426,8 +426,103 @@ jobs:
           if: steps.run.outputs.status != '0'
           run: exit "${{ steps.run.outputs.status }}"
 
+  Update-README-Statistics:
+    needs: [Add-New-Ticker, Update-Compression-Files, Update-Categorization-Files]
+    runs-on: ubuntu-latest
+    steps:
+      - name: checkout repo content
+        uses: actions/checkout@v3
+      - name: pull changes
+        run: git pull https://${{secrets.PAT}}@github.com/JerBouma/FinanceDatabase.git main
+      - name: setup python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.13'
+      - run: pip install pandas
+      - name: Refresh the statistics tables in README.md
+        uses: jannekem/run-python-script-action@v1
+        with:
+          script: |
+            import re
+            from pathlib import Path
+            import pandas as pd
+
+            def _n(x: int) -> str:
+                """Format an integer with a thousand-separator dot (e.g. 160.113)."""
+                return f"{x:,}".replace(",", ".")
+
+            # Table 1: Equities / ETFs / Funds — Quantity, Sectors, Industries, Countries, Exchanges
+            table_1_rows = []
+            for asset, sectors_col, industries_col in [
+                ("Equities", "sector", "industry"),
+                ("ETFs",     "family", "category"),
+                ("Funds",    "family", "category"),
+            ]:
+                df = pd.read_csv(f"database/{asset.lower()}.csv", dtype=str)
+                table_1_rows.append((
+                    asset,
+                    _n(len(df)),
+                    _n(df[sectors_col].nunique()),
+                    _n(df[industries_col].nunique()),
+                    _n(df["country"].nunique()),
+                    _n(df["exchange"].nunique()),
+                ))
+
+            # Table 2: Currencies / Cryptocurrencies / Indices / Money Markets
+            table_2_rows = []
+            for asset, fname, col, label in [
+                ("Currencies",       "currencies.csv",   "quote_currency", "Currencies"),
+                ("Cryptocurrencies", "cryptos.csv",      "cryptocurrency", "Cryptocurrencies"),
+                ("Indices",          "indices.csv",      "exchange",       "Exchanges"),
+                ("Money Markets",    "moneymarkets.csv", "exchange",       "Exchanges"),
+            ]:
+                df = pd.read_csv(f"database/{fname}", dtype=str)
+                table_2_rows.append((asset, _n(len(df)), f"{_n(df[col].nunique())} {label}"))
+
+            def _fmt_table_1(rows):
+                h = "| Product           | Quantity   | Sectors    | Industries    | Countries | Exchanges |\n"
+                h += "| ----------------- | ---------- | ---------- | ------------- | --------- | --------- |"
+                b = "\n".join(
+                    f"| {a:<17} | {q:<10} | {s:<10} | {i:<13} | {c:<9} | {e:<9} |"
+                    for (a, q, s, i, c, e) in rows
+                )
+                return h + "\n" + b
+
+            def _fmt_table_2(rows):
+                h = "| Product           | Quantity  | Category              |\n"
+                h += "| ----------------- | --------- | --------------------- |"
+                b = "\n".join(
+                    f"| {a:<17} | {q:<9} | {c:<21} |"
+                    for (a, q, c) in rows
+                )
+                return h + "\n" + b
+
+            readme = Path("README.md")
+            text = readme.read_text(encoding="utf-8")
+            text = re.sub(
+                r"\| Product\s+\| Quantity\s+\| Sectors[^\n]*\| Industries[^\n]*\n\|[^\n]*\n(?:\|[^\n]*\n){3}",
+                _fmt_table_1(table_1_rows) + "\n",
+                text,
+                count=1,
+            )
+            text = re.sub(
+                r"\| Product\s+\| Quantity\s+\| Category[^\n]*\n\|[^\n]*\n(?:\|[^\n]*\n){4}",
+                _fmt_table_2(table_2_rows) + "\n",
+                text,
+                count=1,
+            )
+            readme.write_text(text, encoding="utf-8")
+            print("README.md statistics tables refreshed.")
+      - name: Commit README update
+        run: |
+          git config --global user.name 'GitHub Action'
+          git config --global user.email 'action@github.com'
+          git add README.md
+          git diff-index --quiet HEAD || git commit -m "Update README statistics"
+          git push
+
   Check-GICS-Categorisation:
-    needs:  [Add-New-Ticker, Update-Compression-Files, Update-Categorization-Files]
+    needs:  [Add-New-Ticker, Update-Compression-Files, Update-Categorization-Files, Update-README-Statistics]
     runs-on: ubuntu-latest
     steps:
       - name: checkout repo content

README.md

@@ -23,16 +23,16 @@ Some key statistics of the database:
 
 | Product           | Quantity   | Sectors    | Industries    | Countries | Exchanges |
 | ----------------- | ---------- | ---------- | ------------- | --------- | --------- |
-| Equities          | 158.429    | 12         | 63            | 111       | 83        | 
-| ETFs              | 36.786     | 295        | 22            | 111       | 53        |
-| Funds             | 57.881     | 1541       | 52            | 111       | 34        |
+| Equities          | 160.113    | 11         | 62            | 113       | 84        |
+| ETFs              | 36.485     | 320        | 51            | 33        | 51        |
+| Funds             | 57.853     | 1.540      | 74            | 24        | 33        |
 
 | Product           | Quantity  | Category              |
 | ----------------- | --------- | --------------------- |
 | Currencies        | 2.556     | 175 Currencies        |
-| Cryptocurrencies  | 3.367     | 352 Cryptocurrencies  |
-| Indices           | 91.183    | 64 Exchanges          |
-| Money Markets     | 1.367     | 3 Exchanges           |
+| Cryptocurrencies  | 3.367     | 351 Cryptocurrencies  |
+| Indices           | 91.178    | 63 Exchanges          |
+| Money Markets     | 1.367     | 2 Exchanges           |
 
 The Finance Database is used within or referenced by: