Add utility for validating security identifiers

[JonArnfred]

This PR adds a utility for validating security identifiers in the source CSV database.

I believe it's important to maintain that the project is primarily financial reference data. For that reason, the implementation:

• lives under scripts/
• is not exposed through the public financedatabase API
• uses the established python-stdnum package instead of implementing checksum algorithms locally
• adds python-stdnum only as a development dependency

The motivation is that identifier accuracy is part of maintaining a high-quality reference database, and these checks can detect malformed values deterministically before they are merged.

Validation performed

The utility checks:

• ISIN format and checksum
• CUSIP format and checksum
• FIGI format and checksum
• composite FIGI format and checksum
• share-class FIGI format and checksum
• consistency between valid US/Canadian ISINs and accompanying CUSIPs

Usage

Report-only mode is the default:

uv run python scripts/validate_identifiers.py

The report includes the CSV file, line number, symbol, field, value, and validation problem.

After reviewing the report, deterministic cleanup can be applied explicitly:

uv run python scripts/validate_identifiers.py --apply

Apply safety

--apply distinguishes between repairable, removable, and review-only values.

It can:

• normalize canonical values returned by python-stdnum
• remove a trailing .0 from identifiers (likely spreadsheet numeric formatting)
• restore lost leading zeros when a valid US/Canadian ISIN independently provides the exact CUSIP
• clear invalid ISIN and FIGI values without deterministic repairs

It does not automatically change:

• unresolved invalid CUSIPs
• individually valid but inconsistent ISIN/CUSIP pairs

Ambiguous cases are therefore left for manual review.

Current database results

The initial audit checked:

• 269,901 populated identifiers
• 14,883 valid US/Canadian ISIN/CUSIP pairs

It found:

• 286 invalid identifier values
• 288 valid but inconsistent ISIN/CUSIP pairs

Applying the deterministic cleanup produced:

• 272 CUSIPs normalized
• 5 invalid ISINs cleared
• 5 invalid FIGIs cleared
• 4 unresolved CUSIPs left unchanged for manual review
• 288 ISIN/CUSIP consistency issues left unchanged for manual review

The utility and resulting data changes are kept in separate commits so the effect of running --apply can be reviewed independently.

[JerBouma]

Thanks for this, isn't this a standard rule set? If it is, I'd like to split it up. A checker (added to GitHub Actions) that checks PRs on valid codes and then an automated GitHub Actions run on push to main to validate and fill potential missing values.

What do you think?

[JonArnfred]

Yes it just checks values against the standard rules for ISINs etc, and also clears .0 suffixes.

Sounds like a good idea. I'll update the PR today or tomorrow.

[JonArnfred]

I've added a third commit (ea5521a) implementing the suggested split into two CI paths.

Pull-request validation

A new Identifier Validation workflow runs for pull requests targeting main.

It audits both the PR database and its base revision using the same validator, then compares the findings. Only identifier problems introduced by the PR are shown as warning annotations.

This means:

• existing findings on untouched rows are not attributed to the contributor
• changing an unrelated field on a row with an existing identifier problem does not produce a new warning
• adding or changing an identifier to an invalid value produces a warning on the affected file and line
• findings remain warnings and do not fail the PR

Maintenance on main

The existing Database Update workflow now contains an identifier-maintenance job. It runs after new tickers have been added and before compression and categorisation files are regenerated.

The job:

1. audits the complete source database
2. applies deterministic repairs
3. uploads unresolved findings as a CSV artifact retained for 30 days
4. commits source-database repairs only when files actually changed
5. allows the subsequent compression jobs to include those repairs

The full report is stored as an artifact instead of creating hundreds of inline warnings. The workflow log contains only the repair and finding counts.

Ambiguous cases—such as individually valid but inconsistent ISIN/CUSIP pairs—remain unchanged and appear in the report for manual review.

Standard checksum rules cannot discover arbitrary missing identifiers. They can only validate existing values and perform repairs supported by deterministic evidence, such as recovering a spreadsheet-damaged CUSIP from a matching valid US or Canadian ISIN.