Skip to content

OPENNLP-1879: Gazetteer and geocoder API with a bundled Natural Earth implementation#1154

Draft
krickert wants to merge 6 commits into
apache:mainfrom
ai-pipestream:gazetteer-api
Draft

OPENNLP-1879: Gazetteer and geocoder API with a bundled Natural Earth implementation#1154
krickert wants to merge 6 commits into
apache:mainfrom
ai-pipestream:gazetteer-api

Conversation

@krickert

@krickert krickert commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Adds the location-intelligence seam and its first implementation.

Contracts in opennlp-api (opennlp.tools.geo): GeoPoint, GazetteerEntry, AttributeValue, Gazetteer, GeoResolution, Geocoder. Generic in shape: the dataset is an implementation, never the contract. Attributes are provenance-tagged, with a documented external-id key convention (fips, geoid, zcta, wikidata, geonames, whosonfirst) so downstream enrichment joins on stable keys, and ISO 3166-1 alpha-2 is the region join key shared with the emoji annotation layer. No new dependencies.

Implementation in a new opennlp-extensions/opennlp-geo module:

  • BundledGazetteer over a project-authored 7,342-row public-domain table derived from Natural Earth Populated Places: fail-loud cursor parser, lazy single load, matching through the shared NFC/case/accent folding chain.
  • PopulationPriorGeocoder: deterministic population-plus-feature-class ranking, documented heuristic confidence, unresolved mentions omitted.

LICENSE gains the Natural Earth public-domain section; audit tests enforce the data file's documented claims row by row.

Data derivation: Natural Earth 1:10m Populated Places via the nvkelso GitHub mirror (upstream commit 789c9904, VERSION 5.2.0-pre), NAMEASCII canonical, wikidata/geonames/whosonfirst ids carried as published, no fabricated fips/geoid/zcta.

Verification: api 328/0 (42 new), opennlp-geo 55/0, 19-module reactor verify green.

Follow-ups (deliberately out of this PR): opennlp-distr wiring plus the binary LICENSE section, a context-minimization Geocoder, the Overture and GeoNames tiers, and the gRPC mirror after one review round.

https://issues.apache.org/jira/browse/OPENNLP-1879

@mawiesne mawiesne marked this pull request as draft July 8, 2026 04:32
@krickert krickert force-pushed the gazetteer-api branch 3 times, most recently from f0adc29 to 6e5a2d8 Compare July 8, 2026 14:44
krickert added 4 commits July 8, 2026 15:05
@krickert
Add gazetteer and geocoder contracts in opennlp.tools.geo
ba6663e 
Define the location-intelligence seam in opennlp-api: GeoPoint (WGS84,
range-validated), GazetteerEntry (immutable, defensive copies,
provenance-tagged attribute map), AttributeValue, Gazetteer (name, id,
and ISO 3166-1 alpha-2 region lookup), GeoResolution (mention span in
original document coordinates, confidence in [0, 1]), and Geocoder.

The shape is deliberately generic: neutral field names, source-scoped
identifiers, and dataset-specific extras carried as provenance-tagged
attribute values, so bundled, downloaded, and user-supplied datasets
all sit behind the same interface. Attribute keys for stable external
identifiers (fips, geoid, zcta, wikidata, geonames, whosonfirst)
follow a documented convention so downstream enrichment data can join
through an entry without any dataset being bundled. The country code
is the join key shared with the emoji annotation layer.

No new dependencies in opennlp-api.
@krickert
Add opennlp-geo module with the bundled gazetteer implementation
675d113 
New opennlp-extensions/opennlp-geo module (spellcheck precedent: the
core jars stay lean) holding the reference Gazetteer implementation.

BundledGazetteer reads a semicolon separated populated-places table
(source;recordId;name;altNames;lat;lon;iso2;containment;population;
featureClass;attributes), cursor-parsed line by line with fail-loud
IllegalArgumentException naming the resource and line for malformed
rows, loaded lazily once and immutable after load. Both indexed names
and queries fold through the same matching chain (UAX apache#29 word
segmentation, then NFC, case fold, accent fold per token), so case,
accent, and hyphenation variants of a name produce the same match key.
Lookup results rank by population descending, then a CITY over ADMIN
over POI feature-class prior, then source and record id, giving a
deterministic total order. byRegion serves the ISO 3166-1 alpha-2 join
key with the most populous entry for the region.

The bundled data table itself follows in the next commit; tests here
run against crafted in-memory rows.
@krickert
Bundle the Natural Earth populated-places table with audit coverage
ea7fe5b 
Adds the project-authored gazetteer table derived from the Natural
Earth 1:10m Populated Places dataset (public domain): 7342 rows, one
per populated place, with the NE id as the stable record id, the
transliterated ASCII name (NAMEASCII), alternate names, WGS84 point
coordinates, ISO 3166-1 alpha-2 country code, the admin-1 name as the
v1 containment element, POP_MAX, featureClass CITY, and the external
identifiers Natural Earth publishes (wikidata, geonames, whosonfirst)
as provenance-tagged attributes. The full derivation record (source,
mirror commit, extraction date, field mapping, public-domain
dedication) is in the file header.

The table is deliberately pure ASCII in v1: non-ASCII primary names
are carried as the ASCII transliteration Natural Earth provides, with
the Unicode forms omitted; the folded matching chain keeps accented
queries working against the ASCII rows. The audit test enforces the
header's claims row by row: row-count floor, pure-ASCII bytes, no
duplicate ids, country-code shape (the emoji-layer join key), lat/lon
range, the attribute-key whitelist, and deterministic lookups.

LICENSE gains the Natural Earth public-domain section; per the ASF
guidance for public-domain data there is no NOTICE entry.
@krickert
Add PopulationPriorGeocoder with a documented heuristic confidence
6e8ece7 
Resolves each location mention independently: candidates come from the
gazetteer lookup and are re-ranked here under the module's shared
deterministic order (population descending, CITY over ADMIN over POI
on population ties, then source and record id), so the result never
depends on the gazetteer's return order. Unresolved mentions are
omitted; results stay aligned to the input spans in input order.

Confidence is an explicitly documented heuristic prior, not a
probability: 0.9 for a single candidate, otherwise
0.5 + 0.4 * (p1 - p2) / (p1 + p2), monotonic in the relative
population separation between winner and runner-up, 0.5 on a dead
tie, and never 1.0. Document-context-aware scoring (co-occurring
mentions constraining each other) is the named follow-up as a second
Geocoder implementation; resolve() already receives the whole
document for that reason.

The candidate order moves into a shared package-private
CandidateRanking used by both BundledGazetteer and the geocoder.
@rzo1

rzo1 commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Hi @krickert - as mentioned on Slack I currently dont have the time for a manual review but I just let Fable do a comprehensive review on this PR. Here is the result:

Correctness / API design

  • opennlp-extensions/opennlp-geo/src/main/resources/opennlp/geo/naturalearth-populated-places.txt: Upstream Natural Earth mojibake is carried verbatim into the bundled table, e.g. indexed names like AmundseniScott South Pole Station, EdDamer, Sdid Bouzid and roughly 20 containment values like ThMi Bmnh, MUdenine, Kasssrine. Corrupted indexed names are silently unfindable by their correct spelling, the header's derivation record ("those containment elements are omitted") is inaccurate for these rows, and the audit test (ASCII/shape checks only) cannot detect letter-level corruption.
  • opennlp-api/src/main/java/opennlp/tools/geo/Geocoder.java: The new Geocoder/Gazetteer seam duplicates the documented use case of the existing non-deprecated opennlp.tools.entitylinker API (whose Javadoc cites gazetteer location lookup as its primary example) with no cross-reference, deprecation, or guidance on which contract new consumers should target.
  • opennlp-api/src/main/java/opennlp/tools/geo/Gazetteer.java: The lookup/byId/byRegion/resolve methods document no failure mode for I/O or backend errors even though the Javadoc explicitly anticipates database-backed and remote-service implementations. Each implementation will invent its own undocumented unchecked exception, callers cannot handle backend failure portably, and a checked exception channel cannot be added compatibly after release (the existing EntityLinker declares IOException).
  • opennlp-api/src/main/java/opennlp/tools/geo/GazetteerEntry.java: Interop-critical string vocabularies (featureClass CITY/ADMIN/POI, attribute keys fips/geoid/zcta/wikidata/geonames/whosonfirst, source marker UNSPECIFIED) exist only as Javadoc conventions, yet CandidateRanking hard-codes the featureClass literals, so a third-party Gazetteer with different spellings (e.g. City, PPL) silently loses the feature-class tie-breaking prior in PopulationPriorGeocoder. These should be public constants or an enum in opennlp.tools.geo.
  • opennlp-api/src/main/java/opennlp/tools/geo/Gazetteer.java (also GazetteerEntry.java, AttributeValue.java, BundledDataAuditTest.java): Published API Javadoc repeatedly references an "emoji annotation layer" and a test comment names a class EmojiAnnotationJoin, but neither exists anywhere in the codebase. These references should be dropped or phrased as forward-looking.
  • opennlp-extensions/opennlp-geo/pom.xml: The jar will bundle the Natural Earth-derived data with only the generic Apache-2.0 META-INF/LICENSE from maven-remote-resources; the new Natural Earth public-domain section exists only in the root LICENSE. The Maven Central jar is itself an ASF distribution bundling third-party data, so its LICENSE should note the public-domain material when the promised LICENSE wiring lands.

Minor

  • opennlp-extensions/opennlp-geo/src/main/java/opennlp/geo/PopulationPriorGeocoder.java: confidence() can overflow long arithmetic (first + second, first - second) for populations near Long.MAX_VALUE, producing a confidence outside [0, 1] that makes the GeoResolution constructor throw. Requires an adversarial custom Gazetteer; computing the ratio in double or clamping closes it.
  • opennlp-api/src/main/java/opennlp/tools/geo/GazetteerEntry.java: population uses 0 as the sentinel for "unknown", conflating missing data with a genuinely zero population; this feeds directly into the confidence formula and cannot be changed compatibly after release, so the choice should be made deliberately now.
  • opennlp-extensions/opennlp-geo/src/main/java/opennlp/geo/BundledGazetteer.java: The only public entry point is a permanent JVM-wide static singleton (getInstance()) with a package-private constructor and parser, so users cannot build a gazetteer over their own rows in the documented table format or release the 7,342-row index. A public constructor/factory taking entries would give the same convenience without locking in the singleton.
  • opennlp-extensions/opennlp-geo/src/main/java/opennlp/geo/BundledGazetteer.java: indexName silently skips a name whose fold key is empty, so a punctuation-only name would load but be unreachable by lookup, contradicting the module's fail-loud stance; either reject such rows or add an audit assertion that every bundled name yields a non-empty fold key.
  • opennlp-extensions/opennlp-geo/src/main/java/opennlp/geo/PopulationPriorGeocoder.java: resolve() copies and re-sorts the candidate list for every mention even though BundledGazetteer.lookup already returns lists pre-sorted with the same comparator; skipping the copy/sort for size-1 lists or a known-trusted gazetteer would take this out of the per-mention hot path.
  • opennlp-extensions/opennlp-geo/src/test/java/opennlp/geo/BundledGazetteerTest.java: Several documented fail-loud parser behaviors have no test, e.g. duplicate attribute keys, empty pipe-list elements from leading/trailing/doubled pipes, and out-of-range longitude (latitude has a test, longitude does not).
  • opennlp-extensions/opennlp-geo/src/test/java/opennlp/geo/BundledDataAuditTest.java: The class Javadoc claims the data file header's claims are enforced, but the exact "7342 rows" claim is only checked as a >= 7000 floor, so a regeneration with a stale header passes; assert the exact count or parse it from the header.
  • opennlp-extensions/opennlp-geo/src/main/resources/opennlp/geo/naturalearth-populated-places.txt: The file carries the standard ASF Apache-2.0 header even though it is substantively a derivation of a third-party public-domain dataset; per ASF source-header policy the ASF header should likely be dropped in favor of the existing derivation-record comment plus the LICENSE pointer.
  • opennlp-extensions/opennlp-geo/src/main/resources/opennlp/geo/naturalearth-populated-places.txt: The ~950 KB generated table is committed without the extraction/transformation script, so refreshing to a newer Natural Earth release means re-implementing the documented rules by hand; please attach or commit the generation tooling.

Human review will follow.

@krickert krickert self-assigned this Jul 10, 2026
@krickert
OPENNLP-1879: Address review: repair upstream mojibake and harden the…
224cfe6 
… gazetteer seam

The bundled Natural Earth table is now produced by a committed generation script
(opennlp-geo/dev/derive-populated-places.py) that detects and handles the encoding damage the
upstream distribution itself carries. The canonical name derives from the upstream display name
folded to ASCII, with the upstream transliteration as a fallback and, where it differs, as an
alternate name; values that announce damage with '?' are omitted and counted, values verified
against accent-bearing upstream siblings or documented place names are repaired through audited
tables in the script, and one-character corruptions of a name at an accented position are
dropped. The script hard-fails on anomalies it does not recognize, so a data refresh forces a
reviewed decision. The regenerated table changes 181 of 7342 rows. The data file now carries the
derivation record and a LICENSE pointer instead of an ASF source header, is listed in
rat-excludes, and the Natural Earth public-domain section is added to the binary distribution
LICENSE alongside the existing root LICENSE section.

On the seam, Gazetteer.lookup, byId, byRegion and Geocoder.resolve declare IOException so
database-backed and remote implementations have a portable failure channel, matching the
EntityLinker precedent, and both interfaces now cross-reference opennlp.tools.entitylinker with
guidance on which contract new consumers should target. The feature-class and attribute-key
vocabularies are published as constants on GazetteerEntry, the UNSPECIFIED provenance marker as a
constant on AttributeValue, and CandidateRanking matches on the constants so any gazetteer using
them gets the feature-class prior. The population zero sentinel is documented as a deliberate
choice inherited from upstream conventions, and javadoc references to an unshipped annotation
layer are rephrased to the underlying ISO 3166-1 join-key fact.

BundledGazetteer gains a public fromEntries factory so callers can index their own rows without
the singleton, and a name that folds to an empty match key now fails loud at load time instead of
loading an unreachable record. PopulationPriorGeocoder computes confidence in double with
clamping so adversarial populations near Long.MAX_VALUE cannot break the GeoResolution contract,
and skips the per-mention copy and sort for single-candidate lookups. The audit test enforces the
header row count exactly, rejects mojibake artifact patterns and damage markers across every
name, alternate name, and containment value, and asserts every name yields a non-empty fold key;
parser fail-loud coverage now includes duplicate attribute keys, empty pipe-list elements, and
out-of-range longitude, with use-case tests for the custom-entries factory and the third-party
feature-class prior.
@krickert

krickert commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, the data finding especially. Addressed in 224cfe6:

  • Mojibake in the bundled table: fixed. The table is now produced by a committed script (opennlp-geo/dev/derive-populated-places.py) that detects the upstream damage by character-class artifacts and by cross-checking against accent-bearing sibling values, repairs values it can verify (67 containment elements, 5 alternate names, 1 canonical name, each recorded in audited tables in the script), drops one-letter corruptions of a name at an accented position (21 values like Zarich for Zurich), and keeps omitting the question-mark-damaged containment (38, now counted exactly in the header). The canonical name derives from the display name folded to ASCII with the upstream transliteration as fallback and alternate, so AmundseniScott, EdDamer, and Sdid Bouzid are gone and the correct spellings resolve. The script hard-fails on unrecognized anomalies, so a data refresh forces a reviewed decision. 181 of 7342 rows changed; the audit scans every name, alternate, and containment value for artifact patterns and asserts the exact header row count.
  • entitylinker overlap: the Gazetteer and Geocoder javadoc now cross-reference opennlp.tools.entitylinker with the guidance that new location consumers target this seam while EntityLinker remains the generic enrichment contract. entitylinker itself is untouched.
  • I/O failure channel: lookup, byId, byRegion, and Geocoder.resolve declare throws IOException, matching the EntityLinker precedent; the in-memory implementations document that they never throw it.
  • String vocabularies: promoted to public constants (GazetteerEntry.FEATURE_CLASS_* and ATTRIBUTE_KEY_*, AttributeValue.SOURCE_UNSPECIFIED). CandidateRanking matches on the constants, and a use-case test shows a third-party gazetteer using them getting the feature-class prior.
  • Emoji annotation references: removed from published javadoc; the remaining text states only the ISO 3166 fact, naming no unshipped class.
  • Jar LICENSE wiring: the Natural Earth public-domain section is in the binary distribution LICENSE. On per-jar META-INF/LICENSE, the project convention for bundled data is root plus distribution LICENSE, the same convention the runtime jar's Unicode data files follow, so opennlp-geo stays consistent rather than special-casing one jar.
  • confidence() overflow: computed in double with clamping to [0, 1]; adversarial test with populations at Long.MAX_VALUE.
  • population zero sentinel: kept deliberately and documented as such; it mirrors the upstream conventions and the confidence prior documents that it treats 0 as absent evidence.
  • Singleton: public BundledGazetteer.fromEntries(List) added with the same indexing and ranking; getInstance() stays for the bundled table, and a use-case test builds a custom gazetteer.
  • indexName silent skip: fails loud at load naming the record; the audit additionally asserts every bundled name folds to a non-empty key.
  • resolve() re-sort: single-candidate lookups skip the copy and sort; multi-candidate lists are still re-sorted because the seam only promises best-effort ordering from arbitrary gazetteers, with a comment saying so.
  • Parser fail-loud tests: duplicate attribute keys, leading, trailing, and doubled pipes, and out-of-range longitude.
  • Row-count floor: the audit parses the count from the header and asserts it exactly.
  • ASF header on the data file: dropped in favor of the derivation record plus the LICENSE pointer; the file rides rat-excludes like the bundled Unicode data files.
  • Generation tooling: committed, with the extraction date and upstream commit recorded in the derivation record.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@krickert krickert

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@krickert @rzo1