OPENNLP-1880: Lexical knowledge base seam with WN-LMF and WNDB readers and a Morphy lemmatizer

[krickert]

Adds the format-agnostic lexical knowledge base seam to opennlp-api (opennlp.tools.wordnet: LexicalKnowledgeBase, Synset, WordNetPos, WordNetRelation), and a new opennlp-extensions/opennlp-wordnet module with two clean-room readers behind it:

• WnLmfReader: a StAX reader for WN-LMF XML, Open English WordNet's interchange format. XXE-hardened per the OWASP posture for DOCTYPE-bearing formats: the DOCTYPE line real OEWN releases ship is tokenized and skipped while external entities and the external DTD subset stay fully disabled, so an unmodified download parses directly; a dedicated test proves a DOCTYPE-declared internal-subset entity payload fails loud rather than expanding.
• WndbReader: a reader for legacy Princeton WNDB directories, validating the format's documented offset contract.

Both produce equivalent immutable, thread-safe lexicon views, pinned by a structural equivalence test over matching miniature fixtures. On top of the seam, MorphyLemmatizer implements the existing Lemmatizer interface with the documented Morphy algorithm: exception-list lookup first, then per-POS detachment rules with every candidate validated against the lexicon.

Naming: the seam interface is LexicalKnowledgeBase rather than a WordNet-branded name; the contract is generic and only the format-specific readers carry the WordNet name.

The module ships code only; users point it at a WordNet database they downloaded. Verified against Princeton 3.0 (117,659 synsets) and OEWN 2024 (120,630 synsets). No third-party WordNet library is used or referenced.

Verification: api 303/0 (+17), opennlp-wordnet 86/0, full reactor verify green.

Small upstream note found along the way: DictionaryLemmatizer javadoc says unknown = "0" but the code returns the letter "O"; Morphy matches the code. Filed separately.

https://issues.apache.org/jira/browse/OPENNLP-1880

[krickert]

There's a minor windows-specific build issue due to line endings and Windoze. Will flip back to review once I find my windows machine and test. Code is the same though.

[rzo1]

Hi @krickert - as mentioned on Slack I currently dont have the time for a manual review but I just let Fable do a comprehensive review on this PR. Here is the result:

• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfReader.java: SynsetRelation elements with relType="other" are rejected with "Unknown relation type other", although the class javadoc documents that "other" relations are skipped. The skip filter is only applied to SenseRelations in build(); SynsetRelations go through parseRelation unconditionally, so a WN-LMF document that uses relType="other" on a SynsetRelation (permitted by the DTD, emitted by several OMW-family wordnets) fails to load. Tests only cover the SenseRelation case.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/MorphyLemmatizer.java: posFromTag's first-character fallback is broader than its javadoc claims. The single-letter WordNet codes "a" and "s" are matched as prefixes, so multi-letter tags like UD AUX, ADP, SCONJ or Penn SYM are misclassified as ADJECTIVE instead of yielding the documented unknown-word result (e.g. AUX "was" is looked up as an adjective and returns "O"). The "a"/"s" cases should require tag.length() == 1, and testPosFromTagMapping should exercise such tags.
• pom.xml: The new opennlp-wordnet artifact is not registered in the root pom's dependencyManagement and not added as a dependency of opennlp-distr, unlike the other extension modules (opennlp-morfologik, opennlp-spellcheck, opennlp-uima). As-is it will not ship in the binary distribution or aggregated apidocs, and reactor modules cannot reference it without an explicit version.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WndbReader.java: The new public loading entry points (WndbReader.read, WnLmfReader.read, MorphyExceptions.load) signal I/O failures as UncheckedIOException and malformed data as IllegalArgumentException, diverging from the established convention of checked IOException / InvalidFormatException used by comparable loaders (DictionaryLemmatizer, model loaders). Once released this contract cannot be tightened without breaking source compatibility, so it should be decided deliberately before the first release.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfReader.java: The "similar"-on-verb-synset to VERB_GROUP mapping has zero test coverage in either reader; the WN-LMF fixture only carries "similar" on adjective synsets and the WNDB fixture has no "$" pointer, so a regression in the only POS-dependent relType mapping would pass the whole suite.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WnLmfReader.java: Duplicate Sense ids and duplicate LexicalEntry ids are silently overwritten (plain map.put) while duplicate Synset ids fail loud, so a malformed document can have sense relations lifted onto the wrong synset without any error.
• opennlp-api/src/main/java/opennlp/tools/wordnet/WordNetPos.java: The name uses mixed-case "Pos" while the rest of the codebase uppercases the abbreviation (POSTagger, POSModel, POSSample); this public type cannot be renamed after release, so consider WordNetPOS or a spelled-out name.
• opennlp-api/src/main/java/opennlp/tools/wordnet/Synset.java: Synset is a public record whose canonical five-component constructor becomes frozen API; since the seam explicitly anticipates additive extension, adding a component later requires permanently keeping an overload of the old constructor. Worth confirming the v1 component set (per-sense data, lexicographer file) before release.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WndbReader.java: Relation-target synset id strings are minted fresh per pointer and never canonicalized against the synset table (~660K pointers in Princeton 3.0), retaining tens of MB of duplicate short Strings in a lexicon designed to be held in memory long-term. Interning against the canonical RawSynset.id during resolve() would collapse them; the same applies to WnLmfReader.resolveRelations.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/InMemoryWordNetLexicon.java: The constructor's referential-integrity validation (key/id mismatch, dangling relation target, sense-order entry referencing unknown synset) has no direct test; a future third reader would rely on these checks and a regression would go unnoticed.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/WndbReader.java: The RawPointer record's lineNumber component is dead code; it is populated at every parse but never read since resolve() reports errors with the owning synset's location. Either drop it or use it for pointer-precise error messages.
• opennlp-extensions/opennlp-wordnet/src/main/java/opennlp/wordnet/MorphyExceptions.java: The lemma-folding logic (replace('_',' ').toLowerCase(Locale.ROOT)) is duplicated verbatim in InMemoryWordNetLexicon.LemmaKey.of, and splitOnSpaces is duplicated in WnLmfReader.Parser. The fold agreement is load-bearing for MorphyLemmatizer, but nothing (shared helper or cross-referencing test) prevents the copies from drifting apart.
• opennlp-extensions/opennlp-wordnet/src/test/java/opennlp/wordnet/WnLmfReaderTest.java: Several documented fail-loud paths are untested: duplicate synset id, synset/member POS mismatch, SenseRelation targeting an undeclared sense, and the structural-misplacement errors (Lemma outside LexicalEntry, Sense before Lemma, SenseRelation outside Sense, SynsetRelation outside Synset).

Human review will follow.

[krickert]

Thanks, every finding is addressed in 69ead68. Point by point:

• SynsetRelation relType="other": now skipped exactly like the SenseRelation case (documented skip, not rejection), applied at parse time. A test covers the SynsetRelation case alongside the existing SenseRelation one.
• posFromTag over-matching: the WordNet letter codes a and s match only when tag.length() == 1, so AUX, ADP, SCONJ, and SYM map to no part of speech and yield the unknown marker. The tag-mapping test pins all four, lemmatizer-level cases pin that AUX "was" is unknown and AUX "taller" no longer detaches to "tall", and the javadoc matches the implementation.
• Root pom and distribution wiring: opennlp-wordnet is in the root dependencyManagement, in opennlp-distr's dependencies, and in the bin assembly's apidocs fileSets, mirroring opennlp-spellcheck.
• Exception contract: all three loading entry points (WndbReader.read, WnLmfReader.read, MorphyExceptions.load) declare IOException; malformed content raises InvalidFormatException, matching DictionaryLemmatizer and the model loaders, while null and argument validation stays IllegalArgumentException. A failing-stream test pins that an underlying I/O error surfaces as IOException rather than a malformed-document report.
• similar-on-verb to VERB_GROUP: covered in both readers. WN-LMF parses an inline document with similar between two verb synsets and asserts VERB_GROUP with SIMILAR_TO absent; WNDB builds a minimal database in a temp directory with programmatically computed byte offsets so the $ pointer resolves, asserting VERB_GROUP in both directions.
• Duplicate Sense and LexicalEntry ids: both fail loud with InvalidFormatException naming the id, symmetric with the duplicate-synset check; tests for both.
• WordNetPos naming: renamed to WordNetPOS across the API and the module, consistent with POSTagger and POSModel.
• Synset record components: kept as the deliberate v1 contract. Per-sense data and lexicographer files are reader-level concerns the seam intentionally omits; a later addition arrives as a new canonical constructor with the old one kept as a documented overload, the standard record-evolution path.
• Relation-target id duplication: relation resolution in both readers stores the synset table's canonical id instance per target, collapsing a full lexicon's pointer strings to the table's instances; tests assert instance identity between a related() target and the target synset's id() in both readers.
• InMemoryWordNetLexicon validation: direct constructor tests for key/id mismatch, dangling relation target, sense-order entry referencing an unknown synset, null maps, and the accepting case.
• RawPointer.lineNumber: now used; dangling-pointer errors report the pointer's own line, tested.
• Fold and split duplication: extracted into a package-private LemmaFolding helper used by MorphyExceptions, the lexicon's lemma key, and the WN-LMF members parsing, with a test pinning the fold and the key/query agreement.
• Untested WN-LMF fail-loud paths: tests for duplicate synset id, synset/member POS mismatch, SenseRelation to an undeclared sense, Lemma outside LexicalEntry, Sense before Lemma, SenseRelation outside Sense, and SynsetRelation outside Synset.

Module tests go from 86 to 112; reactor verify is clean.