OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms#1167
Draft
krickert wants to merge 11 commits into
Draft
OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms#1167krickert wants to merge 11 commits into
krickert wants to merge 11 commits into
Conversation
…edgeBase Renames the public seam interface so consumers describe what they are injecting (a lexical knowledge base) rather than naming the WordNet brand in a public API type. Concrete implementations that genuinely are WordNet-format readers (InMemoryWordNetLexicon, WnLmfReader, WndbReader) keep the WordNet name, since they describe the data format they read, not the contract itself.
…declarations Completes the LexicalKnowledgeBase rename in this file, and switches DOCTYPE handling from outright rejection to the OWASP-documented alternative for formats that require DOCTYPE support: the DOCTYPE event is skipped rather than resolved, while SUPPORT_DTD stays off (so no custom entity, internal or external, is ever declared), IS_SUPPORTING_EXTERNAL_ENTITIES and ACCESS_EXTERNAL_DTD stay off, and the XMLResolver still throws on any resolution attempt. This closes the same attack surface as outright rejection while letting the reader parse Open English WordNet releases unmodified, since real OEWN files ship a DOCTYPE line referencing the schema DTD. Test changes: testRejectsDoctype becomes testSkipsDoctypeDeclaration and now asserts a DOCTYPE-bearing document parses correctly instead of throwing. Added testInternalSubsetEntityIsNeverExpanded, which proves a classic XXE payload (a DOCTYPE-declared internal-subset entity pointing at a local file) still fails loud rather than expanding into the parsed output.
WNDB data lines embed their own byte offset, which WndbReader validates against the actual position. The repo's default `* text=auto` lets Git normalize these fixtures to CRLF on Windows checkout, inserting a CR before every LF and shifting every offset after the first line; every WndbReaderTest case failed on Windows CI as a result. Scoping -text to the fixture directory keeps the checkout byte-identical everywhere.
krickert
changed the title
krickert
changed the title
…iring, fail-loud coverage The three loading entry points (WndbReader.read, WnLmfReader.read, MorphyExceptions.load) now follow the codebase's loader convention: they declare IOException, malformed content raises InvalidFormatException, and null or argument problems stay IllegalArgumentException. Every internal content-error throw site, the javadocs, and all affected test assertions moved with the contract, and a failing-stream test pins that an I/O error surfaces as itself rather than as a malformed-document report. The public enum WordNetPos is renamed to WordNetPOS to match POSTagger and POSModel before the name freezes at release. Several reader gaps close: SynsetRelation elements with relType "other" are now skipped like their SenseRelation counterparts instead of rejected, duplicate LexicalEntry and Sense ids fail loud like duplicate synset ids, and relation resolution in both readers now stores the synset table's canonical id instance per target so a full lexicon keeps one copy of each id regardless of pointer count. Dangling WNDB pointer errors name the pointer's own line through the previously unread RawPointer component, and posFromTag accepts the WordNet letter codes a and s only as one-letter tags, so AUX, ADP, SCONJ, and SYM map to the unknown-word result instead of adjective lookups. The duplicated lemma fold and space-split logic collapse into a package-private LemmaFolding helper shared by the exception lists, the sense index, and the WN-LMF parser. The module is now wired into the root pom's dependencyManagement, the binary distribution's dependencies, and the aggregated apidocs assembly, mirroring the other extension modules. New coverage pins the verb-group mapping in both readers (a WN-LMF document with similar on verb synsets, and a constructed WNDB directory with computed byte offsets exercising the dollar pointer), the InMemoryWordNetLexicon constructor validation, the LemmaFolding behavior, and every documented WN-LMF structural fail-loud path.
… lexical knowledge base Expands a term into the synonyms sharing its synsets, the lemmas of its hypernym ancestors up to a configured depth (following both the direct and the instance relation), and optionally its direct hyponyms. Each expansion carries a deterministic heuristic weight: sense rank and every relation step multiply configurable decays, so consumers can discount looser expansions instead of treating them as the original term. Results exclude the input, deduplicate case-insensitively keeping the highest weight, and order stably by weight, kind, and term. Inflected input works through an optional Lemmatizer invoked with the WordNetPos name as the tag, which the Morphy lemmatizer understands: dogs expands through dog, mice through the exception list to mouse. Hypernym walks are visited-checked so malformed cyclic data terminates. Tests cover the graph behavior on a hand-built lexicon (sense ranking, decay arithmetic, dedup, cycle termination, validation) and run the whole stack over the miniature WN-LMF and WNDB fixtures, asserting both readers produce identical expansions.
krickert
force-pushed
the
wordnet-expansion
branch
from
a000fcc to
99cf800
Compare
krickert
force-pushed
the
wordnet-expansion
branch
from
a358c78 to
99cf800
Compare
Applies the review conventions from the OPENNLP-1869 review to the expansion layer: the default-choice essay shrinks to its contract sentence and the private helpers are documented.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
STACKED on #1155 (wordnet-api): this branch builds on that PR's LexicalKnowledgeBase seam, and since that head lives on a fork it cannot be this PR's base ref. Until #1155 merges, the diff here shows its commits too; only the last commit (a000fcc) is this PR. After #1155 lands, the branch rebases onto main and the diff collapses to this change alone.
Adds LexicalExpander to the opennlp-wordnet module: given a term, it produces weighted related terms from a LexicalKnowledgeBase, the synonyms sharing its synsets, the lemmas of hypernym ancestors up to a configured depth (following both the direct and the instance relation, visited-checked so cyclic data terminates), and optionally direct hyponyms. Each expansion carries a deterministic heuristic weight (sense-rank decay times depth decay, both configurable), results are deduplicated case-insensitively keeping the highest weight, the input term is never returned, and ordering is stable. An optional Lemmatizer fallback expands inflected input; the Morphy lemmatizer of this module plugs in directly, so dogs expands via dog and mice via mouse.
Tests cover the behavior over a hand-built graph with a controlled shape (sense ranking, depth decay, dedupe, cycle termination, hyponym opt-in, configuration validation) plus integration over the miniature WN-LMF and WNDB fixtures asserting both readers expand identically.