OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures#1166
Draft
krickert wants to merge 11 commits into
Draft
OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures#1166krickert wants to merge 11 commits into
krickert wants to merge 11 commits into
Conversation
…d SharingStemmer SnowballStemmer now routes each call to a per-thread generated engine via OwnerOrPerThreadState, the same pattern as the thread-safe *ME components, so one instance can be shared across threads without touching the generated Snowball code. StemmerFactory (api) captures stemmer configuration as an immutable, thread-safe seam for stateful engines; SharingStemmer adapts a factory into a shareable Stemmer for implementations that are not thread-safe themselves (e.g. PorterStemmer). NormalizationProfile.matchingAnalyzer() and TermAnalyzer are now safe to share across threads when stemming.
…unit tests MultiThreadedStemmerEval hammers shared SnowballStemmer instances for every algorithm, a SharingStemmer-wrapped PorterStemmer, and every NormalizationProfile matching analyzer from 8 threads, comparing each result against a single-threaded reference, mirroring MultiThreadedToolsEval. The unit tests now also cover all 21 algorithms under concurrency, the owner thread stemming concurrently with pool threads, virtual threads (a fresh thread per task), and Porter sharing.
…atch baseline Measures the thread-safe SnowballStemmer shared across threads and one per thread against a replica of the pre-patch plain-field implementation, at 1/8/32 threads. Results in BENCHMARKS.md: the OwnerOrPerThreadState overhead is within error bars up to 8 threads and only shows (~1.5x) at full 32-thread saturation.
…to-stem mappings Natural text is Zipf-distributed, so the same words are stemmed constantly. CachingStemmer wraps a StemmerFactory with a bounded per-thread LRU (default 1024 entries) following the same OwnerOrPerThreadState pattern as the *ME components: no cross-thread sharing, thread-safe regardless of the delegate. On the JMH zipf workload the cache is a ~34x throughput multiplier; on a cache-hostile uniform workload over 8x the cache capacity it still breaks even. Results and a corrected JMH invocation are documented in BENCHMARKS.md.
…ngStemmer Builder.stem(StemmerFactory) now wraps the factory in a CachingStemmer (default capacity), with a stem(StemmerFactory, int) overload for explicit cache sizing, and NormalizationProfile.matchingAnalyzer() builds through the factory again so profile analyzers get the per-thread stem cache for free. Analyzer results are unchanged; repeated words resolve from the cache instead of being re-stemmed.
krickert
changed the title
…-indirection factory products SharingStemmer forwards stemAll to its per-thread delegate so a multi-output engine keeps its full result list through the wrapper, and all three thread-routing stemmers expose clearThreadLocalState(), mirroring the thread-safe ME components, so pooled threads can release their engine and cache instead of retaining them until the instance is collected. The caching javadoc and BENCHMARKS.md now state that the memoization is keyed to the physical thread and what that means on a virtual-thread-per-task executor. SnowballStemmerFactory products are now thread-confined stemmers driving their engine through a plain field, so the one-stemmer-per-thread pattern and the per-thread delegates inside the caching and sharing wrappers pay none of the shared-instance thread routing. The SnowballStemmer constructor validates algorithm and repeat like the factory, validation is IllegalArgumentException across the new surface, the LRU map is sized for its load factor so a full cache never rehashes, and StemmerFactory documents that it is a plain supplier, not one of the BaseToolFactory tool factories; its one-shot stem convenience method is gone. Tests cover the multi-output forwarding, the repeat pass-through with a witness word that stems differently at repeat one and two, the owner cache reset and worker-thread delegate release behind clearThreadLocalState, and the constructor validation.
krickert
changed the title
…g, cache controls The Stemmer and StemmerFactory interfaces state their contracts without implementation or threading claims, and null words throw IllegalArgumentException across the API (Snowball, factory products, caching and sharing wrappers, Porter), pinned by tests. SnowballStemmer now composes SharingStemmer so the per-thread routing pattern lives in one class. CachingStemmer gains clearCache() for the calling thread, interns cached stems through StringInterners, and both stateful classes document the container release path of the *ME components.
…erflow The cached stems are no longer routed through the JVM-wide interner: its default implementation holds strong references forever, so open-vocabulary input would grow without bound and defeat the documented per-thread cache cap. The LRU alone keeps the bound honest. The LRU table sizing is computed in double arithmetic because the int expression overflows to a negative capacity near Integer.MAX_VALUE; a test pins construction at the extreme. clearCache() documents that it initializes state for a thread that has not stemmed yet.
…ures Sixteen stemmers in the new opennlp.tools.stemmer.light package: light and minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk varieties), Swedish, plus light stemmers for Finnish, Hungarian, Italian, Portuguese, Russian and the English minimal stemmer. Adapted from Apache Lucene's analysis-common module; the UniNE algorithms carry Jacques Savoy's BSD notice in each source file and in the distribution LICENSE. These fill the tier between no stemming and the aggressive Snowball algorithms. Every stemmer is stateless and thread-safe, implements Stemmer, and is its own StemmerFactory. Input is expected lowercase, matching the algorithms' original contract; null input fails loudly. Parity is asserted against the original implementations: bundled fixtures sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per stemmer, all pairs for the small Norwegian lists) regenerated by running the Lucene classes, so any behavioral drift fails the test. The Galician and Portuguese minimal stemmers are excluded because they build on the RSLP rule engine, which is its own effort.
The review conventions keep JIRA keys out of source comments; the sizing note stands on its own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
STACKED on #1163 (stemmer-factory): this branch builds on that PR's StemmerFactory contract, and since that head lives on a fork it cannot be this PR's base ref. Until #1163 merges, the diff here shows its commits too; only the last commit (592380e) is this PR. After #1163 lands, the branch rebases onto main and the diff collapses to this change alone.
Adds sixteen stemmers in a new opennlp.tools.stemmer.light package: light and minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk varieties), and Swedish, plus light stemmers for Finnish, Hungarian, Italian, Portuguese, and Russian, and the English minimal stemmer. These fill the tier between no stemming and the aggressive Snowball algorithms: they remove plural and inflectional endings with far less conflation. Adapted from Apache Lucene's analysis-common module; the UniNE algorithms carry Jacques Savoy's BSD notice in each source file and in the distribution LICENSE.
Every stemmer is stateless and thread-safe, implements Stemmer, and is its own StemmerFactory. Input is expected lowercase, matching the algorithms' original contract; null input fails loudly.
Parity is asserted against the original implementations: bundled fixtures sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per stemmer, complete lists for Norwegian) regenerated by running the originals, so any behavioral drift fails the test. The Galician and Portuguese minimal stemmers are excluded because they build on the RSLP rule engine, which is its own effort.
JIRA to follow; title gets the key once filed.