Thread safety is implementation specific.
*/ public interface Stemmer { + /** + * Stems {@code word}. + * + * @param word The input word. Must not be {@code null}. + * @return The stemmed form. + * @throws IllegalArgumentException Thrown if {@code word} is {@code null}. + */ CharSequence stem(CharSequence word); + + /** + * {@return all stem forms for {@code word}} Defaults to a single-element list from + * {@link #stem(CharSequence)}. Dictionary-based engines may override this to return + * multiple roots; delegating wrappers must forward it to preserve the full list. + * + * @param word The input word. Must not be {@code null}. + * @throws IllegalArgumentException Thrown if {@code word} is {@code null}. + */ + default ListDespite the name, this is not one of the {@code BaseToolFactory}-based tool factories (such + * as {@code LemmatizerFactory}) that are instantiated by name from a model manifest. It is a + * plain supplier of configured {@link Stemmer} instances and takes no part in the model-loading + * mechanism.
+ */ +public interface StemmerFactory { + + /** + * {@return a new {@link Stemmer}} + */ + Stemmer newStemmer(); +} diff --git a/opennlp-core/opennlp-runtime/BENCHMARKS.md b/opennlp-core/opennlp-runtime/BENCHMARKS.md index 68c3ac185e..21cb71a664 100644 --- a/opennlp-core/opennlp-runtime/BENCHMARKS.md +++ b/opennlp-core/opennlp-runtime/BENCHMARKS.md @@ -13,6 +13,8 @@ variance reporting. | `TokenizerMEBenchmark` | TokenizerME | 3 approaches | | `SentenceDetectorMEBenchmark` | SentenceDetectorME | 3 approaches | | `POSTaggerMEBenchmark` | POSTaggerME | 3 approaches x 2 cache configs | +| `SnowballStemmerBenchmark` | SnowballStemmer | 3 approaches (incl. plain-field baseline) | +| `CachingStemmerBenchmark` | CachingStemmer | cached vs uncached x 2 workloads | ### Approaches measured @@ -30,32 +32,88 @@ mvn test-compile -Pjmh \ -pl opennlp-core/opennlp-runtime -am \ -Dforbiddenapis.skip=true -Dcheckstyle.skip=true +# Materialize the test classpath once (JMH's forked JVMs inherit +# java.class.path, which mvn exec:java does not populate, running +# through exec:java fails with ClassNotFoundException: ForkedMain) +mvn dependency:build-classpath -pl opennlp-core/opennlp-runtime \ + -Pjmh -DincludeScope=test -Dmdep.outputFile=/tmp/cp.txt + +CP="opennlp-core/opennlp-runtime/target/classes:opennlp-core/opennlp-runtime/target/test-classes:$(cat /tmp/cp.txt)" + # Run all ME benchmarks -mvn exec:java -pl opennlp-core/opennlp-runtime \ - -Pjmh -Dexec.classpathScope=test \ - -Dexec.mainClass=org.openjdk.jmh.Main \ - -Dexec.args="opennlp.tools.*.ME*" +java -cp "$CP" org.openjdk.jmh.Main 'opennlp.tools.*.ME*' # Run POSTagger only (includes cacheSize param) -mvn exec:java -pl opennlp-core/opennlp-runtime \ - -Pjmh -Dexec.classpathScope=test \ - -Dexec.mainClass=org.openjdk.jmh.Main \ - -Dexec.args="POSTaggerMEBenchmark" +java -cp "$CP" org.openjdk.jmh.Main POSTaggerMEBenchmark ``` -### Regression testing (stock vs patched) +### Baseline comparison -Run the `newInstancePerCall` benchmark on both stock and patched -builds. The throughput numbers should be within JMH's error margin. +Run the `newInstancePerCall` benchmark on both `main` and this +branch. The throughput numbers should be within JMH's error margin. ```bash -# On stock (upstream/main): +# On main: # ... build and run as above, save output -# On patched (feature/thread-safe-me): +# On this branch: # ... build and run as above, compare ``` +### SnowballStemmer results (Linux, JDK 25, 32 cores, 2 forks x 10 iterations) + +`SnowballStemmerBenchmark` compares the thread-safe `SnowballStemmer` +(engine behind `OwnerOrPerThreadState`) against a baseline replica +that keeps the engine in a plain field (not shareable). +One op = stemming 16 English words. + +| Strategy | 1 thread | 8 threads | 32 threads | +|----------|---------:|----------:|-----------:| +| `sharedInstance` (one shared stemmer) | 560k ± 3k ops/s | 1.55M ± 0.17M | 3.16M ± 0.34M | +| `instancePerThread` (stemmer per thread) | 509k ± 26k ops/s | 1.60M ± 0.17M | 2.94M ± 0.11M | +| `legacyInstancePerThread` (plain-field baseline, stemmer per thread) | 544k ± 19k ops/s | 1.46M ± 0.16M | 4.77M ± 0.39M | + +At 1 and 8 threads the three strategies are within (or nearly within) +each other's error bars: the `OwnerOrPerThreadState` lookup is not +measurable against the cost of stemming itself. Only at full +saturation (32 threads, hyperthreaded) does the legacy plain-field +baseline pull ahead (~1.5x): with every hardware thread busy, the +per-call owner check plus `ThreadLocal` lookup is no longer hidden by +memory-level parallelism. Real pipelines stem as one stage among many, +so the saturated-microbenchmark gap is an upper bound, and the legacy +strategy was not shareable across threads in the first place. + +### CachingStemmer results (same environment) + +`CachingStemmerBenchmark` compares a `CachingStemmer` (per-thread LRU, +default 1024 entries, wrapping the English Snowball stemmer) against +the uncached shared stemmer. One op = 16 tokens from a 64k-token +stream. The `zipf` workload samples a 512-word vocabulary with 1/rank +weights (real-text repetition; the cache holds the whole vocabulary); +`diverse` samples an 8192-word vocabulary uniformly (8x cache +capacity: mostly misses plus constant eviction). + +| Workload | Strategy | 8 threads | 32 threads | +|----------|----------|----------:|-----------:| +| `zipf` | `cachedShared` | 48.5M ± 0.7M ops/s | 95.4M ± 0.9M | +| `zipf` | `uncachedShared` | 1.43M ± 0.13M | 2.75M ± 0.34M | +| `diverse` | `cachedShared` | 1.81M ± 0.51M | 3.45M ± 0.18M | +| `diverse` | `uncachedShared` | 1.08M ± 0.09M | 3.13M ± 0.12M | + +On the Zipf workload the cache is a ~34x throughput multiplier (raw +stemming becomes a hash lookup for the dominant vocabulary). On the +cache-hostile workload it still does not lose: the ~12% residual hit +rate pays for the eviction overhead. The cache more than recovers the +`OwnerOrPerThreadState` lookup cost observed in +`SnowballStemmerBenchmark` at full saturation. + +The cache is keyed to the physical thread, and these runs use a fixed +platform-thread pool whose threads live for the whole measurement. On +a virtual-thread-per-task executor every task starts with an empty +cache, so the multiplier only applies to repeats within one task; +workloads that stem a handful of words per task should expect +uncached-level throughput there. + ### POSTagger cache impact The `POSTaggerMEBenchmark` uses `@Param({"0", "3"})` for cache diff --git a/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/stemmer/CachingStemmerBenchmark.java b/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/stemmer/CachingStemmerBenchmark.java new file mode 100644 index 0000000000..ae0050b2b0 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/jmh/java/opennlp/tools/stemmer/CachingStemmerBenchmark.java @@ -0,0 +1,199 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer; + +import java.util.ArrayList; +import java.util.List; +import java.util.Random; +import java.util.concurrent.TimeUnit; + +import org.openjdk.jmh.annotations.Benchmark; +import org.openjdk.jmh.annotations.BenchmarkMode; +import org.openjdk.jmh.annotations.Fork; +import org.openjdk.jmh.annotations.Level; +import org.openjdk.jmh.annotations.Measurement; +import org.openjdk.jmh.annotations.Mode; +import org.openjdk.jmh.annotations.OutputTimeUnit; +import org.openjdk.jmh.annotations.Param; +import org.openjdk.jmh.annotations.Scope; +import org.openjdk.jmh.annotations.Setup; +import org.openjdk.jmh.annotations.State; +import org.openjdk.jmh.annotations.Threads; +import org.openjdk.jmh.annotations.Warmup; +import org.openjdk.jmh.infra.Blackhole; +import org.openjdk.jmh.runner.Runner; +import org.openjdk.jmh.runner.options.Options; +import org.openjdk.jmh.runner.options.OptionsBuilder; + +import opennlp.tools.stemmer.snowball.SnowballStemmer; +import opennlp.tools.stemmer.snowball.SnowballStemmerFactory; + +/** + * JMH benchmark for {@link CachingStemmer} against an uncached shared {@link SnowballStemmer}. + * + *Two workloads drive both strategies:
+ *One op stems 16 consecutive tokens from the stream; each benchmark thread walks the stream + * from its own cursor.
+ */ +@BenchmarkMode(Mode.Throughput) +@OutputTimeUnit(TimeUnit.SECONDS) +@Warmup(iterations = 5, time = 2) +@Measurement(iterations = 10, time = 2) +@Fork(2) +public class CachingStemmerBenchmark { + + private static final int STREAM_LENGTH = 65536; + private static final int WORDS_PER_OP = 16; + + private static final String[] ROOTS = { + "run", "walk", "talk", "develop", "nation", "connect", "form", "create", + "act", "direct", "govern", "manage", "operate", "organize", "present", "relate", + "report", "state", "structure", "test", "train", "transform", "translate", "value", + "view", "wonder", "yield", "zone", "note", "mark", "place", "point" + }; + private static final String[] PREFIXES = { + "", "re", "un", "over", "under", "out", "pre", "post", + "non", "anti", "de", "dis", "mis", "sub", "super", "inter" + }; + private static final String[] SUFFIXES = { + "", "s", "ed", "ing", "er", "ers", "ation", "ations", + "ly", "ness", "ment", "ments", "ize", "ized", "izing", "al" + }; + + @State(Scope.Benchmark) + public static class WorkloadState { + + @Param({"zipf", "diverse"}) + String workload; + + String[] stream; + + @Setup(Level.Trial) + public void build() { + Random random = new Random(42); + ListThree strategies are measured, all at {@link Threads#MAX}:
+ *Each thread gets its own delegate stemmer (minted from the supplied {@link StemmerFactory}) + * and its own cache, so instances are safe to share regardless of whether the factory's stemmers + * are, and memory is bounded to {@code capacity} entries per thread that stems. Caching pays off + * on threads reused across many words, such as a fixed platform-thread pool; on a + * virtual-thread-per-task executor every task starts with an empty cache, so repeats are served + * only within one task. {@link #stemAll(CharSequence)} is forwarded to the delegate uncached.
+ * + *Long-running environments such as application containers should call + * {@link #clearThreadLocalState()} when a pooled thread no longer uses this stemmer.
+ */ +@ThreadSafe +public final class CachingStemmer extends DelegatingStemmerthe per-thread payload: a bare delegate stemmer, or a delegate paired with per-thread + * working state such as a cache. + */ +abstract class DelegatingStemmer
implements Stemmer { + + /** The per-thread payload holder. */ + final OwnerOrPerThreadState
state; + + /** + * @param init Mints the per-thread payload on first use by the owner and by each extra thread. + * @param cleanup Releases a per-thread payload when its thread clears state. + */ + DelegatingStemmer(Supplier
init, Consumer
cleanup) { + this.state = new OwnerOrPerThreadState<>(init, cleanup); + } + + /** + * Validates a factory argument before it is captured by a subclass supplier, so a {@code null} + * factory fails at construction rather than on first use. + * + * @param factory The factory to validate. Must not be {@code null}. + * @return {@code factory}. + * @throws IllegalArgumentException if {@code factory} is {@code null}. + */ + static StemmerFactory requireFactory(StemmerFactory factory) { + if (factory == null) { + throw new IllegalArgumentException("factory must not be null"); + } + return factory; + } + + /** + * Removes this thread's payload to prevent classloader leaks in container environments. Call + * when the thread is returned to a pool or the stemmer is no longer needed, mirroring + * {@code clearThreadLocalState()} on the thread-safe {@code *ME} components. + */ + public void clearThreadLocalState() { + state.clearForCurrentThread(); + } +} diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/PorterStemmer.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/PorterStemmer.java index 623a2d3d2c..aac1c49c38 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/PorterStemmer.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/PorterStemmer.java @@ -595,10 +595,16 @@ public String stem(String s) { } /** - * Stem a word provided as a CharSequence. - * Returns the result as a CharSequence. + * Stem a word provided as a CharSequence. Returns the result as a CharSequence. + * + * @param word The word to stem. Must not be {@code null}. + * @return the stemmed word. + * @throws IllegalArgumentException if {@code word} is {@code null}. */ public CharSequence stem(CharSequence word) { + if (word == null) { + throw new IllegalArgumentException("word must not be null"); + } return stem(word.toString()); } diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/PorterStemmerFactory.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/PorterStemmerFactory.java new file mode 100644 index 0000000000..992772778b --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/PorterStemmerFactory.java @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer; + +import opennlp.tools.commons.ThreadSafe; + +/** + * A thread-safe factory for {@link PorterStemmer} instances. + */ +@ThreadSafe +public record PorterStemmerFactory() implements StemmerFactory { + + @Override + public Stemmer newStemmer() { + return new PorterStemmer(); + } +} diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/SharingStemmer.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/SharingStemmer.java new file mode 100644 index 0000000000..abcd21a1d9 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/stemmer/SharingStemmer.java @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer; + +import java.util.List; + +import opennlp.tools.commons.ThreadSafe; +import opennlp.tools.util.OwnerOrPerThreadState; + +/** + * A {@link Stemmer} that is safe to share across threads by routing each call to a per-thread + * delegate minted from a {@link StemmerFactory}. Use it to share a stemmer whose implementation is + * not itself thread-safe, such as {@link PorterStemmer}; thread-safe implementations like + * {@code SnowballStemmer} do not need this wrapper. + * + *
The first thread uses a single owner delegate without touching {@link ThreadLocal} until a + * second thread appears, matching the {@link OwnerOrPerThreadState} pattern used by the + * thread-safe {@code *ME} components.
+ */ +@ThreadSafe +public final class SharingStemmer extends DelegatingStemmerThe generated engines hold mutable per-call buffers, so this class routes each call to a + * per-thread engine, the same pattern the thread-safe {@code *ME} components use. A single + * {@code SnowballStemmer} instance is therefore safe to share across threads.
+ * + *As with the {@code *ME} components, long-running environments such as application + * containers should call {@link #clearThreadLocalState()} when a pooled thread no longer uses + * this stemmer; otherwise the thread retains its engine until the instance is unreachable, + * which can pin the defining classloader on redeploys.
+ */ +@ThreadSafe public class SnowballStemmer implements Stemmer { - private final AbstractSnowballStemmer stemmer; - private final int repeat; + private final SharingStemmer sharing; + /** + * Creates a stemmer for the given algorithm and repeat count. + * + * @param algorithm The Snowball algorithm. Must not be {@code null}. + * @param repeat How many times to apply the stemmer per word; must be positive. + * @throws IllegalArgumentException Thrown if {@code algorithm} is {@code null} or + * {@code repeat} is not positive. + */ public SnowballStemmer(ALGORITHM algorithm, int repeat) { - this.repeat = repeat; - - if (ALGORITHM.ARABIC.equals(algorithm)) { - stemmer = new arabicStemmer(); - } else if (ALGORITHM.DANISH.equals(algorithm)) { - stemmer = new danishStemmer(); - } else if (ALGORITHM.DUTCH.equals(algorithm)) { - stemmer = new dutchStemmer(); - } else if (ALGORITHM.CATALAN.equals(algorithm)) { - stemmer = new catalanStemmer(); - } else if (ALGORITHM.ENGLISH.equals(algorithm)) { - stemmer = new englishStemmer(); - } else if (ALGORITHM.FINNISH.equals(algorithm)) { - stemmer = new finnishStemmer(); - } else if (ALGORITHM.FRENCH.equals(algorithm)) { - stemmer = new frenchStemmer(); - } else if (ALGORITHM.GERMAN.equals(algorithm)) { - stemmer = new germanStemmer(); - } else if (ALGORITHM.GREEK.equals(algorithm)) { - stemmer = new greekStemmer(); - } else if (ALGORITHM.HUNGARIAN.equals(algorithm)) { - stemmer = new hungarianStemmer(); - } else if (ALGORITHM.INDONESIAN.equals(algorithm)) { - stemmer = new indonesianStemmer(); - } else if (ALGORITHM.IRISH.equals(algorithm)) { - stemmer = new irishStemmer(); - } else if (ALGORITHM.ITALIAN.equals(algorithm)) { - stemmer = new italianStemmer(); - } else if (ALGORITHM.NORWEGIAN.equals(algorithm)) { - stemmer = new norwegianStemmer(); - } else if (ALGORITHM.PORTER.equals(algorithm)) { - stemmer = new porterStemmer(); - } else if (ALGORITHM.PORTUGUESE.equals(algorithm)) { - stemmer = new portugueseStemmer(); - } else if (ALGORITHM.ROMANIAN.equals(algorithm)) { - stemmer = new romanianStemmer(); - } else if (ALGORITHM.RUSSIAN.equals(algorithm)) { - stemmer = new russianStemmer(); - } else if (ALGORITHM.SPANISH.equals(algorithm)) { - stemmer = new spanishStemmer(); - } else if (ALGORITHM.SWEDISH.equals(algorithm)) { - stemmer = new swedishStemmer(); - } else if (ALGORITHM.TURKISH.equals(algorithm)) { - stemmer = new turkishStemmer(); - } else { - throw new IllegalStateException("Unexpected stemmer algorithm: " + algorithm); - } + this.sharing = new SharingStemmer(new SnowballStemmerFactory(algorithm, repeat)); } + /** + * Creates a stemmer for the given algorithm with {@code repeat = 1}. + * + * @param algorithm The Snowball algorithm. Must not be {@code null}. + */ public SnowballStemmer(ALGORITHM algorithm) { this(algorithm, 1); } - public CharSequence stem(CharSequence word) { - - stemmer.setCurrent(word.toString()); + /** + * Returns a supplier of a fresh Snowball engine for the given algorithm, shared with + * {@link SnowballStemmerFactory} whose products embed one engine directly. + * + * @param algorithm The Snowball algorithm. + * @return a supplier that mints a new engine on each call. + */ + static SupplierUse this factory for the classic one-stemmer-per-thread pattern: {@link #newStemmer()} + * returns a plain, thread-confined stemmer without the per-call thread routing of the + * shareable {@link SnowballStemmer}.
+ * + * @param algorithm The Snowball algorithm. Must not be {@code null}. + * @param repeat How many times to apply the stemmer per word; must be positive. + */ +@ThreadSafe +public record SnowballStemmerFactory(SnowballStemmer.ALGORITHM algorithm, int repeat) + implements StemmerFactory { + + /** + * Creates a factory with {@code repeat = 1}. + * + * @param algorithm The Snowball algorithm. Must not be {@code null}. + */ + public SnowballStemmerFactory(SnowballStemmer.ALGORITHM algorithm) { + this(algorithm, 1); + } + + /** + * Validates the components. + * + * @throws IllegalArgumentException Thrown if {@code algorithm} is {@code null} or + * {@code repeat} is not positive. + */ + public SnowballStemmerFactory { + if (algorithm == null) { + throw new IllegalArgumentException("algorithm must not be null"); + } + if (repeat <= 0) { + throw new IllegalArgumentException("repeat must be positive, got " + repeat); + } + } + + @Override + public Stemmer newStemmer() { + return new ConfinedStemmer(SnowballStemmer.engineFor(algorithm).get(), repeat); + } + + /** A plain, thread-confined stemmer that applies one Snowball engine directly; not thread-safe. */ + private static final class ConfinedStemmer implements Stemmer { + + private final AbstractSnowballStemmer engine; + private final int repeat; + + /** + * Creates a thread-confined stemmer over the given engine. + * + * @param engine The Snowball engine to apply. + * @param repeat How many times to apply the engine per word. + */ + private ConfinedStemmer(AbstractSnowballStemmer engine, int repeat) { + this.engine = engine; + this.repeat = repeat; + } + + /** + * {@inheritDoc} + * + * @throws IllegalArgumentException if {@code word} is {@code null}. + */ + @Override + public CharSequence stem(CharSequence word) { + if (word == null) { + throw new IllegalArgumentException("word must not be null"); + } + engine.setCurrent(word.toString()); + for (int i = 0; i < repeat; i++) { + engine.stem(); + } + return engine.getCurrent(); + } + } +} diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfile.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfile.java index e7a47477ed..57f4ea27dc 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfile.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/NormalizationProfile.java @@ -4,7 +4,7 @@ * this work for additional information regarding copyright ownership. * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at + * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * @@ -17,7 +17,9 @@ package opennlp.tools.util.normalizer; import opennlp.tools.stemmer.Stemmer; +import opennlp.tools.stemmer.StemmerFactory; import opennlp.tools.stemmer.snowball.SnowballStemmer; +import opennlp.tools.stemmer.snowball.SnowballStemmerFactory; /** * Per-language normalization settings, mirroring how OpenNLP already selects a Snowball stemmer by @@ -61,8 +63,17 @@ public record NormalizationProfile(String language, SnowballStemmer.ALGORITHM st } /** - * {@return a new {@link Stemmer} for this language} A fresh instance is returned on each call - * because the Snowball stemmers are stateful and not thread-safe. + * {@return a thread-safe factory for this language's Snowball stemmer} The factory may be + * shared; each {@linkplain StemmerFactory#newStemmer() minted stemmer} is confined to the + * calling thread. + */ + public StemmerFactory stemmerFactory() { + return new SnowballStemmerFactory(stemmerAlgorithm); + } + + /** + * {@return a new {@link Stemmer} for this language} The returned {@link SnowballStemmer} is + * thread-safe and may be shared across threads. */ public Stemmer newStemmer() { return new SnowballStemmer(stemmerAlgorithm); @@ -70,8 +81,10 @@ public Stemmer newStemmer() { /** * Returns a matching analyzer for this language: NFC, case folding, the language's - * {@linkplain #accentFold() diacritic fold} when it has one, then stemming. Each call builds an - * independent analyzer with its own stemmer, so use one per thread when stemming. + * {@linkplain #accentFold() diacritic fold} when it has one, then stemming. The returned + * analyzer is thread-safe, and repeated words resolve from a bounded per-thread stem cache + * instead of being re-stemmed. The cache is keyed to the thread, so build the analyzer once + * and reuse it from threads that live across many calls, such as a fixed platform-thread pool. * * @return the analyzer. */ @@ -80,6 +93,6 @@ public TermAnalyzer matchingAnalyzer() { if (accentFold != null) { builder.transform(Dimension.ACCENT_FOLD, accentFold); } - return builder.stem(newStemmer()).build(); + return builder.stem(stemmerFactory()).build(); } } diff --git a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/TermAnalyzer.java b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/TermAnalyzer.java index 39f213aaad..934002d7eb 100644 --- a/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/TermAnalyzer.java +++ b/opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/util/normalizer/TermAnalyzer.java @@ -25,7 +25,9 @@ import java.util.Set; import opennlp.tools.lemmatizer.Lemmatizer; +import opennlp.tools.stemmer.CachingStemmer; import opennlp.tools.stemmer.Stemmer; +import opennlp.tools.stemmer.StemmerFactory; import opennlp.tools.tokenize.uax29.WordTokenizer; import opennlp.tools.util.Span; @@ -41,10 +43,12 @@ * {@link Dimension#LEMMA} are enabled by supplying a {@link Stemmer} or {@link Lemmatizer}. * *An instance is immutable and is thread-safe when its configured transforms are. The built-in - * character normalizers are stateless, but the Snowball stemmers are not, so an analyzer configured - * with a {@link Stemmer} (for example through {@code NormalizationProfile.matchingAnalyzer()}) should - * not be shared across threads when {@link Dimension#STEM} is used. Build one with - * {@link #builder()}.
+ * character normalizers are stateless and the Snowball stemmers are thread-safe. When + * {@link Dimension#STEM} is enabled through {@link Builder#stem(StemmerFactory)}, stemming is + * routed through a {@link CachingStemmer}: the analyzer is safe to share across threads even when + * the factory mints stateful stemmers, and repeated words resolve from a bounded per-thread cache + * instead of being re-stemmed. A raw {@link Stemmer} passed to {@link Builder#stem(Stemmer)} is + * only safe when that stemmer is itself thread-safe or the analyzer is confined to one thread. */ public final class TermAnalyzer { @@ -392,6 +396,37 @@ public Builder stem(Stemmer value) { return this; } + /** + * Enables {@link Dimension#STEM} through a {@link StemmerFactory}. The analyzer receives a + * {@link CachingStemmer} with the {@linkplain CachingStemmer#DEFAULT_CAPACITY default cache + * capacity}: it can be shared across threads, and repeated words resolve from a bounded + * per-thread cache instead of being re-stemmed. The cache is keyed to the thread, so the + * memoization pays off on threads reused across many tokens (a fixed platform-thread pool); + * on a virtual-thread-per-task executor every task starts with an empty cache. + * + * @param factory The stemmer factory. Must not be {@code null}. + * @return this builder + * @throws IllegalArgumentException if {@code factory} is {@code null}. + */ + public Builder stem(StemmerFactory factory) { + return stem(factory, CachingStemmer.DEFAULT_CAPACITY); + } + + /** + * Enables {@link Dimension#STEM} through a {@link StemmerFactory} with an explicit stem cache + * capacity, otherwise identical to {@link #stem(StemmerFactory)}. + * + * @param factory The stemmer factory. Must not be {@code null}. + * @param cacheSize The maximum number of word-to-stem entries kept per thread; must be + * positive. + * @return this builder + * @throws IllegalArgumentException if {@code factory} is {@code null} or {@code cacheSize} is + * not positive. + */ + public Builder stem(StemmerFactory factory, int cacheSize) { + return stem(new CachingStemmer(factory, cacheSize)); + } + /** * Enables {@link Dimension#LEMMA} through the given lemmatizer. * diff --git a/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/CachingStemmerTest.java b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/CachingStemmerTest.java new file mode 100644 index 0000000000..6890dbca80 --- /dev/null +++ b/opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/stemmer/CachingStemmerTest.java @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package opennlp.tools.stemmer; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import opennlp.tools.stemmer.snowball.SnowballStemmer; +import opennlp.tools.stemmer.snowball.SnowballStemmerFactory; + +class CachingStemmerTest { + + private static final SnowballStemmerFactory ENGLISH = + new SnowballStemmerFactory(SnowballStemmer.ALGORITHM.ENGLISH); + + @Test + void cachedResultsMatchUncachedResults() { + Stemmer reference = ENGLISH.newStemmer(); + CachingStemmer cached = new CachingStemmer(ENGLISH); + + List