From e5c7af9f997f3cc8a5aef045bc7828e741e29582 Mon Sep 17 00:00:00 2001 From: shayush622 Date: Fri, 29 May 2026 10:47:11 +0530 Subject: [PATCH 1/4] Document Hunspell hot-reload and ref_path refactor for 3.7 Update the Hunspell token filter docs to reflect the hot-reload support and ref_path path-resolution change landing in OpenSearch 3.7 (opensearch-project/OpenSearch#21516, #21559 and opensearch-project/index-management#1638): - Add the new `updateable` filter parameter. - Update `ref_path` to resolve under `` directly (was `/analyzers/`) and allow forward slashes for nested paths; add an upgrade note for users coming from 3.6. - Rename "Package-based dictionary loading" to "Custom dictionary loading with ref_path" and refresh examples to use the new layout. - Add a "Hot-reloading Hunspell dictionaries" section covering the `updateable` + `_refresh_search_analyzers?reload_cached_resources=true` workflow. - Document the new `reload_cached_resources` query parameter on the Refresh search analyzer API page. Signed-off-by: shayush622 --- _analyzers/token-filters/hunspell.md | 91 ++++++++++++++++++++++++---- _im-plugin/refresh-analyzer.md | 8 +++ 2 files changed, 88 insertions(+), 11 deletions(-) diff --git a/_analyzers/token-filters/hunspell.md b/_analyzers/token-filters/hunspell.md index 80322836a1e..1b7dfad25af 100644 --- a/_analyzers/token-filters/hunspell.md +++ b/_analyzers/token-filters/hunspell.md @@ -11,7 +11,9 @@ The `hunspell` token filter is used for stemming and morphological analysis of w The Hunspell dictionary files are automatically loaded at startup from the `/hunspell/` directory. For example, the `en_GB` locale must have at least one `.aff` file and one or more `.dic` files in the `/hunspell/en_GB/` directory. -Alternatively, you can configure package-based dictionary loading using the `ref_path` parameter to maintain multiple independent dictionary sets for the same locale. For more information, see [Package-based dictionary loading](#package-based-dictionary-loading). +Alternatively, you can load dictionaries from a custom directory by using the `ref_path` parameter to maintain multiple independent dictionary sets for the same locale. For more information, see [Custom dictionary loading with ref_path](#custom-dictionary-loading-with-ref_path). + +You can also hot-reload Hunspell dictionaries at runtime without restarting the node. For more information, see [Hot-reloading Hunspell dictionaries](#hot-reloading-hunspell-dictionaries). You can download these files from [LibreOffice dictionaries](https://github.com/LibreOffice/dictionaries). @@ -22,10 +24,11 @@ The `hunspell` token filter can be configured with the following parameters. Parameter | Required/Optional | Data type | Description :--- | :--- | :--- | :--- `language/lang/locale` | At least one of the three is required | String | Specifies the language for the Hunspell dictionary. Can contain only alphanumeric characters, hyphens, and underscores (for example, `en_US`, `de_DE`). -`ref_path` | Optional | String | Specifies a package name used to load dictionaries from the `/analyzers//hunspell//` directory instead of the default `/hunspell//` directory. When specified, the `locale` parameter is required. Both `ref_path` and `locale` parameters can contain only alphanumeric characters, hyphens, and underscores. See [Package-based dictionary loading](#package-based-dictionary-loading). +`ref_path` | Optional | String | Specifies a relative path used to load dictionaries from the `//hunspell//` directory instead of the default `/hunspell//` directory. When specified, the `locale` parameter is required. The `ref_path` value can contain alphanumeric characters, hyphens, underscores, and forward slashes (for nested paths such as `analyzers/my-dict`). The `locale` value can contain only alphanumeric characters, hyphens, and underscores. See [Custom dictionary loading with ref_path](#custom-dictionary-loading-with-ref_path). **Note**: Starting in OpenSearch 3.7, `ref_path` resolves directly under ``. In earlier versions (3.6), it resolved under `/analyzers/`. To preserve the previous layout, prefix your `ref_path` value with `analyzers/` (for example, `analyzers/my-dict`). `dedup` | Optional | Boolean | Determines whether to remove multiple duplicate stemming terms for the same token. Default is `true`. -`dictionary` | Optional | Array of strings | Configures the dictionary files to be used for the Hunspell dictionary. Default is all files in the `/hunspell/` directory if `ref_path` is not specified or all files in the `/analyzers//hunspell//` directory when `ref_path` is specified. See [Package-based dictionary loading](#package-based-dictionary-loading). +`dictionary` | Optional | Array of strings | Configures the dictionary files to be used for the Hunspell dictionary. Default is all files in the `/hunspell/` directory if `ref_path` is not specified or all files in the `//hunspell//` directory when `ref_path` is specified. See [Custom dictionary loading with ref_path](#custom-dictionary-loading-with-ref_path). `longest_only` | Optional | Boolean | Specifies whether only the longest stemmed version of the token should be returned. Default is `false`. +`updateable` | Optional | Boolean | When set to `true`, the filter operates in search-time analysis mode, allowing dictionaries to be hot-reloaded by using the [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/refresh-analyzer/) API without restarting the node. Default is `false`. **Introduced 3.7.** ## Example @@ -60,20 +63,23 @@ PUT /my_index ``` {% include copy-curl.html %} -## Package-based dictionary loading +## Custom dictionary loading with ref_path + +When you specify a `ref_path` parameter, dictionaries are loaded from a custom directory instead of the default directory. This is useful when you need multiple independent dictionary sets for the same locale, for example, when different indexes require different custom dictionaries. -When you specify a `ref_path` parameter, dictionaries are loaded from a package-specific directory instead of the default directory. This is useful when you need multiple independent dictionary sets for the same locale, for example, when different indexes require different custom dictionaries. +Starting in OpenSearch 3.7, `ref_path` is resolved relative to `` and can contain forward slashes for nested paths. In OpenSearch 3.6, `ref_path` was resolved relative to `/analyzers/`. To preserve the previous directory layout when upgrading from 3.6, prefix your existing `ref_path` value with `analyzers/`. +{: .note} Place dictionary files in the following directory structure: ```xml -/analyzers//hunspell// +//hunspell// ├── .aff (exactly one .aff file required) ├── .dic (one or more .dic files) └── _custom.dic ``` -The following example loads a Hunspell dictionary from the package directory `/analyzers/pkg-1234/hunspell/en_US/`: +The following example loads a Hunspell dictionary from `/analyzers/my-dict/hunspell/en_US/`: ```json PUT /my_index @@ -83,7 +89,7 @@ PUT /my_index "filter": { "my_custom_hunspell": { "type": "hunspell", - "ref_path": "pkg-1234", + "ref_path": "analyzers/my-dict", "locale": "en_US" } }, @@ -103,7 +109,7 @@ PUT /my_index ``` {% include copy-curl.html %} -Multiple indexes can use different packages configured for the same locale. Each package maintains its own independent dictionary cache: +Multiple indexes can use different `ref_path` directories configured for the same locale. Each `ref_path` maintains its own independent dictionary cache: ```json PUT /index_medical @@ -113,7 +119,7 @@ PUT /index_medical "filter": { "medical_hunspell": { "type": "hunspell", - "ref_path": "medical-dict", + "ref_path": "analyzers/medical-dict", "locale": "en_US" } } @@ -131,7 +137,7 @@ PUT /index_legal "filter": { "legal_hunspell": { "type": "hunspell", - "ref_path": "legal-dict", + "ref_path": "analyzers/legal-dict", "locale": "en_US" } } @@ -141,6 +147,69 @@ PUT /index_legal ``` {% include copy-curl.html %} +## Hot-reloading Hunspell dictionaries + +**Introduced 3.7** +{: .label .label-purple } + +You can update Hunspell dictionaries at runtime without restarting the node by setting the `updateable` parameter to `true` on the filter and then calling the [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/refresh-analyzer/) API with the `reload_cached_resources` query parameter. + +When `updateable` is `true`, the filter is registered in search-time analysis mode, which allows in-place reloading of cached dictionary resources. The filter can then only be used at search time (for example, in a `search_analyzer`), not at index time. + +The hot-reload workflow is as follows: + +1. Replace the `.aff` and `.dic` files on disk on every node that holds a shard for the index. +2. Send a request to the Refresh search analyzer API with `reload_cached_resources=true`: + + ```json + POST /_plugins/_refresh_search_analyzers/?reload_cached_resources=true + ``` + {% include copy-curl.html %} + +When `reload_cached_resources` is `false` (the default), the API rebuilds analyzer factories but reuses the previously cached Hunspell dictionary. Set the parameter to `true` to force the dictionary to be reloaded from disk. + +The following example creates an index that uses a hot-reloadable Hunspell filter as a `search_analyzer`: + +```json +PUT /my_index +{ + "settings": { + "analysis": { + "filter": { + "my_reloadable_hunspell": { + "type": "hunspell", + "ref_path": "analyzers/my-dict", + "locale": "en_US", + "updateable": true + } + }, + "analyzer": { + "my_search_analyzer": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "lowercase", + "my_reloadable_hunspell" + ] + } + } + } + }, + "mappings": { + "properties": { + "content": { + "type": "text", + "analyzer": "standard", + "search_analyzer": "my_search_analyzer" + } + } + } +} +``` +{% include copy-curl.html %} + +For more information about the Refresh search analyzer API, see [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/refresh-analyzer/). + ## Generated tokens Use the following request to examine the tokens generated using the analyzer: diff --git a/_im-plugin/refresh-analyzer.md b/_im-plugin/refresh-analyzer.md index 09fc2adc0cc..4288d819bd1 100644 --- a/_im-plugin/refresh-analyzer.md +++ b/_im-plugin/refresh-analyzer.md @@ -38,3 +38,11 @@ To work, the token filter must have an `updateable` flag of `true`: } } ``` + +## Query parameters + +The following table lists the supported query parameters. + +Parameter | Data type | Description +:--- | :--- | :--- +`reload_cached_resources` | Boolean | When set to `true`, in-place reloads cached resources for token filters that load files from disk (for example, the [`hunspell`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/hunspell/) filter's dictionary files). When `false` (the default), analyzer factories are rebuilt but cached resources are reused. **Introduced 3.7.** From e8456d787402b2c25df203bb8d095f2575216215 Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Wed, 3 Jun 2026 09:39:56 -0400 Subject: [PATCH 2/4] Doc review Signed-off-by: Fanit Kolchina --- _analyzers/token-filters/hunspell.md | 94 +++++++++++++--------------- 1 file changed, 44 insertions(+), 50 deletions(-) diff --git a/_analyzers/token-filters/hunspell.md b/_analyzers/token-filters/hunspell.md index 1b7dfad25af..f25795c08d1 100644 --- a/_analyzers/token-filters/hunspell.md +++ b/_analyzers/token-filters/hunspell.md @@ -63,7 +63,7 @@ PUT /my_index ``` {% include copy-curl.html %} -## Custom dictionary loading with ref_path +## Custom dictionary loading When you specify a `ref_path` parameter, dictionaries are loaded from a custom directory instead of the default directory. This is useful when you need multiple independent dictionary sets for the same locale, for example, when different indexes require different custom dictionaries. @@ -148,67 +148,61 @@ PUT /index_legal {% include copy-curl.html %} ## Hot-reloading Hunspell dictionaries - **Introduced 3.7** {: .label .label-purple } -You can update Hunspell dictionaries at runtime without restarting the node by setting the `updateable` parameter to `true` on the filter and then calling the [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/refresh-analyzer/) API with the `reload_cached_resources` query parameter. - -When `updateable` is `true`, the filter is registered in search-time analysis mode, which allows in-place reloading of cached dictionary resources. The filter can then only be used at search time (for example, in a `search_analyzer`), not at index time. +You can update Hunspell dictionaries at runtime without restarting the node. To enable this, set the `updateable` parameter to `true` on the Hunspell token filter. This registers the filter in search-time analysis mode, so it can only be used at search time (for example, in a `search_analyzer`), not at index time. -The hot-reload workflow is as follows: +To hot-reload a Hunspell dictionary, follow these steps: -1. Replace the `.aff` and `.dic` files on disk on every node that holds a shard for the index. -2. Send a request to the Refresh search analyzer API with `reload_cached_resources=true`: +1. Configure the Hunspell token filter with `updateable` set to `true`. The following example creates an index that uses a hot-reloadable Hunspell filter as a `search_analyzer`: ```json - POST /_plugins/_refresh_search_analyzers/?reload_cached_resources=true + PUT /my_index + { + "settings": { + "analysis": { + "filter": { + "my_reloadable_hunspell": { + "type": "hunspell", + "ref_path": "analyzers/my-dict", + "locale": "en_US", + "updateable": true + } + }, + "analyzer": { + "my_search_analyzer": { + "type": "custom", + "tokenizer": "standard", + "filter": [ + "lowercase", + "my_reloadable_hunspell" + ] + } + } + } + }, + "mappings": { + "properties": { + "content": { + "type": "text", + "analyzer": "standard", + "search_analyzer": "my_search_analyzer" + } + } + } + } ``` {% include copy-curl.html %} -When `reload_cached_resources` is `false` (the default), the API rebuilds analyzer factories but reuses the previously cached Hunspell dictionary. Set the parameter to `true` to force the dictionary to be reloaded from disk. +1. Replace the `.aff` and `.dic` files on disk on every node that holds a shard for the index. -The following example creates an index that uses a hot-reloadable Hunspell filter as a `search_analyzer`: +1. Call the [Refresh Search Analyzer API]({{site.url}}{{site.baseurl}}/im-plugin/refresh-analyzer/). When `reload_cached_resources` is `false` (the default), the API rebuilds analyzer factories but reuses the previously cached Hunspell dictionary. Specify `reload_cached_resources=true` to force the dictionary to be reloaded from disk: -```json -PUT /my_index -{ - "settings": { - "analysis": { - "filter": { - "my_reloadable_hunspell": { - "type": "hunspell", - "ref_path": "analyzers/my-dict", - "locale": "en_US", - "updateable": true - } - }, - "analyzer": { - "my_search_analyzer": { - "type": "custom", - "tokenizer": "standard", - "filter": [ - "lowercase", - "my_reloadable_hunspell" - ] - } - } - } - }, - "mappings": { - "properties": { - "content": { - "type": "text", - "analyzer": "standard", - "search_analyzer": "my_search_analyzer" - } - } - } -} -``` -{% include copy-curl.html %} - -For more information about the Refresh search analyzer API, see [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/refresh-analyzer/). + ```json + POST /_plugins/_refresh_search_analyzers/my_index?reload_cached_resources=true + ``` + {% include copy-curl.html %} ## Generated tokens From 3c32d114b0b8791fd9dd02f0836ee7fcd96c7418 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Wed, 3 Jun 2026 09:41:41 -0400 Subject: [PATCH 3/4] Apply suggestion from @kolchfa-aws Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- _im-plugin/refresh-analyzer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_im-plugin/refresh-analyzer.md b/_im-plugin/refresh-analyzer.md index 4288d819bd1..a1dcff9fe86 100644 --- a/_im-plugin/refresh-analyzer.md +++ b/_im-plugin/refresh-analyzer.md @@ -45,4 +45,4 @@ The following table lists the supported query parameters. Parameter | Data type | Description :--- | :--- | :--- -`reload_cached_resources` | Boolean | When set to `true`, in-place reloads cached resources for token filters that load files from disk (for example, the [`hunspell`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/hunspell/) filter's dictionary files). When `false` (the default), analyzer factories are rebuilt but cached resources are reused. **Introduced 3.7.** +`reload_cached_resources` | Boolean | When set to `true`, reloads cached resources from disk without rebuilding caches for token filters that load files from disk (for example, the [`hunspell`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/hunspell/) filter's dictionary files). When `false` (the default), analyzer factories are rebuilt but cached resources are reused. From b300d122496a59511e670bba754c31ae14ac5176 Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Wed, 3 Jun 2026 09:56:39 -0400 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --- _analyzers/token-filters/hunspell.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_analyzers/token-filters/hunspell.md b/_analyzers/token-filters/hunspell.md index f25795c08d1..0e8395cae8d 100644 --- a/_analyzers/token-filters/hunspell.md +++ b/_analyzers/token-filters/hunspell.md @@ -11,7 +11,7 @@ The `hunspell` token filter is used for stemming and morphological analysis of w The Hunspell dictionary files are automatically loaded at startup from the `/hunspell/` directory. For example, the `en_GB` locale must have at least one `.aff` file and one or more `.dic` files in the `/hunspell/en_GB/` directory. -Alternatively, you can load dictionaries from a custom directory by using the `ref_path` parameter to maintain multiple independent dictionary sets for the same locale. For more information, see [Custom dictionary loading with ref_path](#custom-dictionary-loading-with-ref_path). +Alternatively, you can load dictionaries from a custom directory by using the `ref_path` parameter to maintain multiple independent dictionary sets for the same locale. For more information, see [Custom dictionary loading with ref_path](#custom-dictionary-loading). You can also hot-reload Hunspell dictionaries at runtime without restarting the node. For more information, see [Hot-reloading Hunspell dictionaries](#hot-reloading-hunspell-dictionaries). @@ -24,9 +24,9 @@ The `hunspell` token filter can be configured with the following parameters. Parameter | Required/Optional | Data type | Description :--- | :--- | :--- | :--- `language/lang/locale` | At least one of the three is required | String | Specifies the language for the Hunspell dictionary. Can contain only alphanumeric characters, hyphens, and underscores (for example, `en_US`, `de_DE`). -`ref_path` | Optional | String | Specifies a relative path used to load dictionaries from the `//hunspell//` directory instead of the default `/hunspell//` directory. When specified, the `locale` parameter is required. The `ref_path` value can contain alphanumeric characters, hyphens, underscores, and forward slashes (for nested paths such as `analyzers/my-dict`). The `locale` value can contain only alphanumeric characters, hyphens, and underscores. See [Custom dictionary loading with ref_path](#custom-dictionary-loading-with-ref_path). **Note**: Starting in OpenSearch 3.7, `ref_path` resolves directly under ``. In earlier versions (3.6), it resolved under `/analyzers/`. To preserve the previous layout, prefix your `ref_path` value with `analyzers/` (for example, `analyzers/my-dict`). +`ref_path` | Optional | String | Specifies a relative path used to load dictionaries from the `//hunspell//` directory instead of the default `/hunspell//` directory. When specified, the `locale` parameter is required. The `ref_path` value can contain alphanumeric characters, hyphens, underscores, and forward slashes (for nested paths such as `analyzers/my-dict`). The `locale` value can contain only alphanumeric characters, hyphens, and underscores. See [Custom dictionary loading with ref_path](#custom-dictionary-loading). **Note**: Starting in OpenSearch 3.7, `ref_path` resolves directly under ``. In earlier versions (3.6), it resolved under `/analyzers/`. To preserve the previous layout, prefix your `ref_path` value with `analyzers/` (for example, `analyzers/my-dict`). `dedup` | Optional | Boolean | Determines whether to remove multiple duplicate stemming terms for the same token. Default is `true`. -`dictionary` | Optional | Array of strings | Configures the dictionary files to be used for the Hunspell dictionary. Default is all files in the `/hunspell/` directory if `ref_path` is not specified or all files in the `//hunspell//` directory when `ref_path` is specified. See [Custom dictionary loading with ref_path](#custom-dictionary-loading-with-ref_path). +`dictionary` | Optional | Array of strings | Configures the dictionary files to be used for the Hunspell dictionary. Default is all files in the `/hunspell/` directory if `ref_path` is not specified or all files in the `//hunspell//` directory when `ref_path` is specified. See [Custom dictionary loading with ref_path](#custom-dictionary-loading). `longest_only` | Optional | Boolean | Specifies whether only the longest stemmed version of the token should be returned. Default is `false`. `updateable` | Optional | Boolean | When set to `true`, the filter operates in search-time analysis mode, allowing dictionaries to be hot-reloaded by using the [Refresh search analyzer]({{site.url}}{{site.baseurl}}/im-plugin/refresh-analyzer/) API without restarting the node. Default is `false`. **Introduced 3.7.**