From 80cd183146fbd2c2f72eae69387121e4b7d9d435 Mon Sep 17 00:00:00 2001 From: Nikhil Bagmar Date: Fri, 13 Mar 2026 13:30:18 -0700 Subject: [PATCH 1/2] Add substring expression function documentation Add documentation for four new Data Prepper expression functions: substringAfter, substringBefore, substringAfterLast, and substringBeforeLast. These functions extract portions of a string by delimiter and were added in opensearch-project/data-prepper#6621. Update the functions index page to include the new functions. Resolves: opensearch-project/data-prepper#6612 Signed-off-by: Nikhil Bagmar --- _data-prepper/pipelines/functions.md | 7 +- .../pipelines/substring-after-last.md | 112 ++++++++++++++++++ _data-prepper/pipelines/substring-after.md | 112 ++++++++++++++++++ .../pipelines/substring-before-last.md | 112 ++++++++++++++++++ _data-prepper/pipelines/substring-before.md | 112 ++++++++++++++++++ 5 files changed, 453 insertions(+), 2 deletions(-) create mode 100644 _data-prepper/pipelines/substring-after-last.md create mode 100644 _data-prepper/pipelines/substring-after.md create mode 100644 _data-prepper/pipelines/substring-before-last.md create mode 100644 _data-prepper/pipelines/substring-before.md diff --git a/_data-prepper/pipelines/functions.md b/_data-prepper/pipelines/functions.md index cc308717672..af67888ac52 100644 --- a/_data-prepper/pipelines/functions.md +++ b/_data-prepper/pipelines/functions.md @@ -17,6 +17,9 @@ OpenSearch Data Prepper offers a range of built-in functions that can be used wi - [`hasTags()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/has-tags/) - [`join()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/join/) - [`length()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/length/) -- [`subList()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/sublist/) - [`startsWith()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/startswith/) - +- [`subList()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/sublist/) +- [`substringAfter()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/substring-after/) +- [`substringAfterLast()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/substring-after-last/) +- [`substringBefore()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/substring-before/) +- [`substringBeforeLast()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/substring-before-last/) diff --git a/_data-prepper/pipelines/substring-after-last.md b/_data-prepper/pipelines/substring-after-last.md new file mode 100644 index 00000000000..86e631e6229 --- /dev/null +++ b/_data-prepper/pipelines/substring-after-last.md @@ -0,0 +1,112 @@ +--- +layout: default +title: substringAfterLast() +parent: Functions +grand_parent: Pipelines +nav_order: 55 +--- + +# substringAfterLast() + +The `substringAfterLast()` function is used to extract the portion of a string that follows the last occurrence of a specified delimiter. It takes two arguments: + +- The first argument is either a literal string or a JSON pointer that represents the source string. + +- The second argument is the delimiter string to search for within the first argument. + +If the delimiter is found, the function returns everything after the last occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. + +For example, if you want to extract the file extension from a path field, you can use the `substringAfterLast()` function as follows: + +``` +'substringAfterLast(/filepath, ".")' +``` +{% include copy.html %} + +If `/filepath` contains `archive.tar.gz`, this returns `gz`. + +Alternatively, you can use a literal string as the first argument: + +``` +'substringAfterLast("one-two-three", "-")' +``` +{% include copy.html %} + +This returns `three` because it extracts everything after the last `-`. + +The `substringAfterLast()` function performs a case-sensitive search. +{: .note} + +## Example + +The following pipeline uses the `substringAfterLast()` function to extract the file name from a full file path and adds it as a new field called `filename`: + +```yaml +substring-after-last-demo: + source: + http: + ssl: false + + processor: + - add_entries: + entries: + - key: filename + value_expression: 'substringAfterLast(/filepath, "/")' + + sink: + - opensearch: + hosts: ["https://opensearch:9200"] + insecure: true + username: admin + password: admin_password + index_type: custom + index: demo-index-%{yyyy.MM.dd} +``` +{% include copy.html %} + +You can test the pipeline using the following command: + +```bash +curl -sS -X POST "http://localhost:2021/log/ingest" \ + -H "Content-Type: application/json" \ + -d '[ + {"filepath":"/var/log/syslog"}, + {"filepath":"/home/user/docs/report.pdf"} + ]' +``` +{% include copy.html %} + +The documents stored in OpenSearch contain the following information: + +```json +{ + ... + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "demo-index-2026.03.13", + "_id": "abc123", + "_score": 1, + "_source": { + "filepath": "/var/log/syslog", + "filename": "syslog" + } + }, + { + "_index": "demo-index-2026.03.13", + "_id": "def456", + "_score": 1, + "_source": { + "filepath": "/home/user/docs/report.pdf", + "filename": "report.pdf" + } + } + ] + } +} +``` diff --git a/_data-prepper/pipelines/substring-after.md b/_data-prepper/pipelines/substring-after.md new file mode 100644 index 00000000000..f103abebcc7 --- /dev/null +++ b/_data-prepper/pipelines/substring-after.md @@ -0,0 +1,112 @@ +--- +layout: default +title: substringAfter() +parent: Functions +grand_parent: Pipelines +nav_order: 45 +--- + +# substringAfter() + +The `substringAfter()` function is used to extract the portion of a string that follows the first occurrence of a specified delimiter. It takes two arguments: + +- The first argument is either a literal string or a JSON pointer that represents the source string. + +- The second argument is the delimiter string to search for within the first argument. + +If the delimiter is found, the function returns everything after the first occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. + +For example, if you want to extract the value after the first `=` in a field named `header`, you can use the `substringAfter()` function as follows: + +``` +'substringAfter(/header, "=")' +``` +{% include copy.html %} + +If `/header` contains `Content-Type=application/json`, this returns `application/json`. + +Alternatively, you can use a literal string as the first argument: + +``` +'substringAfter("hello-world-foo", "-")' +``` +{% include copy.html %} + +This returns `world-foo` because it extracts everything after the first `-`. + +The `substringAfter()` function performs a case-sensitive search. +{: .note} + +## Example + +The following pipeline uses the `substringAfter()` function to extract the domain from an email address field and adds it as a new field called `domain`: + +```yaml +substring-after-demo: + source: + http: + ssl: false + + processor: + - add_entries: + entries: + - key: domain + value_expression: 'substringAfter(/email, "@")' + + sink: + - opensearch: + hosts: ["https://opensearch:9200"] + insecure: true + username: admin + password: admin_password + index_type: custom + index: demo-index-%{yyyy.MM.dd} +``` +{% include copy.html %} + +You can test the pipeline using the following command: + +```bash +curl -sS -X POST "http://localhost:2021/log/ingest" \ + -H "Content-Type: application/json" \ + -d '[ + {"email":"user@example.com"}, + {"email":"admin@opensearch.org"} + ]' +``` +{% include copy.html %} + +The documents stored in OpenSearch contain the following information: + +```json +{ + ... + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "demo-index-2026.03.13", + "_id": "abc123", + "_score": 1, + "_source": { + "email": "user@example.com", + "domain": "example.com" + } + }, + { + "_index": "demo-index-2026.03.13", + "_id": "def456", + "_score": 1, + "_source": { + "email": "admin@opensearch.org", + "domain": "opensearch.org" + } + } + ] + } +} +``` diff --git a/_data-prepper/pipelines/substring-before-last.md b/_data-prepper/pipelines/substring-before-last.md new file mode 100644 index 00000000000..236268e1895 --- /dev/null +++ b/_data-prepper/pipelines/substring-before-last.md @@ -0,0 +1,112 @@ +--- +layout: default +title: substringBeforeLast() +parent: Functions +grand_parent: Pipelines +nav_order: 60 +--- + +# substringBeforeLast() + +The `substringBeforeLast()` function is used to extract the portion of a string that precedes the last occurrence of a specified delimiter. It takes two arguments: + +- The first argument is either a literal string or a JSON pointer that represents the source string. + +- The second argument is the delimiter string to search for within the first argument. + +If the delimiter is found, the function returns everything before the last occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. + +For example, if you want to strip the file extension from a filename field, you can use the `substringBeforeLast()` function as follows: + +``` +'substringBeforeLast(/filename, ".")' +``` +{% include copy.html %} + +If `/filename` contains `archive.tar.gz`, this returns `archive.tar`. + +Alternatively, you can use a literal string as the first argument: + +``` +'substringBeforeLast("one-two-three", "-")' +``` +{% include copy.html %} + +This returns `one-two` because it extracts everything before the last `-`. + +The `substringBeforeLast()` function performs a case-sensitive search. +{: .note} + +## Example + +The following pipeline uses the `substringBeforeLast()` function to extract the directory path from a full file path and adds it as a new field called `directory`: + +```yaml +substring-before-last-demo: + source: + http: + ssl: false + + processor: + - add_entries: + entries: + - key: directory + value_expression: 'substringBeforeLast(/filepath, "/")' + + sink: + - opensearch: + hosts: ["https://opensearch:9200"] + insecure: true + username: admin + password: admin_password + index_type: custom + index: demo-index-%{yyyy.MM.dd} +``` +{% include copy.html %} + +You can test the pipeline using the following command: + +```bash +curl -sS -X POST "http://localhost:2021/log/ingest" \ + -H "Content-Type: application/json" \ + -d '[ + {"filepath":"/var/log/syslog"}, + {"filepath":"/home/user/docs/report.pdf"} + ]' +``` +{% include copy.html %} + +The documents stored in OpenSearch contain the following information: + +```json +{ + ... + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "demo-index-2026.03.13", + "_id": "abc123", + "_score": 1, + "_source": { + "filepath": "/var/log/syslog", + "directory": "/var/log" + } + }, + { + "_index": "demo-index-2026.03.13", + "_id": "def456", + "_score": 1, + "_source": { + "filepath": "/home/user/docs/report.pdf", + "directory": "/home/user/docs" + } + } + ] + } +} +``` diff --git a/_data-prepper/pipelines/substring-before.md b/_data-prepper/pipelines/substring-before.md new file mode 100644 index 00000000000..c9690ada769 --- /dev/null +++ b/_data-prepper/pipelines/substring-before.md @@ -0,0 +1,112 @@ +--- +layout: default +title: substringBefore() +parent: Functions +grand_parent: Pipelines +nav_order: 50 +--- + +# substringBefore() + +The `substringBefore()` function is used to extract the portion of a string that precedes the first occurrence of a specified delimiter. It takes two arguments: + +- The first argument is either a literal string or a JSON pointer that represents the source string. + +- The second argument is the delimiter string to search for within the first argument. + +If the delimiter is found, the function returns everything before the first occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. + +For example, if you want to extract the username from an email address field, you can use the `substringBefore()` function as follows: + +``` +'substringBefore(/email, "@")' +``` +{% include copy.html %} + +If `/email` contains `user@example.com`, this returns `user`. + +Alternatively, you can use a literal string as the first argument: + +``` +'substringBefore("hello-world-foo", "-")' +``` +{% include copy.html %} + +This returns `hello` because it extracts everything before the first `-`. + +The `substringBefore()` function performs a case-sensitive search. +{: .note} + +## Example + +The following pipeline uses the `substringBefore()` function to extract the protocol from a URL field and adds it as a new field called `protocol`: + +```yaml +substring-before-demo: + source: + http: + ssl: false + + processor: + - add_entries: + entries: + - key: protocol + value_expression: 'substringBefore(/url, "://")' + + sink: + - opensearch: + hosts: ["https://opensearch:9200"] + insecure: true + username: admin + password: admin_password + index_type: custom + index: demo-index-%{yyyy.MM.dd} +``` +{% include copy.html %} + +You can test the pipeline using the following command: + +```bash +curl -sS -X POST "http://localhost:2021/log/ingest" \ + -H "Content-Type: application/json" \ + -d '[ + {"url":"https://opensearch.org/docs"}, + {"url":"http://example.com/page"} + ]' +``` +{% include copy.html %} + +The documents stored in OpenSearch contain the following information: + +```json +{ + ... + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "demo-index-2026.03.13", + "_id": "abc123", + "_score": 1, + "_source": { + "url": "https://opensearch.org/docs", + "protocol": "https" + } + }, + { + "_index": "demo-index-2026.03.13", + "_id": "def456", + "_score": 1, + "_source": { + "url": "http://example.com/page", + "protocol": "http" + } + } + ] + } +} +``` From 810ace2e9e836e233a1561d21e8a30e868da38f0 Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Mon, 16 Mar 2026 13:45:51 -0400 Subject: [PATCH 2/2] Doc review Signed-off-by: Fanit Kolchina --- _data-prepper/pipelines/sublist.md | 2 +- _data-prepper/pipelines/substring-after-last.md | 16 ++++++++-------- _data-prepper/pipelines/substring-after.md | 16 ++++++++-------- _data-prepper/pipelines/substring-before-last.md | 16 ++++++++-------- _data-prepper/pipelines/substring-before.md | 16 ++++++++-------- 5 files changed, 33 insertions(+), 33 deletions(-) diff --git a/_data-prepper/pipelines/sublist.md b/_data-prepper/pipelines/sublist.md index 158fe17a21c..1071f21d654 100644 --- a/_data-prepper/pipelines/sublist.md +++ b/_data-prepper/pipelines/sublist.md @@ -3,7 +3,7 @@ layout: default title: subList() parent: Functions grand_parent: Pipelines -nav_order: 35 +nav_order: 50 --- # subList(, , ) diff --git a/_data-prepper/pipelines/substring-after-last.md b/_data-prepper/pipelines/substring-after-last.md index 86e631e6229..7eb2757d774 100644 --- a/_data-prepper/pipelines/substring-after-last.md +++ b/_data-prepper/pipelines/substring-after-last.md @@ -3,27 +3,27 @@ layout: default title: substringAfterLast() parent: Functions grand_parent: Pipelines -nav_order: 55 +nav_order: 70 --- # substringAfterLast() The `substringAfterLast()` function is used to extract the portion of a string that follows the last occurrence of a specified delimiter. It takes two arguments: -- The first argument is either a literal string or a JSON pointer that represents the source string. +1. The first argument is either a literal string or a JSON pointer that represents the source string. -- The second argument is the delimiter string to search for within the first argument. +1. The second argument is the delimiter string to search for within the first argument. -If the delimiter is found, the function returns everything after the last occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. +If the delimiter is found, the function returns the portion of the string after the last occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. -For example, if you want to extract the file extension from a path field, you can use the `substringAfterLast()` function as follows: +For example, to extract the file extension from the `/filepath` field containing a file path, use the `substringAfterLast()` function as follows: ``` 'substringAfterLast(/filepath, ".")' ``` {% include copy.html %} -If `/filepath` contains `archive.tar.gz`, this returns `gz`. +If the `/filepath` field contains `archive.tar.gz`, the function returns `gz`. Alternatively, you can use a literal string as the first argument: @@ -32,14 +32,14 @@ Alternatively, you can use a literal string as the first argument: ``` {% include copy.html %} -This returns `three` because it extracts everything after the last `-`. +The function returns `three` because it extracts the portion of the string after the last `-` character. The `substringAfterLast()` function performs a case-sensitive search. {: .note} ## Example -The following pipeline uses the `substringAfterLast()` function to extract the file name from a full file path and adds it as a new field called `filename`: +The following pipeline uses the `substringAfterLast()` function to extract the file name from a full file path. It adds the extracted file name as a new field called `filename`: ```yaml substring-after-last-demo: diff --git a/_data-prepper/pipelines/substring-after.md b/_data-prepper/pipelines/substring-after.md index f103abebcc7..13768e12280 100644 --- a/_data-prepper/pipelines/substring-after.md +++ b/_data-prepper/pipelines/substring-after.md @@ -3,27 +3,27 @@ layout: default title: substringAfter() parent: Functions grand_parent: Pipelines -nav_order: 45 +nav_order: 60 --- # substringAfter() The `substringAfter()` function is used to extract the portion of a string that follows the first occurrence of a specified delimiter. It takes two arguments: -- The first argument is either a literal string or a JSON pointer that represents the source string. +1. The first argument is either a literal string or a JSON pointer that represents the source string. -- The second argument is the delimiter string to search for within the first argument. +1. The second argument is the delimiter string to search for within the first argument. -If the delimiter is found, the function returns everything after the first occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. +If the delimiter is found, the function returns the portion of the string after the first occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. -For example, if you want to extract the value after the first `=` in a field named `header`, you can use the `substringAfter()` function as follows: +For example, to extract the value after the first occurrence of the `=` character in a field named `header`, use the `substringAfter()` function as follows: ``` 'substringAfter(/header, "=")' ``` {% include copy.html %} -If `/header` contains `Content-Type=application/json`, this returns `application/json`. +If `/header` contains `Content-Type=application/json`, the function returns `application/json`. Alternatively, you can use a literal string as the first argument: @@ -32,14 +32,14 @@ Alternatively, you can use a literal string as the first argument: ``` {% include copy.html %} -This returns `world-foo` because it extracts everything after the first `-`. +The function returns `world-foo` because it extracts the portion of the string after the first `-` character. The `substringAfter()` function performs a case-sensitive search. {: .note} ## Example -The following pipeline uses the `substringAfter()` function to extract the domain from an email address field and adds it as a new field called `domain`: +The following pipeline uses the `substringAfter()` function to extract the domain name from an email address field. It adds the extracted domain name as a new field called `domain`: ```yaml substring-after-demo: diff --git a/_data-prepper/pipelines/substring-before-last.md b/_data-prepper/pipelines/substring-before-last.md index 236268e1895..a5ed2805c83 100644 --- a/_data-prepper/pipelines/substring-before-last.md +++ b/_data-prepper/pipelines/substring-before-last.md @@ -3,27 +3,27 @@ layout: default title: substringBeforeLast() parent: Functions grand_parent: Pipelines -nav_order: 60 +nav_order: 90 --- # substringBeforeLast() The `substringBeforeLast()` function is used to extract the portion of a string that precedes the last occurrence of a specified delimiter. It takes two arguments: -- The first argument is either a literal string or a JSON pointer that represents the source string. +1. The first argument is either a literal string or a JSON pointer that represents the source string. -- The second argument is the delimiter string to search for within the first argument. +1. The second argument is the delimiter string to search for within the first argument. -If the delimiter is found, the function returns everything before the last occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. +If the delimiter is found, the function returns the portion of the string before the last occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. -For example, if you want to strip the file extension from a filename field, you can use the `substringBeforeLast()` function as follows: +For example, to remove the file extension from a filename field, use the `substringBeforeLast()` function as follows: ``` 'substringBeforeLast(/filename, ".")' ``` {% include copy.html %} -If `/filename` contains `archive.tar.gz`, this returns `archive.tar`. +If the `/filename` field contains `archive.tar.gz`, the function returns `archive.tar`. Alternatively, you can use a literal string as the first argument: @@ -32,14 +32,14 @@ Alternatively, you can use a literal string as the first argument: ``` {% include copy.html %} -This returns `one-two` because it extracts everything before the last `-`. +The function returns `one-two` because it extracts the portion of the string before the last `-` character. The `substringBeforeLast()` function performs a case-sensitive search. {: .note} ## Example -The following pipeline uses the `substringBeforeLast()` function to extract the directory path from a full file path and adds it as a new field called `directory`: +The following pipeline uses the `substringBeforeLast()` function to extract the directory path from a full file path. It adds the extracted directory path as a new field called `directory`: ```yaml substring-before-last-demo: diff --git a/_data-prepper/pipelines/substring-before.md b/_data-prepper/pipelines/substring-before.md index c9690ada769..47796d6678c 100644 --- a/_data-prepper/pipelines/substring-before.md +++ b/_data-prepper/pipelines/substring-before.md @@ -3,27 +3,27 @@ layout: default title: substringBefore() parent: Functions grand_parent: Pipelines -nav_order: 50 +nav_order: 80 --- # substringBefore() The `substringBefore()` function is used to extract the portion of a string that precedes the first occurrence of a specified delimiter. It takes two arguments: -- The first argument is either a literal string or a JSON pointer that represents the source string. +1. The first argument is either a literal string or a JSON pointer that represents the source string. -- The second argument is the delimiter string to search for within the first argument. +1. The second argument is the delimiter string to search for within the first argument. -If the delimiter is found, the function returns everything before the first occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. +If the delimiter is found, the function returns the portion of the string before the first occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. -For example, if you want to extract the username from an email address field, you can use the `substringBefore()` function as follows: +For example, to extract the username from an email address field, use the `substringBefore()` function as follows: ``` 'substringBefore(/email, "@")' ``` {% include copy.html %} -If `/email` contains `user@example.com`, this returns `user`. +If the `/email` field contains `user@example.com`, the function returns `user`. Alternatively, you can use a literal string as the first argument: @@ -32,14 +32,14 @@ Alternatively, you can use a literal string as the first argument: ``` {% include copy.html %} -This returns `hello` because it extracts everything before the first `-`. +The function returns `hello` because it extracts the portion of the string before the first `-` character. The `substringBefore()` function performs a case-sensitive search. {: .note} ## Example -The following pipeline uses the `substringBefore()` function to extract the protocol from a URL field and adds it as a new field called `protocol`: +The following pipeline uses the `substringBefore()` function to extract the URL protocol from a URL field. It adds the extracted protocol as a new field called `protocol`: ```yaml substring-before-demo: