diff --git a/_data-prepper/pipelines/functions.md b/_data-prepper/pipelines/functions.md index cc308717672..af67888ac52 100644 --- a/_data-prepper/pipelines/functions.md +++ b/_data-prepper/pipelines/functions.md @@ -17,6 +17,9 @@ OpenSearch Data Prepper offers a range of built-in functions that can be used wi - [`hasTags()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/has-tags/) - [`join()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/join/) - [`length()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/length/) -- [`subList()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/sublist/) - [`startsWith()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/startswith/) - +- [`subList()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/sublist/) +- [`substringAfter()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/substring-after/) +- [`substringAfterLast()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/substring-after-last/) +- [`substringBefore()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/substring-before/) +- [`substringBeforeLast()`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/substring-before-last/) diff --git a/_data-prepper/pipelines/sublist.md b/_data-prepper/pipelines/sublist.md index 158fe17a21c..1071f21d654 100644 --- a/_data-prepper/pipelines/sublist.md +++ b/_data-prepper/pipelines/sublist.md @@ -3,7 +3,7 @@ layout: default title: subList() parent: Functions grand_parent: Pipelines -nav_order: 35 +nav_order: 50 --- # subList(, , ) diff --git a/_data-prepper/pipelines/substring-after-last.md b/_data-prepper/pipelines/substring-after-last.md new file mode 100644 index 00000000000..7eb2757d774 --- /dev/null +++ b/_data-prepper/pipelines/substring-after-last.md @@ -0,0 +1,112 @@ +--- +layout: default +title: substringAfterLast() +parent: Functions +grand_parent: Pipelines +nav_order: 70 +--- + +# substringAfterLast() + +The `substringAfterLast()` function is used to extract the portion of a string that follows the last occurrence of a specified delimiter. It takes two arguments: + +1. The first argument is either a literal string or a JSON pointer that represents the source string. + +1. The second argument is the delimiter string to search for within the first argument. + +If the delimiter is found, the function returns the portion of the string after the last occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. + +For example, to extract the file extension from the `/filepath` field containing a file path, use the `substringAfterLast()` function as follows: + +``` +'substringAfterLast(/filepath, ".")' +``` +{% include copy.html %} + +If the `/filepath` field contains `archive.tar.gz`, the function returns `gz`. + +Alternatively, you can use a literal string as the first argument: + +``` +'substringAfterLast("one-two-three", "-")' +``` +{% include copy.html %} + +The function returns `three` because it extracts the portion of the string after the last `-` character. + +The `substringAfterLast()` function performs a case-sensitive search. +{: .note} + +## Example + +The following pipeline uses the `substringAfterLast()` function to extract the file name from a full file path. It adds the extracted file name as a new field called `filename`: + +```yaml +substring-after-last-demo: + source: + http: + ssl: false + + processor: + - add_entries: + entries: + - key: filename + value_expression: 'substringAfterLast(/filepath, "/")' + + sink: + - opensearch: + hosts: ["https://opensearch:9200"] + insecure: true + username: admin + password: admin_password + index_type: custom + index: demo-index-%{yyyy.MM.dd} +``` +{% include copy.html %} + +You can test the pipeline using the following command: + +```bash +curl -sS -X POST "http://localhost:2021/log/ingest" \ + -H "Content-Type: application/json" \ + -d '[ + {"filepath":"/var/log/syslog"}, + {"filepath":"/home/user/docs/report.pdf"} + ]' +``` +{% include copy.html %} + +The documents stored in OpenSearch contain the following information: + +```json +{ + ... + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "demo-index-2026.03.13", + "_id": "abc123", + "_score": 1, + "_source": { + "filepath": "/var/log/syslog", + "filename": "syslog" + } + }, + { + "_index": "demo-index-2026.03.13", + "_id": "def456", + "_score": 1, + "_source": { + "filepath": "/home/user/docs/report.pdf", + "filename": "report.pdf" + } + } + ] + } +} +``` diff --git a/_data-prepper/pipelines/substring-after.md b/_data-prepper/pipelines/substring-after.md new file mode 100644 index 00000000000..13768e12280 --- /dev/null +++ b/_data-prepper/pipelines/substring-after.md @@ -0,0 +1,112 @@ +--- +layout: default +title: substringAfter() +parent: Functions +grand_parent: Pipelines +nav_order: 60 +--- + +# substringAfter() + +The `substringAfter()` function is used to extract the portion of a string that follows the first occurrence of a specified delimiter. It takes two arguments: + +1. The first argument is either a literal string or a JSON pointer that represents the source string. + +1. The second argument is the delimiter string to search for within the first argument. + +If the delimiter is found, the function returns the portion of the string after the first occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. + +For example, to extract the value after the first occurrence of the `=` character in a field named `header`, use the `substringAfter()` function as follows: + +``` +'substringAfter(/header, "=")' +``` +{% include copy.html %} + +If `/header` contains `Content-Type=application/json`, the function returns `application/json`. + +Alternatively, you can use a literal string as the first argument: + +``` +'substringAfter("hello-world-foo", "-")' +``` +{% include copy.html %} + +The function returns `world-foo` because it extracts the portion of the string after the first `-` character. + +The `substringAfter()` function performs a case-sensitive search. +{: .note} + +## Example + +The following pipeline uses the `substringAfter()` function to extract the domain name from an email address field. It adds the extracted domain name as a new field called `domain`: + +```yaml +substring-after-demo: + source: + http: + ssl: false + + processor: + - add_entries: + entries: + - key: domain + value_expression: 'substringAfter(/email, "@")' + + sink: + - opensearch: + hosts: ["https://opensearch:9200"] + insecure: true + username: admin + password: admin_password + index_type: custom + index: demo-index-%{yyyy.MM.dd} +``` +{% include copy.html %} + +You can test the pipeline using the following command: + +```bash +curl -sS -X POST "http://localhost:2021/log/ingest" \ + -H "Content-Type: application/json" \ + -d '[ + {"email":"user@example.com"}, + {"email":"admin@opensearch.org"} + ]' +``` +{% include copy.html %} + +The documents stored in OpenSearch contain the following information: + +```json +{ + ... + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "demo-index-2026.03.13", + "_id": "abc123", + "_score": 1, + "_source": { + "email": "user@example.com", + "domain": "example.com" + } + }, + { + "_index": "demo-index-2026.03.13", + "_id": "def456", + "_score": 1, + "_source": { + "email": "admin@opensearch.org", + "domain": "opensearch.org" + } + } + ] + } +} +``` diff --git a/_data-prepper/pipelines/substring-before-last.md b/_data-prepper/pipelines/substring-before-last.md new file mode 100644 index 00000000000..a5ed2805c83 --- /dev/null +++ b/_data-prepper/pipelines/substring-before-last.md @@ -0,0 +1,112 @@ +--- +layout: default +title: substringBeforeLast() +parent: Functions +grand_parent: Pipelines +nav_order: 90 +--- + +# substringBeforeLast() + +The `substringBeforeLast()` function is used to extract the portion of a string that precedes the last occurrence of a specified delimiter. It takes two arguments: + +1. The first argument is either a literal string or a JSON pointer that represents the source string. + +1. The second argument is the delimiter string to search for within the first argument. + +If the delimiter is found, the function returns the portion of the string before the last occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. + +For example, to remove the file extension from a filename field, use the `substringBeforeLast()` function as follows: + +``` +'substringBeforeLast(/filename, ".")' +``` +{% include copy.html %} + +If the `/filename` field contains `archive.tar.gz`, the function returns `archive.tar`. + +Alternatively, you can use a literal string as the first argument: + +``` +'substringBeforeLast("one-two-three", "-")' +``` +{% include copy.html %} + +The function returns `one-two` because it extracts the portion of the string before the last `-` character. + +The `substringBeforeLast()` function performs a case-sensitive search. +{: .note} + +## Example + +The following pipeline uses the `substringBeforeLast()` function to extract the directory path from a full file path. It adds the extracted directory path as a new field called `directory`: + +```yaml +substring-before-last-demo: + source: + http: + ssl: false + + processor: + - add_entries: + entries: + - key: directory + value_expression: 'substringBeforeLast(/filepath, "/")' + + sink: + - opensearch: + hosts: ["https://opensearch:9200"] + insecure: true + username: admin + password: admin_password + index_type: custom + index: demo-index-%{yyyy.MM.dd} +``` +{% include copy.html %} + +You can test the pipeline using the following command: + +```bash +curl -sS -X POST "http://localhost:2021/log/ingest" \ + -H "Content-Type: application/json" \ + -d '[ + {"filepath":"/var/log/syslog"}, + {"filepath":"/home/user/docs/report.pdf"} + ]' +``` +{% include copy.html %} + +The documents stored in OpenSearch contain the following information: + +```json +{ + ... + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "demo-index-2026.03.13", + "_id": "abc123", + "_score": 1, + "_source": { + "filepath": "/var/log/syslog", + "directory": "/var/log" + } + }, + { + "_index": "demo-index-2026.03.13", + "_id": "def456", + "_score": 1, + "_source": { + "filepath": "/home/user/docs/report.pdf", + "directory": "/home/user/docs" + } + } + ] + } +} +``` diff --git a/_data-prepper/pipelines/substring-before.md b/_data-prepper/pipelines/substring-before.md new file mode 100644 index 00000000000..47796d6678c --- /dev/null +++ b/_data-prepper/pipelines/substring-before.md @@ -0,0 +1,112 @@ +--- +layout: default +title: substringBefore() +parent: Functions +grand_parent: Pipelines +nav_order: 80 +--- + +# substringBefore() + +The `substringBefore()` function is used to extract the portion of a string that precedes the first occurrence of a specified delimiter. It takes two arguments: + +1. The first argument is either a literal string or a JSON pointer that represents the source string. + +1. The second argument is the delimiter string to search for within the first argument. + +If the delimiter is found, the function returns the portion of the string before the first occurrence of the delimiter. If the delimiter is not found, the original string is returned. If the source resolves to `null`, the function returns `null`. If the delimiter is `null` or empty, the original string is returned. + +For example, to extract the username from an email address field, use the `substringBefore()` function as follows: + +``` +'substringBefore(/email, "@")' +``` +{% include copy.html %} + +If the `/email` field contains `user@example.com`, the function returns `user`. + +Alternatively, you can use a literal string as the first argument: + +``` +'substringBefore("hello-world-foo", "-")' +``` +{% include copy.html %} + +The function returns `hello` because it extracts the portion of the string before the first `-` character. + +The `substringBefore()` function performs a case-sensitive search. +{: .note} + +## Example + +The following pipeline uses the `substringBefore()` function to extract the URL protocol from a URL field. It adds the extracted protocol as a new field called `protocol`: + +```yaml +substring-before-demo: + source: + http: + ssl: false + + processor: + - add_entries: + entries: + - key: protocol + value_expression: 'substringBefore(/url, "://")' + + sink: + - opensearch: + hosts: ["https://opensearch:9200"] + insecure: true + username: admin + password: admin_password + index_type: custom + index: demo-index-%{yyyy.MM.dd} +``` +{% include copy.html %} + +You can test the pipeline using the following command: + +```bash +curl -sS -X POST "http://localhost:2021/log/ingest" \ + -H "Content-Type: application/json" \ + -d '[ + {"url":"https://opensearch.org/docs"}, + {"url":"http://example.com/page"} + ]' +``` +{% include copy.html %} + +The documents stored in OpenSearch contain the following information: + +```json +{ + ... + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "demo-index-2026.03.13", + "_id": "abc123", + "_score": 1, + "_source": { + "url": "https://opensearch.org/docs", + "protocol": "https" + } + }, + { + "_index": "demo-index-2026.03.13", + "_id": "def456", + "_score": 1, + "_source": { + "url": "http://example.com/page", + "protocol": "http" + } + } + ] + } +} +```