diff --git a/content/en/security/sensitive_data_scanner/guide/redact_all_emails_except_from_specific_domain_logs.md b/content/en/security/sensitive_data_scanner/guide/redact_all_emails_except_from_specific_domain_logs.md deleted file mode 100644 index fc7a3ea33e2..00000000000 --- a/content/en/security/sensitive_data_scanner/guide/redact_all_emails_except_from_specific_domain_logs.md +++ /dev/null @@ -1,76 +0,0 @@ ---- -title: Redact All Emails Except Those from a Specific Domain in Logs -disable_toc: false -aliases: - - /sensitive_data_scanner/guide/redact_all_emails_except_from_specific_domain_logs -further_reading: -- link: "/sensitive_data_scanner/" - tag: "Documentation" - text: "Set up Sensitive Data Scanner" -private: true ---- - -
This guide has been deprecated. See Suppress specific matches to ignore risk-accepted data to redact emails from a specific domain.
- -## Overview - -This guide walks you through how to redact all emails, except the ones from a specific email domain (for example, `@test.com`), in your logs. - -## Set up a grok parser in your logs pipeline - -If the email domain you do not want redacted is not an existing log attribute, set up a grok parser to identify all logs with the email domain and add it as an attribute. - -1. Navigate to [Log Pipeline][1]. -1. Select your pipeline. -1. Click **Add processor**. -1. Select **Grok Parser**. -1. Enter a name for the grok parser. -1. Define the parsing rules to identify all logs with the email address. For example, if these are the log messages that contain email addresses with the domain: - ``` - message successfully sent to 123@test.com - ``` - ``` - message successfully received from 256@test.com - ``` - Then use the following parsing rules: - ``` - MyParsingRule1 message successfully sent to %{notSpace:user_handle}@%{notSpace:domain} - - MyParsingRule2 message successfully received from %{notSpace:user_handle}@%{notSpace:domain} - ``` - **Note:** You don't need to keep the username. For example, if you want to redact all emails with the domain `test.com`, then for an email like `hello@test.com`, discard the username `hello` and just keep the domain `test.com`. -1. Click **Save**. - -Navigate to [Log Explorer][2] to confirm that new logs coming in with those emails are getting processed as expected. - -{{< img src="sensitive_data_scanner/guides/domain_attribute.png" alt="The domain attribute in log side panel" style="width:80%;" >}} - -## Add the email domain attribute as a facet - -1. In [Log Explorer][2], select a log that contains an email with the specified domain. -1. Click on the cog next to the domain attribute you just created. -1. Select **Create facet for...**. -1. Optionally, add the facet to a group in the **Advanced Options** section. -1. Click **Add**. - -## Configure the Sensitive Data Scanner scanning group to filter out logs with your domain attribute - -Update your Sensitive Data Scanner's scanning group to filter out logs with the domain attribute that you created, so only logs that do not have that email domain are redacted. - -1. Navigate to the Sensitive Data Scanner [Configuration][3] page. -1. Click the pencil icon to the left of the scanning group you want to update. -1. In the **Filter** field, add the domain attribute so that logs with that attribute are filtered out. For example, to filter out logs with the email domain `test.com`, add `-@domain:test.com` to the filter query. -{{< img src="sensitive_data_scanner/guides/scanning_group_filter_domain.png" alt="The scanning group's filter query with -@domain:test.com" style="width:100%;" >}} -1. Click **Update**. - -Navigate to [Log Explorer][2] to confirm that the new logs coming in do not have emails with the specified domain redacted. - -{{< img src="sensitive_data_scanner/guides/log_explorer_domain.png" alt="The Log Explorer showing logs with redacted email addresses and one log showing the test.com email unredacted" style="width:100%;" >}} - -## Further reading - -{{< partial name="whats-next/whats-next.html" >}} - -[1]: https://app.datadoghq.com/logs/pipelines -[2]: https://app.datadoghq.com/logs -[3]: https://app.datadoghq.com/organization-settings/sensitive-data-scanner/configuration \ No newline at end of file diff --git a/content/en/security/sensitive_data_scanner/guide/redact_uuids_in_logs.md b/content/en/security/sensitive_data_scanner/guide/redact_uuids_in_logs.md deleted file mode 100644 index 1d24ebb0b0f..00000000000 --- a/content/en/security/sensitive_data_scanner/guide/redact_uuids_in_logs.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: Redact Universal Unique IDs (UUIDs) in Logs -disable_toc: false -further_reading: -- link: "/security/sensitive_data_scanner/scanning_rules/custom_rules" - tag: "Documentation" - text: "Regex for custom scanning rules" -private: true ---- - -
This guide has been deprecated and is no longer maintained
- -## Overview - -This guide goes over how to create a custom scanning rule using a regular expression (regex) pattern to match and redact a Universally Unique Identified (UUID). For example, your organization might have a UUID for internal identification, with additional information appended, such as the user's: -- User ID -- Department code -- Status code - -If you want internal users to access these logs without exposing the UUID and user ID, you can create a custom scanning rule to redact the information. - -## Set up a custom rule to match a UUID - -For this guide, `01e2402104ca99-8641-43ba-b499-642610-0012` is the example internal identifier being used, where: -- `01e2402104ca99-8641-43ba-b499` is the UUID. -- `6462610` is a 6-digit value that represents the ID in byte format. -- `0012` is A 2-digit department code and 2-digit status code of a user: - - `00` is used for an active user. - - `12` for the department code. - -In this example, you want to match the format of the example identifier (`01e2402104ca99-8641-43ba-b499-642610-0012`) and: -- Redact the UUID, user ID, and the ID in byte format. -- But not redact the department and status code of the user. - -You can use the following basic regex to match the UUID and user ID that you want to redact: - -``` -[a-z0-9]{14}-\d{4}-[a-z0-9]{4}-[a-z0-9]{4}-\d{6} -``` - -1. Navigate to the [Sensitive Data Scanner settings][1] page. -1. Click **Add** and select **Add Scanning Rule**. -1. Click **Custom Rule**. -1. Select the scanning group to which you want to add this rule. -1. Enter a name for the rule. -1. Select the priority you want for the rule. -1. Enter a description for the rule. -1. In the **Match conditions** section, enter `[a-z0-9]{14}-\d{4}-[a-z0-9]{4}-[a-z0-9]{4}-\d{6}` in the regex field. - {{< img src="sensitive_data_scanner/guides/regex_text_matched.png" alt="The regex test section showing that the UUID and user ID are matched" style="width:100%;" >}} -1. Use a keyword dictionary to refine detection accuracy and avoid false positives. For this example, you want to match within 10 characters of the word `user`: - 1. Enter `user` as a key word. - 1. Enter `10` for **Characters before match**. -1. In the **Action on Match** section and for this example: -1. Select **Entire Event** for how much of the event to scan. If you have the log parsed out using the Grok Parser, you can scan by specific attributes. - 1. Select **Redact** for the action on match. - 1. Enter `[removed]` for the replacement text. - 1. Enter `matched_on:user_id` for the **Add tags** field. -1. Click **Add rule**. - -If this example log, which contains the different components of the UUID, is sent to Datadog: - -``` -2024-11-14 14:20:22 INFO [transaction-logger] 200 OK shoe:200.000, pen:42.95. iron, 221.55, tasty-sandwich:10.95, big-coffee:11.95, user.name:fred91, user.id:01e2402104ca99-8641-43ba-b499-642610-0012, user.email:fred.jones@scooby.com function:transaction-complete.js, payment.ccn:1111-1111-1111-1111, payment.ccexp:10/30} -``` - -The result is the `user.id` is redacted and replaced with `[removed]`: - -{{< img src="sensitive_data_scanner/guides/redacted_log.png" alt="The log event with the UUID and user ID redacted and replaced with removed" style="width:100%;" >}} - - -## Further reading - -{{< partial name="whats-next/whats-next.html" >}} - -[1]: https://app.datadoghq.com/organization-settings/sensitive-data-scanner/configuration \ No newline at end of file diff --git a/content/en/security/sensitive_data_scanner/setup/telemetry_data.md b/content/en/security/sensitive_data_scanner/setup/telemetry_data.md index 0575a50ae93..dc8adb622c4 100644 --- a/content/en/security/sensitive_data_scanner/setup/telemetry_data.md +++ b/content/en/security/sensitive_data_scanner/setup/telemetry_data.md @@ -5,6 +5,8 @@ aliases: - /sensitive_data_scanner/setup/telemetry_data - /security/sensitive_data_scanner/guide/best_practices_for_creating_custom_rules - /sensitive_data_scanner/guide/best_practices_for_creating_custom_rules + - /security/sensitive_data_scanner/guide/redact_uuids_in_logs/ + - /security/sensitive_data_scanner/guide/redact_all_emails_except_from_specific_domain_logs/ further_reading: - link: "/security/sensitive_data_scanner/scanning_rules/library_rules" tag: "Documentation" diff --git a/translate.yaml b/translate.yaml index 5667a5e2fda..32d91cab632 100644 --- a/translate.yaml +++ b/translate.yaml @@ -9,8 +9,6 @@ sync_dry_run_enabled: false # These files will be ignored or deleted when syncing with Transifex. ignores: - - "content/en/security/sensitive_data_scanner/guide/redact_all_emails_except_from_specific_domain_logs.md" - - "content/en/security/sensitive_data_scanner/guide/redact_uuids_in_logs.md" - "content/en/observability_pipelines/legacy/*.md" - "content/en/**/faq/*.md" - "content/en/security/default_rules/*.md"