[Azure] Fix degraded auditlogs: raise keyword limits, fix missing field mappings, add pipeline tests#20132
Merged
Merged
Conversation
…ded documents Fleet's strings_as_keyword dynamic template applies ignore_above: 1024 to all keyword fields. Azure Entra ID audit payloads routinely exceed this limit on result_description, result_reason, additional_details.value, and the modified_properties leaf fields (old_value/new_value/display_name), causing those field values to be silently dropped and documents marked degraded. Raises ignore_above to 8191 (the safe UTF-8 keyword ceiling; Lucene's hard term limit is 32766 bytes, 32766/4 = 8191 chars) on the affected fields. The modified_properties group is also restructured from per-leaf keyword declarations to a single object_type: keyword template on modified_properties.*.*, working around a Fleet/EPM limitation where only the first declared leaf under a doubly-nested wildcard path gets its own dynamic template. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Elastic Docs Style Checker (Vale)Summary: 1 warning found
|
| File | Line | Rule | Message |
|---|---|---|---|
| packages/azure/data_stream/auditlogs/fields/fields.yml | 106 | Elastic.Latinisms | Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'. |
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.
Contributor
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
springcloudlogs |
8849.56 | 6172.84 | -2676.72 (-30.25%) | 💔 |
aadgraphactivitylogs |
1587.3 | 1317.52 | -269.78 (-17%) | 💔 |
activitylogs |
2320.19 | 1733.1 | -587.09 (-25.3%) | 💔 |
application_gateway |
4784.69 | 3846.15 | -938.54 (-19.62%) | 💔 |
auditlogs |
4784.69 | 3787.88 | -996.81 (-20.83%) | 💔 |
graphactivitylogs |
3003 | 2439.02 | -563.98 (-18.78%) | 💔 |
identity_protection |
9523.81 | 3937.01 | -5586.8 (-58.66%) | 💔 |
To see the full report comment with /test benchmark fullreport
andrewkroh
added
documentation
… pipeline The top-level resultDescription field from Azure audit log events was not being renamed to snake_case, unlike operationName, resultSignature, and operationVersion. The raw camelCase field landed as a dynamic mapping under strings_as_keyword (ignore_above: 1024) while the explicit keyword mapping for azure.auditlogs.result_description was never matched. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… fields Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
shmsr
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reduces “degraded documents” in the azure.auditlogs data stream by ensuring long keyword values aren’t silently dropped (via a higher ignore_above) and by normalizing the top-level resultDescription field to the correctly-mapped snake_case field name.
Changes:
- Raise
ignore_aboveto8191for selectedazure.auditlogskeyword fields and formodified_propertiesleaf values via a wildcard object mapping. - Update the auditlogs ingest pipeline to rename
azure.auditlogs.resultDescription→azure.auditlogs.result_description. - Bump the Azure package version and update docs/changelog accordingly.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/azure/manifest.yml | Bumps package version to 1.37.1. |
| packages/azure/docs/adlogs.md | Updates exported-fields documentation for the new modified_properties.*.* mapping shape. |
| packages/azure/data_stream/auditlogs/fields/fields.yml | Raises ignore_above to 8191 and replaces the per-leaf modified_properties mapping with a wildcard object mapping. |
| packages/azure/data_stream/auditlogs/elasticsearch/ingest_pipeline/default.yml | Adds rename processor to normalize resultDescription to result_description. |
| packages/azure/changelog.yml | Adds a 1.37.1 bugfix entry describing the degraded-document reduction changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
shmsr
changed the title
Adds a test case exercising the resultDescription → result_description rename processor added in the previous commit, as flagged in code review. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…test PR elastic#9701 added a pipeline rename for properties.resultDescription → properties.result_description but never declared the field in fields.yml or added test coverage. This adds both: - fields.yml: declare properties.result_description as keyword/8191 - adlogs.md: add the field to the docs table - test fixture: second event in test-auditlogs-result-description.log exercises the properties rename end-to-end Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
shmsr
changed the title
Member
Author
|
@elastic/obs-ds-hosted-services and @elastic/security-service-integrations could you PTAL? |
ishleenk17
approved these changes
Jul 20, 2026
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
zmoog
approved these changes
Jul 20, 2026
Mention the missing properties.result_description mapping and rollover scope in the changelog, and replace e.g. with for example.
shmsr
commented
Jul 20, 2026
shmsr
enabled auto-merge (squash)
July 20, 2026 18:06
Contributor
|
✅ All changelog entries have the correct PR link. |
💚 Build Succeeded
History
cc @shmsr |
Contributor
|
Package azure - 1.37.1 containing this change is available at https://epr.elastic.co/package/azure/1.37.1/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
This change reduces those degraded documents by:
ignore_aboveto8191for:azure.auditlogs.result_descriptionazure.auditlogs.properties.result_reasonazure.auditlogs.properties.result_description(newly declared — see below)azure.auditlogs.properties.additional_details.valueazure.auditlogs.properties.target_resources.*.modified_properties.*.*azure.auditlogs.resultDescriptiontoazure.auditlogs.result_descriptionin the ingest pipelinemodified_properties.*.{new_value,display_name,old_value}declarations with a single
object_type: keywordmapping onmodified_properties.*.*properties.resultDescription → properties.result_descriptionwas renamed in the pipeline but the target fieldwas never declared in
fields.yml, leaving it unmapped since 2023resultDescriptionrenamesThe
modified_propertiespath is a doubly-nested wildcard path(
target_resources.* -> modified_properties.*). The previous per-leaf mappingappears to hit the same Fleet/EPM shared-section mapping-generation issue
described in elastic/kibana#270746, where only the first declared leaf receives
the intended package dynamic template and later leaves can fall through to
Fleet's fallback
strings_as_keywordtemplate.ignore_above: 8191is used instead of32766becauseignore_aboveismeasured in characters while Lucene's hard term limit is 32,766 bytes. UTF-8
characters can use up to 4 bytes, so
32766 / 4 = 8191is the largest safecharacter limit that avoids hard document rejection for all UTF-8 input. This is
also the logsdb default for keyword fields in Elasticsearch 9.x.
This is not a field-type breaking change. The affected fields remain
keyword,so field capabilities remain compatible across old and new backing indices.
Existing documents that were already degraded are not retroactively repaired.
Existing backing indices also keep their current mappings unless their mappings
are explicitly updated; the package template change applies to new backing
indices created after upgrade.
Checklist
changelog.ymlfile.Author's Checklist
ignore_above: 8191.resultDescription.modified_properties.*leaf mappings into onemodified_properties.*.*keyword object mapping.resultDescriptionrename.properties.resultDescriptionrename; declared missingproperties.result_descriptionfield infields.yml.