[aws] Add var_groups for credential type selection with Identity Federation

[andrewkroh]

Proposed commit message

[aws] Add var_groups for credential type selection with Identity Federation

Bump format_version to 3.6.0 and version to 7.0.0. Reorganize AWS
credential configuration into a `Setup Access` var_groups selector with
six options: Identity Federation, Direct Access Keys, Temporary Access
Keys, Assume Role, Assume Role with External ID, and Shared Credentials.

Key changes:
- format_version: 3.4.0 → 3.6.0
- version: 6.20.3 → 7.0.0
- kibana.version: "^8.19.4 || ^9.2.1" → "^9.4.0"
- agent.version: "^9.4.0"
- var_groups: credential_type selector with 6 options
- external_id is now secret: true
- New vars: assume_role_duration, assume_role_expiry_window,
  supports_cloud_connectors
- hide_in_var_group_options for 13 inputs across services that don't
  support Identity Federation
- GuardDuty httpjson stream: switch to auth.aws: block and add Identity
  Federation policy tests
- Add conditions.agent.version: ^9.4.0 because guardduty now requires
  it.

Source: elastic/integrations#19278 (Omolola-Akinleye/integrations)

Summary

This PR carries the var_groups / Identity Federation work from #19278 forward on top of the processor-tags pre-landing from #19824.

What changed

• Bump format_version from 3.4.0 to 3.6.0 and package version from 6.20.3 to 7.0.0.
• Require Kibana and Elastic Agent ^9.4.0 for the var_groups UI feature and auth.aws runtime support.
• Add a credential_type var_groups selector with Identity Federation, Direct Access Keys, Temporary Access Keys, Assume Role, Assume Role with External ID, and Shared Credentials options.
• Add assume_role_duration, assume_role_expiry_window, and supports_cloud_connectors variables.
• Mark external_id as secret.
• Hide the Identity Federation option for inputs where that path has not yet been validated.
• Switch the GuardDuty httpjson stream from manual SigV4 HMAC signing to the native auth.aws block.
• Add GuardDuty Identity Federation and legacy credential policy test fixtures.
• Align Transit Gateway policy template categories with its metrics data stream.

Related

• [aws] Add var_groups for credential type selection with Cloud Connector support #19278 remake of 18762 with major version bump, basis for this PR
• [Cloud-Security] Adding var-groups to support authentication vars - related to cloud connector feature #18762 original PR
• elastic/security-team#17611 (internal)

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package aws 👍(9) 💚(5) 💔(8)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
firewall_logs	3968.25	3278.69	-689.56 (-17.38%)	💔
lambda_logs	10869.57	7575.76	-3293.81 (-30.3%)	💔
rds	31250	18867.92	-12382.08 (-39.62%)	💔
route53_public_logs	20000	10000	-10000 (-50%)	💔
route53_resolver_logs	7936.51	6622.52	-1313.99 (-16.56%)	💔
cloudwatch_logs	200000	142857.14	-57142.86 (-28.57%)	💔
config	4000	3039.51	-960.49 (-24.01%)	💔
emr_logs	18867.92	12345.68	-6522.24 (-34.57%)	💔

To see the full report comment with /test benchmark fullreport

[andrewkroh]

The PR's aws.config script test is failing because of the policy ID matching issues caused by the conditions.agent.version. Fix for this is in elastic/elastic-package#3729.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[infra-vault-gh-plugin-prod]

💔 Build Failed

• Buildkite Build
• Commit: 8f9dffa

Failed CI Steps

• Check integrations aws

History

• 💔 Build #45338 failed 7710222
• 💔 Build #45328 failed 7aa1477
• 💔 Build #45323 failed 9dea76e
• 💔 Build #45320 failed 51f64b7
• 💔 Build #45318 failed feb0a03

[github-actions]

TL;DR

Check integrations aws is failing on the AWS package script-test path because the package now sets conditions.agent.version: ^9.4.0, which exposes an elastic-package Fleet policy ID matching bug. This looks like a tooling/dependency issue, not an AWS integration code bug; the fix is in elastic/elastic-package#3729.

Remediation

• Wait for the elastic-package fix from internal/kibana/agents.go: Fix agent version policy ID matching elastic-package#3729 to be included in the Buildkite test environment, or rerun this job with an elastic-package build that includes that change.
• Re-run the Check integrations aws Buildkite step for commit 8f9dffa801ccca8363a96978a737ff04bacdb90d once the fixed tooling is available.

Investigation details

Root Cause

This PR introduces an Elastic Agent version condition in packages/aws/manifest.yml:20-27:

20: conditions:
23:   kibana:
24:     version: "^9.4.0"
25:   # auth.aws support in CEL and HTTPJSON inputs requires Elastic Agent 9.4.0+.
26:   agent:
27:     version: "^9.4.0"

With that condition present, Fleet can report a version-suffixed agent policy ID such as <policy-id>#9.5. The current elastic-package policy-assignment wait logic compares the IDs for exact equality, so it can time out even when the intended policy was assigned.

The upstream fix changes internal/kibana/agents.go from exact matching to accepting policyID + "#..." suffixes:

if assignedPolicyIDMatches(agent.PolicyID, p.ID) && agent.PolicyRevision >= p.Revision
...
return agentPolicyID == policyID || strings.HasPrefix(agentPolicyID, policyID+"#")

The PR head also points at this same failure mode in packages/aws/data_stream/config/_dev/test/scripts/missing_credentials.txt:4:

skip 'conditions.agent.version breaks this test. https://github.com/elastic/elastic-package/pull/3729'

Evidence

• Build: https://buildkite.com/elastic/integrations/builds/45361
• Job/step: Check integrations aws
• Key log excerpt available to this workflow:

--- [aws] failed
Error: The command exited with status 1
user command error: exit status 1

The pre-fetched Buildkite log starts at stack teardown/artifact upload, so it does not include the earlier script-test timeout text. The root-cause trace above is based on the PR-head AWS files, the linked elastic-package fix, and the maintainer note already on this PR that the aws.config script test is failing due to policy ID matching caused by conditions.agent.version.

Verification

Not run locally; Docker-backed elastic-package package testing is unavailable in this environment, and the provided Buildkite log lacks the earlier failing command output.

Follow-up

If this remains red after the fixed elastic-package is present in CI, inspect the uploaded build/test-results/aws-*.xml artifacts from the rerun to confirm whether a new AWS package failure replaced the policy-ID matching issue.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.