Skip to content

Add ASIM Network Session parser and hunting queries for AWS GuardDuty - #14769

Open
OluOlus wants to merge 5 commits into
Azure:masterfrom
OluOlus:add-guardduty-asim-parser
Open

Add ASIM Network Session parser and hunting queries for AWS GuardDuty#14769
OluOlus wants to merge 5 commits into
Azure:masterfrom
OluOlus:add-guardduty-asim-parser

Conversation

@OluOlus

@OluOlus OluOlus commented Jul 27, 2026

Copy link
Copy Markdown

Adds ASimNetworkSessionAWSGuardDuty (ASIM Network Session schema v0.2.6) and three hunting queries to the Amazon Web Services solution, extending GuardDuty coverage beyond the existing analytic rule with structured, cross-source-hunt-ready content.

  • Parser queries the AWSGuardDuty table directly — no dependency on any private/unsubmitted function.
  • Hunting queries: high-severity findings clustered by account/actor, EKS privilege escalation & credential access, and public/weakly-protected S3 buckets.
  • Validation: schema mapping and redacted sample findings for network/IAM/S3/EKS finding families are documented; ASimTester schema/data test evidence from a live workspace to follow as a PR update.

Related discussion: #14768

OluOlus added 2 commits July 27, 2026 23:23
Adds ASimNetworkSessionAWSGuardDuty, querying the AWSGuardDuty table
directly with no dependency on any private/unsubmitted function, plus
three hunting queries: high-severity findings clustered by account and
actor, EKS privilege escalation/credential access, and public or
weakly-protected S3 buckets.
@OluOlus
OluOlus requested review from a team as code owners July 27, 2026 22:47
@v-atulyadav v-atulyadav self-assigned this Jul 28, 2026
@v-atulyadav v-atulyadav added ASIM Solution Solution specialty review needed labels Jul 28, 2026
@v-atulyadav
v-atulyadav requested a review from Copilot July 28, 2026 04:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds AWS GuardDuty hunting content and an ASIM Network Session parser to make GuardDuty findings more usable for cross-source hunting and normalized analytics.

Changes:

  • Added 3 new GuardDuty hunting queries (S3 exposure, high-severity clustering, EKS privilege escalation/credential access).
  • Added ASimNetworkSessionAWSGuardDuty parser targeting ASIM Network Session schema v0.2.6.
  • Introduced entity mappings for hunting query results (account/actor) and ASIM normalization fields for GuardDuty network-context findings.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 9 comments.

File Description
Solutions/Amazon Web Services/Hunting Queries/AWS_GuardDuty_S3PublicExposure.yaml New hunting query to group S3 public exposure / weak protection findings by bucket.
Solutions/Amazon Web Services/Hunting Queries/AWS_GuardDuty_HighSeverityFindingsCrossAccount.yaml New hunting query to cluster high-severity findings by account/activity/actor.
Solutions/Amazon Web Services/Hunting Queries/AWS_GuardDuty_EKSPrivilegeEscalationCredentialAccess.yaml New hunting query to investigate EKS/Kubernetes privilege escalation and credential access patterns.
Parsers/ASimNetworkSession/Parsers/ASimNetworkSessionAWSGuardDuty.yaml New ASIM NetworkSession parser to normalize GuardDuty findings that include network context.

Comment thread Solutions/Amazon Web Services/Hunting Queries/AWS_GuardDuty_S3PublicExposure.yaml Outdated
Comment thread Parsers/ASimNetworkSession/Parsers/ASimNetworkSessionAWSGuardDuty.yaml Outdated
Comment thread Parsers/ASimNetworkSession/Parsers/ASimNetworkSessionAWSGuardDuty.yaml Outdated
Comment thread Parsers/ASimNetworkSession/Parsers/ASimNetworkSessionAWSGuardDuty.yaml Outdated
Comment thread Parsers/ASimNetworkSession/Parsers/ASimNetworkSessionAWSGuardDuty.yaml Outdated
Comment thread Parsers/ASimNetworkSession/Parsers/ASimNetworkSessionAWSGuardDuty.yaml Outdated
- Parser: drop DNS_REQUEST (no remoteIpDetails to normalize), resolve
  Src/Dst by connectionDirection instead of always treating the AWS
  resource as Src, treat AWS_API_CALL's caller IP as Src with no
  fabricated destination, use TimeCreated instead of ingestion time for
  ASIM event timestamps.
- Hunting queries: remove Account entity mappings keyed on fields that
  mix instance ids, usernames, and the literal "unknown" (high-severity
  and EKS queries); stop folding the caller IP into the S3 query's actor
  field now that it has its own CallerIp column.
@OluOlus

OluOlus commented Jul 28, 2026

Copy link
Copy Markdown
Author

Thanks for the review — pushed a fix addressing all 9 comments:

  • Dropped DNS_REQUEST from the parser's scope (GuardDuty's dnsRequestAction has no remoteIpDetails, so those rows were always silently filtered out anyway).
  • Parser now resolves Src/Dst by connectionDirection instead of always treating the AWS resource as the source — inbound connections and port probes now correctly put the remote party in Src* and the AWS resource in Dst*. AWS_API_CALL findings now put the caller IP in Src* with no fabricated destination endpoint.
  • Switched EventStartTime/EventEndTime to TimeCreated instead of ingestion time.
  • Removed the Account entity mappings in the high-severity and EKS hunting queries — the underlying fields mix instance IDs, IAM usernames, and the literal unknown, so mapping them to Account risked incorrect entity enrichment. Kept the CloudApplication/AccountId mapping, which is unambiguous.
  • S3 hunting query's ActorUserName no longer falls back to an IP address (it now only holds the IAM username, if any); CallerIp already carries the IP separately.

@v-atulyadav v-atulyadav added the SafeToRun This is used only for ASim parsers Fork PR Pipeline run. label Jul 28, 2026
@OluOlus

OluOlus commented Jul 28, 2026

Copy link
Copy Markdown
Author

@OluOlus please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

  • (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
  • (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="Microsoft"

@OluOlus
OluOlus requested a review from Copilot July 28, 2026 12:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (10)

Parsers/ASimNetworkSession/Parsers/ASimNetworkSessionAWSGuardDuty.yaml:129

  • NetworkDirection will be "Unknown" for PORT_PROBE (and likely AWS_API_CALL) because NetworkAction_ is null for those action types. This contradicts the earlier logic that treats PORT_PROBE as effectively inbound. Consider deriving NetworkDirection from IsInbound_ / IsApiCall_ (e.g., force PORT_PROBE to "Inbound", and set API calls to a consistent value such as "Unknown" or a documented convention).
        NetworkDirection = case(
            tostring(NetworkAction_.connectionDirection) == "INBOUND",  "Inbound",
            tostring(NetworkAction_.connectionDirection) == "OUTBOUND", "Outbound",
            "Unknown"
        ),

Parsers/ASimNetworkSession/Parsers/ASimNetworkSessionAWSGuardDuty.yaml:151

  • DvcAction is derived from NetworkAction_.blocked, but NetworkAction_ is null for AWS_API_CALL and PORT_PROBE, which will default these records to "Allow". That’s misleading for non-network-connection findings. Consider setting DvcAction only for NETWORK_CONNECTION (and otherwise set to empty/unknown), or compute it per ActionType_.
        DvcAction   = iff(tobool(NetworkAction_.blocked) == true, "Deny", "Allow"),

Parsers/ASimNetworkSession/Parsers/ASimNetworkSessionAWSGuardDuty.yaml:132

  • Setting NetworkBytes/NetworkPackets to 0 implies a confirmed zero-byte/zero-packet session, which is different from “unknown/not provided by GuardDuty”. Prefer using typed nulls so downstream analytics don’t treat these as actual measured zeros.
        NetworkBytes   = int(0),
        NetworkPackets = int(0),

Parsers/ASimNetworkSession/Parsers/ASimNetworkSessionAWSGuardDuty.yaml:44

  • PORT_PROBE findings can contain multiple portProbeDetails entries, but the parser hard-codes [0], discarding additional probed ports/IP details. Consider mv-expand over portProbeDetails (one normalized row per probe detail) or aggregating details into sets (e.g., ports) depending on the intended ASIM cardinality.
        ActionType_ == "PORT_PROBE",         PortProbeAction_.portProbeDetails[0].remoteIpDetails,

Parsers/ASimNetworkSession/Parsers/ASimNetworkSessionAWSGuardDuty.yaml:59

  • PORT_PROBE findings can contain multiple portProbeDetails entries, but the parser hard-codes [0], discarding additional probed ports/IP details. Consider mv-expand over portProbeDetails (one normalized row per probe detail) or aggregating details into sets (e.g., ports) depending on the intended ASIM cardinality.
            ActionType_ == "PORT_PROBE", toint(PortProbeAction_.portProbeDetails[0].localPortDetails.port),

Solutions/Amazon Web Services/Hunting Queries/AWS_GuardDuty_HighSeverityFindingsCrossAccount.yaml:30

  • If Id is unique per GuardDuty finding row (typical for finding IDs), dcount(Id) adds unnecessary overhead versus count(). Consider replacing FindingCount = dcount(Id) with FindingCount = count() (and apply the same change in the other new hunting queries that use dcount(Id)).
      FindingCount = dcount(Id),
      DistinctFindingTypes = dcount(ActivityType),

Solutions/Amazon Web Services/Hunting Queries/AWS_GuardDuty_S3PublicExposure.yaml:24

  • Indexing s3BucketDetails[0] can yield empty strings when the array is missing/empty, which can cause unrelated findings to collapse into a single group (empty BucketName/BucketArn). Consider filtering to require a non-empty bucket identifier (preferably ARN), or handling absent/multi-entry s3BucketDetails explicitly before summarizing.
      BucketName = tostring(ResourceDetails.s3BucketDetails[0].name),
      BucketArn = tostring(ResourceDetails.s3BucketDetails[0].arn),

Solutions/Amazon Web Services/Hunting Queries/AWS_GuardDuty_S3PublicExposure.yaml:37

  • Indexing s3BucketDetails[0] can yield empty strings when the array is missing/empty, which can cause unrelated findings to collapse into a single group (empty BucketName/BucketArn). Consider filtering to require a non-empty bucket identifier (preferably ARN), or handling absent/multi-entry s3BucketDetails explicitly before summarizing.
      by AccountId, Region, ThreatPurpose, BucketName, BucketArn

Solutions/Amazon Web Services/Hunting Queries/AWS_GuardDuty_HighSeverityFindingsCrossAccount.yaml:26

  • As written, RemoteIp can become an empty string when neither source exists, and make_set(RemoteIp, 10) will then include "" in results. Consider filtering out empty RemoteIp values before summarizing, or using a conditional set pattern so the output isn’t polluted with empty entries.
      RemoteIp = coalesce(
          tostring(ServiceDetails.action.networkConnectionAction.remoteIpDetails.ipAddressV4),
          tostring(ServiceDetails.action.awsApiCallAction.remoteIpDetails.ipAddressV4)
      )

Solutions/Amazon Web Services/Hunting Queries/AWS_GuardDuty_HighSeverityFindingsCrossAccount.yaml:32

  • As written, RemoteIp can become an empty string when neither source exists, and make_set(RemoteIp, 10) will then include "" in results. Consider filtering out empty RemoteIp values before summarizing, or using a conditional set pattern so the output isn’t polluted with empty entries.
      FindingTypes = make_set(ActivityType, 10),
      RemoteIps = make_set(RemoteIp, 10),

Parser:
- Expand PORT_PROBE's portProbeDetails array instead of hard-coding
  index [0], so multi-port probes normalize to one row per probe.
- Derive NetworkDirection from the same Src/Dst direction logic used
  elsewhere, not raw connectionDirection (which is null for
  AWS_API_CALL/PORT_PROBE and was always falling through to "Unknown").
- NetworkBytes/NetworkPackets are now typed nulls, not 0 - GuardDuty
  doesn't report counts, so 0 would misrepresent an unknown as measured.
- DvcAction is now only set for NETWORK_CONNECTION; AWS_API_CALL and
  PORT_PROBE no longer default to a fabricated "Allow".

Hunting queries:
- FindingCount now uses count() instead of dcount(Id); Id is already
  unique per finding row.
- S3 query drops rows with no bucket ARN before summarizing, instead of
  silently collapsing them into one group with an empty BucketName/Arn.
- Set aggregations (RemoteIps, Workloads, Namespaces, CallerIps,
  PublicAccessStates, Actors) use make_set_if(..., isnotempty(...))
  instead of make_set, so empty strings don't pollute the output.
@OluOlus

OluOlus commented Jul 28, 2026

Copy link
Copy Markdown
Author

Pushed a fix for the second round of automated review feedback (10 low-confidence comments, all valid):

  • Parser: expand `portProbeDetails` instead of hard-coding `[0]` (multi-port probes were silently dropping all but the first probe); derive `NetworkDirection` from the same direction logic used for Src/Dst rather than raw `connectionDirection` (was always "Unknown" for PORT_PROBE/AWS_API_CALL since `NetworkAction_` is null there); `NetworkBytes`/`NetworkPackets` are now typed nulls instead of `0` (GuardDuty doesn't report counts — 0 implied a measured value); `DvcAction` is now only set for `NETWORK_CONNECTION` instead of defaulting other action types to a fabricated "Allow".
  • Hunting queries: `FindingCount` now uses `count()` instead of `dcount(Id)` (Id is already unique per row); the S3 query now drops rows with no bucket ARN before summarizing instead of collapsing them into one bogus group; set aggregations (`RemoteIps`, `Workloads`, `Namespaces`, `CallerIps`, `PublicAccessStates`, `Actors`) use `make_set_if(..., isnotempty(...))` instead of `make_set` so empty strings don't pollute the output.

Adds the 3 new hunting queries to Data/Solution_AmazonWebServices.json
and bumps the solution version to 3.0.11 with a ReleaseNotes.md entry,
per maintainer guidance that content added to an existing Solution
needs to be registered in its manifest. Does not regenerate
Package/mainTemplate.json or produce a new package zip - left to the
packaging pipeline/maintainer given the blast radius of repackaging the
whole existing solution.
@v-atulyadav v-atulyadav added SafeToRun This is used only for ASim parsers Fork PR Pipeline run. and removed SafeToRun This is used only for ASim parsers Fork PR Pipeline run. labels Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ASIM SafeToRun This is used only for ASim parsers Fork PR Pipeline run. Solution Solution specialty review needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants