AWS CloudWatch Logs Setup Guide for AuditFlow

Complete guide for setting up AWS CloudWatch Logs as a sink destination for AuditFlow audit events.

Overview
Prerequisites
Setup Options
Configuration Examples
IAM Permissions
Testing
Troubleshooting

Overview

The AWS CloudWatch Logs sink sends audit events to CloudWatch Logs for:

Real-time monitoring and alerting
Centralized log aggregation
Integration with AWS services (Lambda, CloudWatch Insights, etc.)
Compliance and audit trail requirements

Prerequisites

AWS Account with CloudWatch Logs access
IAM Permissions to create/write to log groups and streams
AWS Credentials configured (IAM role, access keys, or instance profile)
Python boto3 library installed in the sink service

Install boto3:

pip install boto3

Setup Options

Option 1: Same AWS Account

Use when AuditFlow runs in the same AWS account where you want to store logs.

Step 1: Create Log Group (Optional)

The sink can auto-create log groups, but for production it's recommended to create them manually:

# Using AWS CLI
aws logs create-log-group \
  --log-group-name /aws/auditflow/production \
  --region us-east-1

# Add retention policy (optional)
aws logs put-retention-policy \
  --log-group-name /aws/auditflow/production \
  --retention-in-days 30 \
  --region us-east-1

# Add tags
aws logs tag-log-group \
  --log-group-name /aws/auditflow/production \
  --tags Environment=production,Application=auditflow \
  --region us-east-1

Step 2: Create IAM Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CloudWatchLogsAccess",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:ACCOUNT-ID:log-group:/aws/auditflow/*"
      ]
    }
  ]
}

Step 3: Configure AuditFlow Sink

Using IAM Role (Recommended for EKS/EC2):

pipelines:
  - name: 'cloudwatch-prod'
    enabled: true
    transformer:
      name: 'zero'
    sink:
      name: 'aws_cloudwatch_sink'
      properties:
        log-group: "/aws/auditflow/production"
        log-stream: "app-events"
        region: "us-east-1"
        create-log-group: "false"  # Already created
        create-log-stream: "true"

Using Access Keys:

pipelines:
  - name: 'cloudwatch-prod'
    enabled: true
    transformer:
      name: 'zero'
    sink:
      name: 'aws_cloudwatch_sink'
      properties:
        log-group: "/aws/auditflow/production"
        log-stream: "app-events"
        region: "us-east-1"
        access-key-id: "${AWS_ACCESS_KEY_ID}"
        secret-access-key: "${AWS_SECRET_ACCESS_KEY}"
        create-log-group: "false"
        create-log-stream: "true"

Option 2: Cross-Account Access

Use when AuditFlow runs in one AWS account but needs to send logs to another account.

Scenario:

Source Account: 111111111111 (where AuditFlow runs)
Target Account: 222222222222 (where logs are stored)

Step 1: Create Log Group in Target Account

# Login to target account (222222222222)
aws logs create-log-group \
  --log-group-name /aws/auditflow/production \
  --region us-east-1

Step 2: Create IAM Role in Target Account

Create role AuditFlowCloudWatchRole with this trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "auditflow-unique-id"
        }
      }
    }
  ]
}

Step 3: Attach Policy to Role in Target Account

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:222222222222:log-group:/aws/auditflow/production:*"
      ]
    }
  ]
}

Step 4: Create IAM Policy in Source Account

Allow the source account's IAM role/user to assume the target role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::222222222222:role/AuditFlowCloudWatchRole"
    }
  ]
}

Step 5: Configure Application to Use AssumeRole

For EKS with IRSA (IAM Roles for Service Accounts):

Create service account with assume role annotation:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: auditflow-sink
  namespace: labs64io
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/AuditFlowSourceRole

Update deployment to use the service account:

spec:
  serviceAccountName: auditflow-sink

Note: The boto3 SDK will automatically handle the cross-account role assumption if configured properly. You may need to update the sink code to explicitly assume the role.

Option 3: AWS Organizations

Use when you have multiple accounts in an AWS Organization and want centralized logging.

Architecture

AWS Organization
├── Management Account (123456789012)
├── Security/Logging Account (999999999999) ← Store all logs here
├── Production Account (111111111111) ← AuditFlow runs here
└── Development Account (222222222222)

Step 1: Create Log Group in Logging Account

# In Security/Logging account (999999999999)
aws logs create-log-group \
  --log-group-name /aws/auditflow/production \
  --region us-east-1

aws logs put-retention-policy \
  --log-group-name /aws/auditflow/production \
  --retention-in-days 365 \
  --region us-east-1

Step 2: Create Cross-Account IAM Role

In the Logging Account (999999999999):

Trust Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::111111111111:root",
          "arn:aws:iam::222222222222:root"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Permissions Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:999999999999:log-group:/aws/auditflow/*"
      ]
    }
  ]
}

Step 3: Configure Each Application Account

In each account where AuditFlow runs (production, development):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::999999999999:role/AuditFlowCentralLoggingRole"
    }
  ]
}

Step 4: AuditFlow Configuration

pipelines:
  - name: 'cloudwatch-central'
    enabled: true
    transformer:
      name: 'zero'
    sink:
      name: 'aws_cloudwatch_sink'
      properties:
        log-group: "/aws/auditflow/production"
        log-stream: "account-${ACCOUNT_ID}"
        region: "us-east-1"
        # Credentials will use IAM role with cross-account access
        create-log-group: "false"
        create-log-stream: "true"

Configuration Examples

Development Environment

pipelines:
  - name: 'cloudwatch-dev'
    enabled: true
    transformer:
      name: 'zero'
    sink:
      name: 'aws_cloudwatch_sink'
      properties:
        log-group: "/aws/auditflow/development"
        log-stream: "dev-events"
        region: "us-east-1"
        create-log-group: "true"
        create-log-stream: "true"

Production with Multiple Streams

pipelines:
  # API events
  - name: 'cloudwatch-api'
    enabled: true
    transformer:
      name: 'zero'
    sink:
      name: 'aws_cloudwatch_sink'
      properties:
        log-group: "/aws/auditflow/production"
        log-stream: "api-events"
        region: "us-east-1"
        create-log-stream: "true"

  # Auth events
  - name: 'cloudwatch-auth'
    enabled: true
    transformer:
      name: 'zero'
    sink:
      name: 'aws_cloudwatch_sink'
      properties:
        log-group: "/aws/auditflow/production"
        log-stream: "auth-events"
        region: "us-east-1"
        create-log-stream: "true"

  # Admin events
  - name: 'cloudwatch-admin'
    enabled: true
    transformer:
      name: 'zero'
    sink:
      name: 'aws_cloudwatch_sink'
      properties:
        log-group: "/aws/auditflow/production"
        log-stream: "admin-events"
        region: "us-east-1"
        create-log-stream: "true"

Multi-Region Setup

pipelines:
  # US East
  - name: 'cloudwatch-us-east'
    enabled: true
    transformer:
      name: 'zero'
    sink:
      name: 'aws_cloudwatch_sink'
      properties:
        log-group: "/aws/auditflow/production"
        log-stream: "us-east-1-events"
        region: "us-east-1"

  # EU West
  - name: 'cloudwatch-eu-west'
    enabled: true
    transformer:
      name: 'zero'
    sink:
      name: 'aws_cloudwatch_sink'
      properties:
        log-group: "/aws/auditflow/production"
        log-stream: "eu-west-1-events"
        region: "eu-west-1"

IAM Permissions

Minimum Permissions (Read/Write)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:PutLogEvents",
        "logs:CreateLogStream",
        "logs:DescribeLogStreams"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/aws/auditflow/*:*"
    }
  ]
}

Full Permissions (Create/Read/Write)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/aws/auditflow/*"
    }
  ]
}

EKS IRSA Service Account Annotation

apiVersion: v1
kind: ServiceAccount
metadata:
  name: auditflow-sink
  namespace: labs64io
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT-ID:role/AuditFlowCloudWatchRole

Testing

Test Log Group Creation

# Create test log group
aws logs create-log-group \
  --log-group-name /aws/auditflow/test \
  --region us-east-1

# Verify it exists
aws logs describe-log-groups \
  --log-group-name-prefix /aws/auditflow/test \
  --region us-east-1

Test Log Stream Creation

# Create log stream
aws logs create-log-stream \
  --log-group-name /aws/auditflow/test \
  --log-stream-name test-stream \
  --region us-east-1

# Verify it exists
aws logs describe-log-streams \
  --log-group-name /aws/auditflow/test \
  --log-stream-name-prefix test-stream \
  --region us-east-1

Test Event Submission

# Send test event
aws logs put-log-events \
  --log-group-name /aws/auditflow/test \
  --log-stream-name test-stream \
  --log-events timestamp=$(date +%s%3N),message='{"test":"event"}' \
  --region us-east-1

Test AuditFlow Sink

# Test the sink endpoint
curl -X POST "http://localhost:8082/sink/aws_cloudwatch_sink" \
  -H "Content-Type: application/json" \
  -d '{
    "event_data": {
      "eventType": "test",
      "timestamp": "2025-12-17T10:00:00Z",
      "message": "Test audit event"
    },
    "properties": {
      "log-group": "/aws/auditflow/test",
      "log-stream": "test-stream",
      "region": "us-east-1",
      "create-log-group": "false",
      "create-log-stream": "false"
    }
  }'

Query Logs

# Get recent log events
aws logs get-log-events \
  --log-group-name /aws/auditflow/test \
  --log-stream-name test-stream \
  --limit 10 \
  --region us-east-1

# Use CloudWatch Insights
aws logs start-query \
  --log-group-name /aws/auditflow/production \
  --start-time $(date -u -d '1 hour ago' +%s) \
  --end-time $(date -u +%s) \
  --query-string 'fields @timestamp, @message | sort @timestamp desc | limit 20' \
  --region us-east-1

Troubleshooting

Error: ResourceNotFoundException

Problem: Log group does not exist.

Solution:

Set create-log-group: "true" in configuration, OR

Manually create the log group:

aws logs create-log-group --log-group-name /aws/auditflow/production --region us-east-1

Error: AccessDeniedException

Problem: IAM permissions are insufficient.

Solution:

Verify IAM policy includes required actions
Check resource ARN matches log group name
For cross-account access, verify trust relationship and assume role permissions

Error: InvalidSequenceTokenException

Problem: Concurrent writes to the same log stream.

Solution:

Use different log streams for different sources
The sink automatically handles sequence tokens - ensure you're not writing from multiple places

Error: DataAlreadyAcceptedException

Problem: Event was already sent (duplicate).

Solution: This is informational - the event was successfully stored, just logged as duplicate.

Verify IAM Role/Permissions

# Check current identity
aws sts get-caller-identity

# Test log group access
aws logs describe-log-groups \
  --log-group-name-prefix /aws/auditflow \
  --region us-east-1

# Simulate policy
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::ACCOUNT-ID:role/AuditFlowRole \
  --action-names logs:PutLogEvents \
  --resource-arns arn:aws:logs:us-east-1:ACCOUNT-ID:log-group:/aws/auditflow/production:*

Debug Boto3 SDK

Add to sink configuration or container environment:

export AWS_SDK_LOGGING=debug
export BOTO_LOG_LEVEL=DEBUG

Check Container/Pod IAM Role

For EKS:

# Check service account
kubectl get serviceaccount auditflow-sink -n labs64io -o yaml

# Check pod IAM role
kubectl describe pod <pod-name> -n labs64io | grep AWS_ROLE_ARN

# Exec into pod and test
kubectl exec -it <pod-name> -n labs64io -- sh
aws sts get-caller-identity
aws logs describe-log-groups --region us-east-1

Best Practices

Use IAM Roles instead of access keys when possible
Pre-create log groups in production for better control

Set retention policies to manage costs:

aws logs put-retention-policy --log-group-name /aws/auditflow/production --retention-in-days 90

Use separate log groups for different environments (dev/staging/prod)

Enable log group encryption using KMS:

aws logs associate-kms-key --log-group-name /aws/auditflow/production --kms-key-id arn:aws:kms:region:account:key/key-id

Monitor CloudWatch costs - use log filtering and sampling for high-volume events
Tag log groups for cost allocation and management
Use CloudWatch Insights for querying and analysis instead of downloading logs

FilesExpand file tree

aws-cloudwatch-setup.md

Latest commit

History

aws-cloudwatch-setup.md

File metadata and controls

AWS CloudWatch Logs Setup Guide for AuditFlow

Table of Contents

Overview

Prerequisites

Setup Options

Option 1: Same AWS Account

Step 1: Create Log Group (Optional)

Step 2: Create IAM Policy

Step 3: Configure AuditFlow Sink

Option 2: Cross-Account Access

Step 1: Create Log Group in Target Account

Step 2: Create IAM Role in Target Account

Step 3: Attach Policy to Role in Target Account

Step 4: Create IAM Policy in Source Account

Step 5: Configure Application to Use AssumeRole

Option 3: AWS Organizations

Architecture

Step 1: Create Log Group in Logging Account

Step 2: Create Cross-Account IAM Role

Step 3: Configure Each Application Account

Step 4: AuditFlow Configuration

Configuration Examples

Development Environment

Production with Multiple Streams

Multi-Region Setup

IAM Permissions

Minimum Permissions (Read/Write)

Full Permissions (Create/Read/Write)

EKS IRSA Service Account Annotation

Testing

Test Log Group Creation

Test Log Stream Creation

Test Event Submission

Test AuditFlow Sink

Query Logs

Troubleshooting

Error: ResourceNotFoundException

Error: AccessDeniedException

Error: InvalidSequenceTokenException

Error: DataAlreadyAcceptedException

Verify IAM Role/Permissions

Debug Boto3 SDK

Check Container/Pod IAM Role

Best Practices

Additional Resources