Skip to content

Commit 89a836d

Browse files
Yara-abYara-ab
authored
Merge branch 'main' into reranker-rag
2 parents bb17d38 + baf6f91 commit 89a836d

8 files changed

Lines changed: 805 additions & 0 deletions

File tree

‎observability-and-management/logging-analytics/finops/aws-focus-to-oci-object-storage/README.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# AWS FOCUS 1.2 Daily Export to OCI Object Storage
2+
3+
This repository contains a reusable reference implementation for moving AWS FOCUS 1.2 daily billing export files from Amazon S3 into OCI Object Storage using AWS Lambda and EventBridge Scheduler.
4+
5+
The solution is designed for internal review and reuse. It includes:
6+
7+
- AWS Data Export setup notes
8+
- Lambda code
9+
- Environment variable examples
10+
- IAM policy examples
11+
- EventBridge payload examples
12+
- Logging and audit guidance
13+
- Troubleshooting notes
14+
- Sanitized screenshots / diagrams can be added later if needed
15+
16+
## High-level flow
17+
18+
AWS Billing and Cost Management Data Export -> Amazon S3 -> AWS Lambda -> OCI Object Storage
19+
20+
CloudWatch Logs capture runtime logs, and the Lambda also writes a monthly JSONL audit log back to S3.
21+
22+
## Repository layout
23+
24+
```text
25+
aws-focus-to-oci-object-storage/
26+
├── README.md
27+
├── docs/
28+
│ ├── architecture.md
29+
│ ├── setup-runbook.md
30+
│ └── troubleshooting.md
31+
├── examples/
32+
│ ├── eventbridge-payload.json
33+
│ ├── iam-policy.json
34+
│ └── lambda-env-vars.example
35+
└── src/
36+
└── lambda_function_focus_daily_flat_transfer.py
37+
```
38+
39+
## What this solution does
40+
41+
- Reads AWS FOCUS 1.2 daily `.csv.gz` files from S3.
42+
- Uses a configurable lookback window to identify recently modified files.
43+
- Flattens the S3 path into a single OCI object name so OCI does not create nested folder structure.
44+
- Uploads the file to OCI Object Storage using a pre-authenticated request (PAR) URL.
45+
- Writes structured execution logs to CloudWatch.
46+
- Appends a monthly transfer audit log into an S3 `transfer-logs/` prefix.
47+
48+
## Quick start
49+
50+
1. Copy `examples/lambda-env-vars.example` into Lambda environment variables.
51+
2. Deploy `src/lambda_function_focus_daily_flat_transfer.py` as the Lambda handler.
52+
3. Create an EventBridge schedule to run the Lambda daily.
53+
4. Verify CloudWatch logs and the S3 audit log file.
54+
5. Confirm OCI receives flat file names at the bucket root (or with a blank destination prefix).
55+
56+
## License
57+
58+
Copyright (c) 2026 Oracle and/or its affiliates.
59+
60+
Licensed under the Universal Permissive License (UPL), Version 1.0.
61+
62+
See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.

‎observability-and-management/logging-analytics/finops/aws-focus-to-oci-object-storage/docs/architecture.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Architecture
2+
3+
```text
4+
AWS FOCUS Data Export
5+
|
6+
v
7+
Amazon S3 (daily .csv.gz files)
8+
|
9+
v
10+
AWS Lambda (scan, flatten, upload)
11+
|
12+
+----------------------+
13+
| |
14+
v v
15+
OCI Object Storage CloudWatch Logs
16+
|
17+
v
18+
Monthly S3 audit log (transfer-logs/)
19+
```
20+
21+
## Core entities
22+
23+
- AWS Data Export job
24+
- S3 bucket: `aws-oci-focus-daily`
25+
- S3 source prefix: `Dataexport/AWSDataExportDailyFocus/data/`
26+
- Lambda function: `s3-to-oci-focus-daily-flat-transfer`
27+
- EventBridge Scheduler
28+
- OCI bucket: `LabBucket`
29+
- CloudWatch log group: `/aws/lambda/s3-to-oci-focus-daily-flat-transfer`
30+
- S3 audit log prefix: `transfer-logs/`

‎observability-and-management/logging-analytics/finops/aws-focus-to-oci-object-storage/docs/setup-runbook.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# Setup Runbook
2+
3+
## 1. Prerequisites
4+
5+
- AWS permissions to manage Billing and Cost Management Data Exports
6+
- AWS permissions for the Lambda execution role to read S3 and write audit logs
7+
- AWS permission to create and manage EventBridge schedules
8+
- OCI PAR URL with write access to the target bucket
9+
- OCI Object Storage bucket
10+
11+
## 2. AWS Data Export
12+
13+
Use the AWS console:
14+
15+
```text
16+
AWS Console -> Billing and Cost Management -> Data Exports
17+
```
18+
19+
Configure or verify:
20+
21+
- Export type: AWS Billing and Cost Management Data Export
22+
- Specification: FOCUS 1.2 with AWS columns
23+
- Granularity: Daily
24+
- File format: `text/csv`
25+
- Compression: `Gzip`
26+
- Behavior: create new daily files instead of overwriting previous files
27+
- S3 destination path: `s3://aws-oci-focus-daily/Dataexport/AWSDataExportDailyFocus/data/`
28+
29+
## 3. Lambda function
30+
31+
Recommended function name:
32+
33+
```text
34+
s3-to-oci-focus-daily-flat-transfer
35+
```
36+
37+
Use the code in `src/lambda_function_focus_daily_flat_transfer.py`.
38+
39+
Recommended runtime settings:
40+
41+
- Python 3.x
42+
- Timeout: 15 minutes
43+
- Memory: 1024 MB
44+
- Ephemeral storage: 1024 MB
45+
46+
## 4. Environment variables
47+
48+
Copy the values from `examples/lambda-env-vars.example`.
49+
50+
Important behavior:
51+
52+
- `DEST_PREFIX` is left blank so OCI does not create folder-like structure.
53+
- `LOOKBACK_HOURS` is used because no DynamoDB / external state store is used.
54+
- `PROCESS_ALL=false` is recommended for scheduled runs.
55+
- `PROCESS_ALL=true` should be used only for backfill or testing.
56+
57+
## 5. OCI destination
58+
59+
The OCI base URL must use the PAR token format:
60+
61+
```text
62+
https://objectstorage.eu-frankfurt-1.oraclecloud.com/p/<PAR_TOKEN>/n/<namespace>/b/<bucket>/o
63+
```
64+
65+
Do not store the real PAR token in source control.
66+
67+
## 6. Scheduling
68+
69+
Use EventBridge Scheduler to run the Lambda daily.
70+
71+
Example cron:
72+
73+
```text
74+
cron(0 8 * * ? *)
75+
```
76+
77+
Example payload:
78+
79+
```json
80+
{
81+
"trigger": "eventbridge_daily_scan",
82+
"lookback_hours": 24,
83+
"process_all": false
84+
}
85+
```
86+
87+
## 7. Logging
88+
89+
- CloudWatch log group: `/aws/lambda/s3-to-oci-focus-daily-flat-transfer`
90+
- S3 audit log prefix: `s3://aws-oci-focus-daily/transfer-logs/`
91+
- Monthly log example: `focus-transfer-log-2026-06.jsonl`

‎observability-and-management/logging-analytics/finops/aws-focus-to-oci-object-storage/docs/troubleshooting.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Troubleshooting
2+
3+
## AccessDenied for `s3:ListBucket`
4+
5+
**Cause:** The Lambda execution role does not have permission to list the S3 bucket or prefixes.
6+
7+
**Fix:** Add `s3:ListBucket` to the Lambda role for:
8+
9+
- `arn:aws:s3:::aws-oci-focus-daily`
10+
- `Dataexport/AWSDataExportDailyFocus/data/`
11+
- `transfer-logs/`
12+
13+
## Wrong `SOURCE_PREFIX`
14+
15+
**Cause:** The Lambda is pointing at an older export path.
16+
17+
**Fix:** Use the current daily export path:
18+
19+
```text
20+
Dataexport/AWSDataExportDailyFocus/data/
21+
```
22+
23+
## OCI shows folders
24+
25+
**Cause:** The OCI object name contains `/` characters.
26+
27+
**Fix:** Keep `DEST_PREFIX` blank and flatten the source path into a single object name.
28+
29+
## Files are overwritten in OCI
30+
31+
**Cause:** The same destination object name is reused.
32+
33+
**Fix:** Keep the unique hash suffix in the output filename.
34+
35+
## No files are picked up
36+
37+
**Cause:** The lookback window is too small, or the export has not delivered files yet.
38+
39+
**Fix:** Increase `LOOKBACK_HOURS` temporarily or run with `PROCESS_ALL=true` for a one-time test/backfill.
40+
41+
## OCI upload fails
42+
43+
**Cause:** The PAR URL is expired, incorrect, or does not allow writes.
44+
45+
**Fix:** Regenerate the OCI PAR URL with write access and update the Lambda environment variable.

‎observability-and-management/logging-analytics/finops/aws-focus-to-oci-object-storage/examples/eventbridge-payload.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"trigger": "eventbridge_daily_scan",
3+
"lookback_hours": 24,
4+
"process_all": false
5+
}

‎observability-and-management/logging-analytics/finops/aws-focus-to-oci-object-storage/examples/iam-policy.json‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
{
2+
"Version": "2012-10-17",
3+
"Statement": [
4+
{
5+
"Sid": "ListFocusDailyExportBucket",
6+
"Effect": "Allow",
7+
"Action": [
8+
"s3:ListBucket"
9+
],
10+
"Resource": "arn:aws:s3:::aws-oci-focus-daily",
11+
"Condition": {
12+
"StringLike": {
13+
"s3:prefix": [
14+
"Dataexport/AWSDataExportDailyFocus/data/",
15+
"Dataexport/AWSDataExportDailyFocus/data/*",
16+
"transfer-logs/",
17+
"transfer-logs/*"
18+
]
19+
}
20+
}
21+
},
22+
{
23+
"Sid": "ReadFocusDailyExportObjects",
24+
"Effect": "Allow",
25+
"Action": [
26+
"s3:GetObject"
27+
],
28+
"Resource": "arn:aws:s3:::aws-oci-focus-daily/Dataexport/AWSDataExportDailyFocus/data/*"
29+
},
30+
{
31+
"Sid": "ReadWriteMonthlyTransferAuditLogs",
32+
"Effect": "Allow",
33+
"Action": [
34+
"s3:GetObject",
35+
"s3:PutObject"
36+
],
37+
"Resource": "arn:aws:s3:::aws-oci-focus-daily/transfer-logs/*"
38+
}
39+
]
40+
}

‎observability-and-management/logging-analytics/finops/aws-focus-to-oci-object-storage/examples/lambda-env-vars.example‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
SOURCE_BUCKET=aws-oci-focus-daily
2+
SOURCE_PREFIX=Dataexport/AWSDataExportDailyFocus/data/
3+
OCI_BASE_URL=https://objectstorage.eu-frankfurt-1.oraclecloud.com/p/<PAR_TOKEN>/n/<OCI_NAMESPACE>/b/<OCI_BUCKET>/o
4+
DEST_PREFIX=
5+
OUTPUT_FILE_PREFIX=aws_focus_daily
6+
LOOKBACK_HOURS=24
7+
PROCESS_ALL=false
8+
LOG_BUCKET=aws-oci-focus-daily
9+
LOG_PREFIX=transfer-logs
10+
ENABLE_S3_AUDIT_LOG=true
11+
OCI_REGION=eu-frankfurt-1
12+
OCI_NAMESPACE=<namespace>
13+
OCI_BUCKET=<bucket>
14+
LOG_LEVEL=INFO

0 commit comments

Comments
 (0)