Skip to content

README.md

Latest commit

 

History

History
137 lines (108 loc) · 5.35 KB

README.md

File metadata and controls

137 lines (108 loc) · 5.35 KB

Observability Stack AWS CLI

Deploy the Observability Stack on AWS managed services with a single command. Creates an OpenSearch domain (or serverless collection), OSIS ingestion pipeline, Amazon Managed Prometheus workspace, and a fully configured OpenSearch UI with dashboards — plus an EC2 instance running demo workloads that generate telemetry out of the box.

Quick Start

Managed domain (default):

npx @opensearch-project/observability-stack

Serverless collection (AOSS):

npx @opensearch-project/observability-stack --serverless

Takes ~15 minutes for managed, ~5 minutes for serverless. When complete, the CLI prints a dashboard URL — open it and you're in.

What Gets Created

Resource Description
OpenSearch domain or serverless collection Stores logs, traces, and service map data
OSIS pipeline Ingests OTLP data (logs, traces, metrics) via SigV4
Amazon Managed Prometheus Stores time-series metrics
Connected Data Source (Prometheus) Connects AMP to OpenSearch for metric queries
OpenSearch Application UI with workspace, index patterns, correlations, dashboards
IAM roles Pipeline role (OSIS → OpenSearch + AMP) and Connected Data Source role
EC2 instance (t3.xlarge) Runs OTel Demo + example agents (29 containers generating telemetry)

All resources are tagged with observability-stack:pipeline-name for identification and cleanup.

Usage

Create a managed domain deployment from scratch:

node bin/cli-installer.mjs --managed \
  --pipeline-name obs-stack-<your-alias> \
  --region us-east-1

Create a serverless (AOSS) deployment from scratch:

node bin/cli-installer.mjs --serverless \
  --pipeline-name obs-stack-<your-alias> \
  --region us-east-1

Reuse existing OpenSearch domain / AMP workspace:

node bin/cli-installer.mjs --managed \
  --pipeline-name obs-stack-<your-alias> \
  --region us-east-1 \
  --opensearch-endpoint https://search-your-domain-xxx.us-east-1.es.amazonaws.com \
  --prometheus-url https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxx/api/v1/remote_write

Skip EC2 demo (just pipeline + UI, no demo workloads):

node bin/cli-installer.mjs --managed \
  --pipeline-name obs-stack-<your-alias> \
  --region us-east-1 \
  --skip-demo

Launch demo workload against an existing pipeline:

node bin/launch-demo.mjs \
  --pipeline-name obs-stack-<your-alias> \
  --region us-east-1

Interactive mode (TUI wizard):

node bin/cli-installer.mjs

Destroy

node bin/cli-installer.mjs destroy \
  --pipeline-name obs-stack-<your-alias> \
  --region us-east-1

Deletes: EC2 instance, OpenSearch Application, Connected Data Source, OSIS pipeline, IAM roles. OpenSearch domain and AMP workspace are preserved (shared resources).

Prerequisites

  • Node.js 18+
  • AWS credentials configured (aws sts get-caller-identity should succeed)
  • IAM permissions: OSIS, OpenSearch, AMP, IAM role creation, EC2, SSM

Known Limitations

  • Index pattern fields need manual refresh — After data starts flowing, go to Management → Index Patterns → select pattern → click 🔄 to pick up new fields.
  • Demo data takes 10-15 minutes — The EC2 instance needs time to bootstrap Docker, pull images, and start sending telemetry.
  • Idempotent but not updateable — Running twice safely no-ops, but won't update existing resources with new config.

Development

Repository Layout

aws/cli-installer/
├── bin/
│   ├── cli-installer.mjs         # Entry point (full pipeline)
│   └── launch-demo.mjs           # Standalone EC2 demo launcher against an existing pipeline
├── src/
│   ├── main.mjs                # CLI orchestration + executePipeline flow
│   ├── cli.mjs                 # Argument parsing + config
│   ├── aws.mjs                 # AWS resource creation (IAM, OSIS, Connected Data Source, Application)
│   ├── render.mjs              # OSIS pipeline YAML generation
│   ├── opensearch-ui-init.mjs  # OpenSearch UI setup (SigV4, workspace, dashboards)
│   ├── ec2-demo.mjs            # EC2 demo workload launcher
│   ├── destroy.mjs             # Resource teardown
│   ├── interactive.mjs         # TUI wizard
│   ├── config.mjs              # Defaults
│   ├── ui.mjs                  # Terminal UI helpers (spinners, boxes, theme)
│   ├── arch-image.mjs          # Base64 architecture image for dashboard
│   └── commands/               # REPL commands (create, list, describe)
└── test/
    └── unit.test.mjs           # Unit tests (16 tests, 6 suites)

Running Tests

cd aws/cli-installer
node --test test/unit.test.mjs

Key Patterns

  • SigV4 signing — opensearch-ui-init.mjs uses @aws-sdk/signature-v4 with service opensearch. Query params must be in the query property of HttpRequest, not embedded in the path.
  • Idempotency — Every resource creation checks for existence first. Correlations use find-before-create; saved objects with fixed IDs are upserted.
  • EC2 demo — User data script installs Docker + Compose, clones the repo, writes a managed-mode collector config, and starts workload services via docker-compose.managed.yml.