Skip to content

monitoring-setup.md

Latest commit

 

History

History
381 lines (256 loc) · 9.12 KB

monitoring-setup.md

File metadata and controls

381 lines (256 loc) · 9.12 KB

Monitoring Stack — Setup Guide

This guide walks you through deploying the KMS observability stack step by step. For a full reference of OTLP options and pipeline internals, see → KMS Telemetry

Table of contents

  1. Download
  2. Prerequisites
  3. Local mode — KMS included
  4. External mode — existing KMS
  5. Explore Grafana
  6. Troubleshooting

Download

The monitoring stack is distributed as a standalone archive containing the Docker Compose file, configuration templates, and helper scripts.

Download the archive from the KMS GitHub Releases page (cosmian-kms-monitoring.zip), or use wget with the version you are deploying:

wget https://package.cosmian.com/kms/<version>/monitoring/cosmian-kms-monitoring.zip
unzip cosmian-kms-monitoring.zip
cd monitoring/

Replace <version> with the KMS version you are deploying (e.g. 5.23.0). The list of available versions is at https://package.cosmian.com/kms/.

Alternatively, clone the KMS repository and navigate to the monitoring/ directory.

Prerequisites

Before getting started, make sure the following tools are installed on your machine.

Required

  • Docker Engine ≥ 20.10 Compose profiles (COMPOSE_PROFILES) require Docker Compose v2 and Docker Engine 20.10+. Verify your version:

    docker --version
docker compose version

  • Docker Compose V2 (included in Docker Desktop 3.4+) The stack uses the docker compose command (with a space), not the legacy docker-compose.

Local mode only

  • openssl — used by generate-demo-cert.sh to generate the demo TLS certificate. Verify it is available:

    openssl version

    On macOS it is pre-installed. On Linux: apt install openssl or dnf install openssl.

External mode only

  • Your existing KMS must be configured to export telemetry via OTLP to the collector:

    # gRPC (recommended)
KMS_OTLP_URL=http://<collector-host>:4317
# or HTTP
KMS_OTLP_URL=http://<collector-host>:4318

    See → KMS Telemetry — OTLP configuration

Ports availability

The following ports must be free on the host before starting the stack:

Port Service
3000 Grafana
4317 OTel Collector (gRPC)
4318 OTel Collector (HTTP)
8428 VictoriaMetrics
9998 KMS (local mode only)

Local mode — KMS included

In this mode, the full stack starts together: KMS, OTel Collector, VictoriaMetrics, and Grafana. Use this mode for local development or to evaluate the stack end-to-end.

All files referenced below (docker-compose.yml, .env, generate-demo-cert.sh) are located in the monitoring/ directory of the KMS repository. Navigate there first:

cd monitoring/

1. Configure .env

Open the .env file — this is the only file you need to edit.

Make sure the following values are set:

# ── KMS Mode / Docker profiles ───────────────────────────────────
COMPOSE_PROFILES=kms-local
KMS_MODE=local

# ── TLS certificate password (used by generate-demo-cert.sh) ─────
KMS_P12_PASSWORD=testpassword

# ── KMS topology metadata ────────────────────────────────────────
KMS_CLUSTER=cosmian-kms-local

# ── Metadata for metrics/traces ──────────────────────────────────
KMS_VERSION=latest
ENVIRONMENT=demo

# ── Grafana ──────────────────────────────────────────────────────
GRAFANA_ADMIN_USER=demo
GRAFANA_ADMIN_PASSWORD=demo

# ── Metrics retention (VictoriaMetrics, in months) ───────────────
METRICS_RETENTION_MONTHS=12

✏️ Screenshot placeholder — .env file open in editor, COMPOSE_PROFILES and KMS_MODE highlighted.

2. Generate the demo TLS certificate

The KMS container requires a PKCS#12 certificate to serve its API over TLS. The provided script generates a self-signed certificate valid for 10 years, using the password defined in KMS_P12_PASSWORD:

bash generate-demo-cert.sh

Expected output:

Generating a 4096 bit RSA private key
.............................................++++
.............................................++++
.............................................++++
writing new private key to './certs/kms-demo.key'
-----
PKCS#12 demo certificate generated at: ./certs/kms.p12
Password: testpassword

monitor00

⚠️ This certificate is for demo purposes only. Do not use it in production.

3. Start the stack

docker compose up -d

Docker will pull the required images and start all containers. On first run, this may take a minute depending on your network speed.

monitor01

4. Verify the stack is healthy

docker compose ps

All containers should show Up or Up (healthy):

NAME                STATUS
kms                 Up (healthy)
otel-collector      Up
victoria-metrics    Up
grafana             Up (healthy)

You can also check the OTel Collector health endpoint:

curl http://localhost:13133

Expected response: {"status":"Server available"}

monitor02

External mode — existing KMS

Use this mode when you already have a running KMS instance and only want to attach the observability stack to it.

All files referenced below are located in the monitoring/ directory of the KMS repository. Navigate there first:

cd monitoring/

1. Configure .env

Only a few lines change compared to local mode:

-COMPOSE_PROFILES=kms-local
-KMS_MODE=local
+COMPOSE_PROFILES=
+KMS_MODE=external

-# KMS_P12_PASSWORD is not needed in external mode
-KMS_P12_PASSWORD=testpassword

 KMS_CLUSTER=cosmian-kms-local
 KMS_VERSION=latest
-ENVIRONMENT=demo
+ENVIRONMENT=production

 GRAFANA_ADMIN_USER=demo
 GRAFANA_ADMIN_PASSWORD=demo
 METRICS_RETENTION_MONTHS=12

monitor03

2. Point your KMS to the collector

On your existing KMS, enable OTLP export toward the collector host:

# via environment variable
KMS_OTLP_URL=http://<collector-host>:4317

# or via CLI argument
./cosmian_kms_server --otlp http://<collector-host>:4317 --enable-metering

If the KMS runs on the same machine as the stack, use http://localhost:4317.

3. Start the stack (without KMS)

docker compose up -d

With COMPOSE_PROFILES left empty, Docker Compose will start only OTel Collector, VictoriaMetrics, and Grafana — the KMS container is skipped.

4. Verify the stack is healthy

docker compose ps

Expected output (no kms container):

NAME                STATUS
otel-collector      Up
victoria-metrics    Up
grafana             Up (healthy)

Verify the collector is receiving data from your KMS:

# Check that otelcol_receiver_accepted_spans is incrementing
curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted_spans

Or

monitor04

Explore Grafana

Once the stack is running, open Grafana in your browser:

http://localhost:3000

Login with:

Field Value
Username value of GRAFANA_ADMIN_USER in .env (default: demo)
Password value of GRAFANA_ADMIN_PASSWORD in .env (default: demo)

monitor05

Pre-provisioned dashboards are available under Dashboards → Browse:

monitor06

After few seconds metrics will be available

monitor07

Troubleshooting

Containers not starting

docker compose logs -f otel-collector
docker compose logs -f kms          # local mode only

No data in Grafana dashboards

  1. Confirm the OTel Collector is up:

    curl http://localhost:13133

  2. Confirm VictoriaMetrics is receiving data:

    curl http://localhost:8428/health

  3. Check that your KMS is sending OTLP — look for incoming spans in collector logs:

    docker compose logs otel-collector | grep "traces"

KMS container fails to start (local mode)

Verify the certificate was generated:

ls -la .certs/kms.p12

If missing, re-run:

bash generate-demo-cert.sh

Port conflict

If a port is already in use, identify the process:

# macOS / Linux
lsof -i :<port>

# Windows
netstat -ano | findstr :<port>

Then either stop the conflicting process or edit the port mapping in docker-compose.yml.

For a complete reference of all OTLP options, log levels, and pipeline internals, see → KMS Telemetry