Elk (#612)

* Add ELK compose example with automated Kibana dashboard setup

Author: frjcomp

docs/guides/gitlab.md

@@ -49,7 +49,7 @@ pipeleek gl vuln -g https://leakycompany.com -t glpat-[redacted]
 2024-11-14T14:29:05+01:00 info Fetching CVEs for this version version=17.5.1-ee
 ```
 
-## Enumerating CI/CD Variables And Secure Files
+## Enumerating Accessible Secrets
 
 If you already have access to projects and groups you can try to enumerate CI/CD variables and use these for potential privilege escalation/lateral movement paths.
 
@@ -63,10 +63,10 @@ pipeleek gl variables -g https://leakycompany.com -t glpat-[redacted]
 pipeleek gl schedule -g https://leakycompany.com -t glpat-[redacted]
 
 # Secure files are an alternative to variables and often times contain sensitive info
-pipeleek gl secureFiles  --gitlab https://leakycompany.com --token glpat-[redacted]
-2024-11-18T15:38:08Z info Fetching project variables
-2024-11-18T15:38:09Z warn Secure file content="this is a secure file!!" downloadUrl=https://leakycompany.com/api/v4/projects/60367314/secure_files/9149327/download
-2024-11-18T15:38:12Z info Fetched all secure files
+pipeleek gl secureFiles -g https://leakycompany.com -t glpat-[redacted]
+
+# Terraform states can contain secrets
+pipeleek gl tf --token -g https://leakycompany.com -t glpat-[redacted]
 ```
 
 ## Secret Detection in Source Code
@@ -79,7 +79,7 @@ Use Trufflehog to find hardcoded secrets in the source code:
 trufflehog gitlab --token=glpat-[redacted]
 ```
 
-Note: this only scanned repository you have access to. You can specify single repositories as well.
+Note: this only scans repositories you have access to. You can specify single repositories as well.
 
 ## Secret Detection in Pipelines And Artifacts
 
@@ -166,7 +166,7 @@ Review the findings manually and tweak the flags according to your needs.
 
 If you found any valid credentials, e.g. personal access tokens, cloud credentials, and so on, check if you can move laterally or escalate privileges.
 
-### An example of privilege escalation
+### An example of lateral movement
 
 Pipeleek identified the following based64 encode secret in the environment variable `CI_REPO_TOKEN`:
 
@@ -209,11 +209,11 @@ pipeleek gl enum -g https://gitlab.example.com -t glpat-[redacted]
 2025-09-29T12:25:52Z info Done
 ```
 
-Abusing this access token grants you access to the `another-project` repository, thus you escalated privileges to this repository.
+Abusing this access token grants you access to the `another-project` repository.
 
 ## Attacking Runners
 
-Chances are high that if pipelines are used, custom runners are registered. These come in different flavors. Most of the time the docker executor is used, which allows pipelines to define container images in which their commands are executed. For a full list of possibilities [rtfm](https://docs.gitlab.example.com/runner/executors/).
+Chances are high that if pipelines are used, custom runners are registered. These come in different flavors. Often times a container / docker based executor is used, which allows pipelines to define container images in which their commands are executed. For a full list of possibilities [rtfm](https://docs.gitlab.example.com/runner/executors/).
 
 If you can create projects or contribute to existing ones, you can interact with runners. We want to test if it is possible to escape from the runner context e.g. escape from the container to the host machine or if the runner leaks additional privileges e.g. in the form of attached files or environment variables set by the runner config.
 
@@ -231,7 +231,6 @@ $ pipeleek gl runners --token glpat-[redacted] --gitlab https://gitlab.example.c
 2024-09-26T14:26:55+02:00 info group runner description=blue-3.saas-linux-large-amd64.runners-manager.gitlab.example.com/default name=comp-test-ia paused=false runner=gitlab-runner tags=saas-linux-large-amd64 type=instance_type
 2024-09-26T14:26:55+02:00 info group runner description=green-1.saas-linux-2xlarge-amd64.runners-manager.gitlab.example.com/default name=comp-test-ia paused=false runner= tags=saas-linux-2xlarge-amd64 type=instance_type
 2024-09-26T14:26:55+02:00 info Unique runner tags tags=gitlab-org,saas-linux-large-arm64,windows,gitlab-org-docker,e2e-runner2,saas-macos-large-m2pro,saas-linux-xlarge-amd64,saas-linux-small-amd64,saas-linux-2xlarge-amd64,saas-linux-medium-amd64,saas-windows-medium-amd64,e2e-runner3,saas-linux-medium-arm64,saas-linux-medium-amd64-gpu-standard,saas-macos-medium-m1,shared-windows,saas-linux-large-amd64,windows-1809
-2024-09-26T14:26:55+02:00 info Done, Bye Bye 🏳️‍🌈🔥
 ```
 
 Review the runners and select the interesting ones. The Gitlab Ci/CD config file allows you to select runners by their tags. Thus we create a list of the most interesting tags, printed by the command above.

docs/introduction/configuration.md

@@ -222,7 +222,8 @@ circle:
     project: [my-org/my-repo] # circle scan --project (optional if org is set)
     vcs: github # circle scan --vcs
     org: my-org # circle scan --org (also enables org-wide discovery when project is omitted)
-    # --org accepts: my-org, github/my-org, or app URL forms like
+    # --org accepts: my-org, github/my-org, circleci/my-org (required for native
+    # CircleCI orgs), or app URL forms like
     # https://app.circleci.com/pipelines/github/my-org/my-repo
     # Note: org-wide discovery requires token visibility to that org. If not,
     # use explicit --project selectors instead.

docs/introduction/elk-dashboard.png

@@ -112,3 +112,28 @@ pipeleek gl scan --token glpat-[redacted] --gitlab https://gitlab.example.com  -
 ```
 
 Using Kibana you can filter for interesting messages, based on the JSON attributes of the output.
+
+### ELK Docker Compose Example (Shared Config + One-Shot Jobs)
+
+A Docker Compose example is available in [examples/compose-elk](https://github.com/CompassSecurity/pipeleek/tree/main/examples/compose-elk).
+
+It includes:
+
+- Elasticsearch storage and Kibana visualization
+- One shared Pipeleek config file mounted read-only into all scanner containers
+- One-shot scanner services for all scan commands (`gl`, `gh`, `bb`, `ad`, `gitea`, `jenkins`)
+- Logstash TCP ingestion for Pipeleek JSON lines
+
+Quick start:
+
+```bash
+git clone https://github.com/CompassSecurity/pipeleek.git
+cd pipeleek/examples/compose-elk
+bash start-elk.sh
+docker compose --profile gitlab run --rm scan-gitlab
+```
+
+Then open Kibana at `http://localhost:5601` and visit the dashboard.
+
+Example after running some scans for a while.
+![ELK dashboard](elk-dashboard.png)

docs/introduction/logging.md

@@ -0,0 +1,116 @@
+# Pipeleek Compose Example: Shared Config + ELK
+
+This example runs Pipeleek scan jobs as one-shot containers and forwards all JSON logs into Elasticsearch via Logstash. Kibana is provided for querying and dashboards.
+
+## What this setup provides
+
+- One shared Pipeleek config file mounted read-only in every scan-job container.
+- One Elasticsearch instance.
+- One Kibana instance.
+- One Logstash instance that accepts newline-delimited JSON on TCP port 5000 and is tuned to 192MB heap for local/dev environments.
+- One service definition per scan job:
+  - `scan-gitlab`
+  - `scan-github`
+  - `scan-bitbucket`
+  - `scan-devops`
+  - `scan-gitea`
+  - `scan-jenkins`
+
+## 1) Prepare shared config
+
+Edit `pipeleek.shared.example.yaml` and fill all token/url placeholders.
+
+The file is mounted into each scanner as:
+
+- Container path: `/config/pipeleek.yaml`
+
+## 2) Start ELK services
+
+Use the automated startup script that starts ELK and configures Kibana automatically:
+
+```bash
+bash start-elk.sh
+```
+
+This script:
+
+1. Starts Elasticsearch, Logstash, and Kibana
+2. Installs an Elasticsearch index template optimized for Pipeleek scan logs
+3. Waits for Kibana to be ready
+4. Automatically creates and configures the Kibana data view for pipeleek logs
+
+**Manual alternative**:
+
+```bash
+docker compose up -d elasticsearch logstash kibana
+```
+
+Then manually setup the index template and Kibana data view:
+
+```bash
+bash setup-elasticsearch.sh
+bash setup-kibana.sh
+```
+
+## 3) Run one-shot scan jobs
+
+Run any individual scan job with:
+
+```bash
+docker compose --profile gitlab run --rm scan-gitlab
+docker compose --profile github run --rm scan-github
+docker compose --profile bitbucket run --rm scan-bitbucket
+docker compose --profile devops run --rm scan-devops
+docker compose --profile gitea run --rm scan-gitea
+docker compose --profile jenkins run --rm scan-jenkins
+```
+
+Each job:
+
+1. Runs its own scan command using the shared config.
+2. Streams each JSON line to Logstash over TCP.
+3. Prints periodic progress updates every 30s to the container stderr so you can see it is still running.
+
+## 4) View data in Kibana
+
+Once you've run `bash start-elk.sh`, the Kibana data view is automatically configured. Simply:
+
+1. Open http://localhost:5601
+2. Open **Dashboards** and select `Pipeleek - Hit Monitoring`
+3. You will see:
+
+- `Pipeleek - High Hits` (count of `scan.confidence: high` hits)
+- `Pipeleek - High-Verified Hits` (count of `scan.confidence: high-verified` hits)
+- `Pipeleek - Log Intake Over Time` (bar chart over the last 4 hours with fixed 10-minute buckets)
+- `Pipeleek - Hits by Asset Type` (pie chart, for example hit logs vs hit artifacts)
+- `Pipeleek - Hits by Confidence` (pie chart of confidence ratings)
+- `Pipeleek - Last 100 Scan Entries` table (hit-level events only)
+- `Pipeleek - Latest 100 Log Entries` table (sorted by newest first, no fixed filter)
+
+4. Use the dashboard filter bar to filter by fields like `scan.platform` and `scan.confidence`
+
+Hit-focused panels use hit events (`level=hit`). If no hits are present in the selected time range, those panels will show no results.
+
+You can also use **Discover** with the same data view for ad-hoc investigation.
+
+For better searchability, logs are also normalized into common fields:
+
+- `scan.platform`
+- `scan.level`
+- `scan.confidence`
+- `scan.job_name`
+- `scan.target_url`
+- `scan.asset_type`
+- `scan.is_hit`
+- `secret.rule_name`
+- `secret.is_present`
+- `event.kind`
+- `event.category`
+- `event.type`
+- `event.outcome`
+
+**If you manually started ELK**, run the setup script:
+
+```bash
+bash setup-kibana.sh
+```