Skip to content

Commit 513e030

Browse files
niesfischniesfisch
committed
clickhouse + grafana + vector
1 parent db2fe99 commit 513e030

File tree

9 files changed

+659
-0
lines changed

9 files changed

+659
-0
lines changed

‎README.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ If you work in a legacy app and ask things like "Can we remove this?" or "Is thi
1010
- [What JCT Does at Runtime](#what-jct-does-at-runtime)
1111
- [Quick Start in 3 Steps](#quick-start-in-3-steps)
1212
- [Recommended Local Workflow: ELK](#recommended-local-workflow-elk)
13+
- [Alternative Stack: ClickHouse + Vector + Grafana](#alternative-stack-clickhouse--vector--grafana)
14+
- [ELK vs. ClickHouse — When to Choose Which](#elk-vs-clickhouse--when-to-choose-which)
1315
- [Project Status](#project-status)
1416
- [Java Version](#java-version)
1517
- [Build](#build)
@@ -92,6 +94,49 @@ This gives you a fast feedback loop: run traffic, query traces, validate code pa
9294
Start here:
9395

9496
- [README-ELK.md](README-ELK.md)
97+
- [doc/README-ClickHouse-Grafana.md](doc/README-ClickHouse-Grafana.md)
98+
99+
## Alternative Stack: ClickHouse + Vector + Grafana
100+
101+
If you want stronger analytics, faster aggregations over large event volumes, or a lighter-weight alternative to Elasticsearch, JCT also ships a second Docker Compose stack based on ClickHouse + Vector + Grafana.
102+
103+
```
104+
-javaagent:jct.jar
105+
+-------------------+
106+
| App + JCT Agent | /tmp/stacks/jct_*.log
107+
| File Processor |--+
108+
+-------------------+ |
109+
| Vector (tail + forward)
110+
+--> ClickHouse: jct_raw
111+
|
112+
Materialized View
113+
|
114+
jct_events (parsed)
115+
|
116+
Grafana :5601
117+
(pre-built dashboard)
118+
```
119+
120+
Start here: [doc/README-ClickHouse-Grafana.md](doc/README-ClickHouse-Grafana.md)
121+
122+
## ELK vs. ClickHouse — When to Choose Which
123+
124+
Both stacks run locally via `docker compose` and need no cloud setup.
125+
126+
| Criterion | ELK (Elastic + Kibana) | ClickHouse + Grafana |
127+
|---|---|---|
128+
| **Setup effort** | Medium — three containers, index pattern setup in Kibana UI | Medium — three containers, datasource and dashboard auto-provisioned |
129+
| **Query style** | KQL / Lucene (full-text search focused) | SQL (aggregation and analytics focused) |
130+
| **Best for** | Searching for specific stack occurrences, filtering by text | Aggregating, counting, trending over high volumes |
131+
| **Event volume** | Good up to low millions; indexing is memory-heavy | Excellent for large volumes; columnar storage compresses well |
132+
| **Ad-hoc exploration** | Kibana Discover is fast for browsing raw events | ClickHouse Play UI or Grafana Explore for SQL queries |
133+
| **Dashboard UX** | Kibana Lens (good) | Grafana (good, pre-built panels included) |
134+
| **Transport** | UDP or TCP → Logstash | File processor → Vector tails log files |
135+
| **When to pick** | You are already familiar with Kibana, or want full-text search | You want SQL analytics, hot-frame counts, or handle high traffic |
136+
137+
**Rule of thumb:**
138+
- Not sure which to pick? Start with **ELK** — Kibana's Discover view is the fastest way to browse raw stacks.
139+
- Running a busy system or want to aggregate across thousands of events? Use **ClickHouse + Grafana** — SQL `GROUP BY class_name` queries are instant even on millions of rows.
95140

96141
## Project Status
97142

‎doc/README-ClickHouse-Grafana.md‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
# JCT with ClickHouse + Vector + Grafana
2+
3+
This is an alternative to ELK for collecting, mining, and visualizing JCT stack events.
4+
5+
## Architecture
6+
7+
```
8+
JCT agent --> /tmp/stacks/jct_*.log
9+
|
10+
Vector (tail + rename field)
11+
|
12+
ClickHouse: default.jct_raw (raw ingest table)
13+
|
14+
ClickHouse: default.jct_events_mv (materialized view)
15+
|
16+
ClickHouse: default.jct_events (parsed, queryable)
17+
|
18+
Grafana dashboards
19+
```
20+
21+
JSON parsing happens entirely in ClickHouse via a materialized view — Vector only forwards raw lines.
22+
This avoids VRL type-system complexity and keeps the Vector config minimal.
23+
24+
## Start the stack
25+
26+
```zsh
27+
cd /home/msauer/dev/workspace-private/java-code-tracer
28+
docker compose -f docker-compose-clickhouse.yml up -d
29+
```
30+
31+
Grafana:
32+
33+
- URL: `http://localhost:5601`
34+
- User: `admin`
35+
- Password: `admin`
36+
37+
Dashboard provisioning:
38+
39+
- Folder: `JCT`
40+
- Dashboard: `JCT Overview`
41+
- Panels:
42+
- Throughput by Minute
43+
- Top Classes
44+
- Top Methods
45+
46+
If Grafana was already running before these files were added, restart the stack once:
47+
48+
```zsh
49+
cd /home/msauer/dev/workspace-private/java-code-tracer
50+
docker compose -f docker-compose-clickhouse.yml down
51+
docker compose -f docker-compose-clickhouse.yml up -d
52+
```
53+
54+
## Stop and reset
55+
56+
```zsh
57+
cd /home/msauer/dev/workspace-private/java-code-tracer
58+
docker compose -f docker-compose-clickhouse.yml down
59+
docker volume rm java-code-tracer_clickhouse-data java-code-tracer_grafana-data
60+
```
61+
62+
## Run your app with JCT (file processor)
63+
64+
Use a file-based config such as `doc/config-sample-application-file.yaml` and write to `/tmp/stacks`.
65+
66+
## Verify ingestion quickly
67+
68+
```zsh
69+
curl -s "http://localhost:8123/?query=SELECT%20count(*)%20FROM%20default.jct_events"
70+
```
71+
72+
Verify dashboard data:
73+
74+
1. Open `http://localhost:5601`.
75+
2. Go to `Dashboards` and open `JCT / JCT Overview`.
76+
3. Select time range `Last 15 minutes` and confirm panels are populated.
77+
78+
## Useful ClickHouse queries
79+
80+
Run queries directly:
81+
82+
```zsh
83+
curl -s "http://localhost:8123/?query=<URL-encoded SQL>"
84+
```
85+
86+
Or open the ClickHouse Play UI at `http://localhost:8123/play`.
87+
88+
Top classes in the last 15 minutes:
89+
90+
```sql
91+
SELECT class_name, count(*) AS hits
92+
FROM default.jct_events
93+
WHERE ingest_time >= now() - INTERVAL 15 MINUTE
94+
GROUP BY class_name
95+
ORDER BY hits DESC
96+
LIMIT 30;
97+
```
98+
99+
Top methods in the last 15 minutes:
100+
101+
```sql
102+
SELECT method_name, count(*) AS hits
103+
FROM default.jct_events
104+
WHERE ingest_time >= now() - INTERVAL 15 MINUTE
105+
GROUP BY method_name
106+
ORDER BY hits DESC
107+
LIMIT 30;
108+
```
109+
110+
Call frequency over time (1-minute buckets):
111+
112+
```sql
113+
SELECT toStartOfMinute(ingest_time) AS minute, count(*) AS hits
114+
FROM default.jct_events
115+
WHERE ingest_time >= now() - INTERVAL 60 MINUTE
116+
GROUP BY minute
117+
ORDER BY minute;
118+
```
119+
120+
Filter by class prefix:
121+
122+
```sql
123+
SELECT ingest_time, class_name, method_name, stack_depth, stack
124+
FROM default.jct_events
125+
WHERE class_name LIKE 'de.marcelsauer.%'
126+
ORDER BY ingest_time DESC
127+
LIMIT 200;
128+
```
129+
130+
Explode frames to count hot frames across all stacks:
131+
132+
```sql
133+
SELECT frame, count(*) AS hits
134+
FROM default.jct_events
135+
ARRAY JOIN stack AS frame
136+
WHERE ingest_time >= now() - INTERVAL 15 MINUTE
137+
GROUP BY frame
138+
ORDER BY hits DESC
139+
LIMIT 50;
140+
```
141+
142+
Check raw ingestion count:
143+
144+
```sql
145+
SELECT count(*) FROM default.jct_raw;
146+
SELECT count(*) FROM default.jct_events;
147+
```
148+
149+
## Troubleshooting
150+
151+
**Dashboard is empty:**
152+
- Check the Grafana time picker — set it to `Last 15 minutes` or wider.
153+
- Verify events exist: `SELECT count(*) FROM default.jct_events WHERE ingest_time >= now() - INTERVAL 15 MINUTE`
154+
- Confirm Vector is running: `docker compose -f docker-compose-clickhouse.yml ps`
155+
156+
**Vector exits on startup:**
157+
- Check logs: `docker compose -f docker-compose-clickhouse.yml logs vector`
158+
- Confirm `/tmp/stacks` contains `jct_*.log` files.
159+
160+
**ClickHouse authentication errors (403/516):**
161+
- Ensure `doc/clickhouse/config/users.d/default-user.xml` is present and mounted.
162+
- Recreate the stack: `docker compose -f docker-compose-clickhouse.yml down -v && docker compose -f docker-compose-clickhouse.yml up -d`
163+

‎doc/clickhouse/config/users.d/default-user.xml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
<?xml version="1.0"?>
2+
<!-- Allow the default user without a password for local dev/tracing use. -->
3+
<clickhouse>
4+
<users>
5+
<default>
6+
<password></password>
7+
<networks>
8+
<ip>::/0</ip>
9+
</networks>
10+
<profile>default</profile>
11+
<quota>default</quota>
12+
</default>
13+
</users>
14+
</clickhouse>
15+

‎doc/clickhouse/init/01_jct_events.sql‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
-- Raw ingest table: Vector writes here.
2+
-- Keeping this simple avoids any JSON parsing in Vector.
3+
CREATE TABLE IF NOT EXISTS default.jct_raw
4+
(
5+
ingest_time DateTime DEFAULT now(),
6+
raw_json String
7+
)
8+
ENGINE = MergeTree
9+
PARTITION BY toDate(ingest_time)
10+
ORDER BY ingest_time
11+
SETTINGS index_granularity = 8192;
12+
13+
-- Parsed events table: populated by the materialized view below.
14+
-- All JSON extraction happens in ClickHouse at insert time.
15+
CREATE TABLE IF NOT EXISTS default.jct_events
16+
(
17+
ingest_time DateTime,
18+
timestamp_millis UInt64,
19+
stack Array(String),
20+
stack_depth UInt16,
21+
top_frame String,
22+
class_name String,
23+
method_name String,
24+
raw_json String
25+
)
26+
ENGINE = MergeTree
27+
PARTITION BY toDate(ingest_time)
28+
ORDER BY (ingest_time, class_name, method_name)
29+
SETTINGS index_granularity = 8192;
30+
31+
-- Materialized view: transforms raw JSON into typed columns on every insert.
32+
CREATE MATERIALIZED VIEW IF NOT EXISTS default.jct_events_mv
33+
TO default.jct_events
34+
AS SELECT
35+
ingest_time,
36+
toUInt64OrDefault(JSONExtractString(raw_json, 'timestampMillis'), toUInt64(0)) AS timestamp_millis,
37+
JSONExtract(raw_json, 'stack', 'Array(String)') AS stack,
38+
toUInt16(length(JSONExtract(raw_json, 'stack', 'Array(String)'))) AS stack_depth,
39+
arrayElement(JSONExtract(raw_json, 'stack', 'Array(String)'), 1) AS top_frame,
40+
replaceRegexpOne(
41+
arrayElement(JSONExtract(raw_json, 'stack', 'Array(String)'), 1),
42+
'^(.+)\\.[^.(]+\\(.*\\)$', '\\1') AS class_name,
43+
replaceRegexpOne(
44+
arrayElement(JSONExtract(raw_json, 'stack', 'Array(String)'), 1),
45+
'^.+\\.([^.(]+\\(.*\\))$', '\\1') AS method_name,
46+
raw_json
47+
FROM default.jct_raw;
48+

‎doc/grafana/provisioning/dashboards/dashboards.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: 1
2+
3+
providers:
4+
- name: JCT Dashboards
5+
orgId: 1
6+
folder: JCT
7+
type: file
8+
disableDeletion: false
9+
editable: true
10+
options:
11+
path: /etc/grafana/provisioning/dashboards
12+

0 commit comments

Comments
 (0)