Skip to content

Commit 94d80aa

Browse files
committed
feat: last copy pipe executions monitoring endpoint
1 parent dfdea8c commit 94d80aa

1 file changed

Lines changed: 51 additions & 0 deletions

File tree

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
DESCRIPTION >
2+
Monitors Tinybird copy job executions and exposes their status as Prometheus gauge metrics. Consumed by datadog scraper (`dd-tinybird-copy-pipe-executions-scraper`)
3+
4+
Tracks the latest execution status for each copy pipe (starting today) and generates one-hot encoded metrics where each pipe gets a value of `1` for its current status (`'ok'`, `'error'`, `'cancelled'`, `'queued'`, `'working'`) and `0` for all other statuses.
5+
6+
**Metric:** `copy_pipes_latest_execution_status` (gauge)
7+
**Labels:** `pipe_name`, `status`
8+
9+
**Note:** The `'queued'` status is virtual—it's derived by detecting the error message `"You have reached the maximum number of copy jobs"` rather than being a native Tinybird status. This is because tinybird retries these again when possible but returns an error status. Similarly, `'ok'` is mapped from the native `'done'` status for datadog color-scheme convention.
10+
11+
12+
TAGS "Monitoring"
13+
14+
NODE copy_pipes_latest_executions
15+
SQL >
16+
17+
SELECT
18+
JSONExtract(job_metadata, 'pipe_name', 'String') AS pipe_name,
19+
multiIf(
20+
error LIKE 'You have reached the maximum number of copy jobs%',
21+
'queued',
22+
status = 'done',
23+
'ok',
24+
status
25+
) AS status,
26+
error,
27+
started_at
28+
FROM tinybird.jobs_log
29+
WHERE
30+
job_type = 'copy' AND started_at > toStartOfDay(now()) and pipe_name <> 'members_with_location'
31+
ORDER BY pipe_id, created_at DESC
32+
LIMIT 1 BY pipe_id
33+
34+
35+
36+
NODE errored_copy_pipes_latest_execution
37+
SQL >
38+
39+
WITH
40+
possible_statuses AS (SELECT array('ok', 'error', 'cancelled', 'queued', 'working') AS statuses)
41+
SELECT
42+
'copy_pipes_latest_execution_status' AS name,
43+
IF(s = e.status, 1, 0) AS value,
44+
'Latest execution status per pipe (one-hot: 1 active, 0 otherwise)' AS help,
45+
'gauge' AS type,
46+
map('pipe_name', e.pipe_name, 'status', s) AS labels
47+
FROM copy_pipes_latest_executions AS e
48+
CROSS JOIN possible_statuses ps ARRAY
49+
JOIN ps.statuses AS s
50+
51+

0 commit comments

Comments
 (0)