Skip to content

Commit bdeb596

Browse files
gaspergromgaspergrom
committed
feat: add ai tracker endpoints
Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
1 parent 8a05d36 commit bdeb596

4 files changed

Lines changed: 212 additions & 0 deletions

File tree

‎services/libs/tinybird/datasources/ai_code_tracker_ds.datasource‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
DESCRIPTION >
2+
- `ai_code_tracker_ds` contains pre-computed monthly aggregates of AI-assisted commits by tool.
3+
- Populated hourly by `ai_code_tracker_copy.pipe` which scans activities for AI tool signatures.
4+
- Each row represents one (month, toolKey) combination with commit counts.
5+
- Also stores total commits per month (toolKey = '__total__') for percentage calculations.
6+
- `monthStart` is the first day of the month (used for both monthly and yearly aggregation at query time).
7+
- `toolKey` identifies the AI tool (e.g., 'github-copilot', 'claude', 'cursor') or '__total__' for all commits.
8+
- `commitCount` is the number of commits for that tool in that month.
9+
10+
TAGS "Report"
11+
12+
SCHEMA >
13+
`monthStart` Date,
14+
`toolKey` LowCardinality(String),
15+
`commitCount` UInt64
16+
17+
ENGINE MergeTree
18+
ENGINE_PARTITION_KEY toYear(monthStart)
19+
ENGINE_SORTING_KEY monthStart, toolKey

‎services/libs/tinybird/pipes/ai_code_tracker.pipe‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
DESCRIPTION >
2+
- `ai_code_tracker.pipe` returns AI-assisted commit counts by tool and time period.
3+
- Reads from pre-computed `ai_code_tracker_ds` datasource (materialized hourly by `ai_code_tracker_copy.pipe`).
4+
- Parameters:
5+
- `granularity`: Required. Either 'monthly' or 'yearly'.
6+
- `startDate`: Optional DateTime filter for commits after this date.
7+
- `endDate`: Optional DateTime filter for commits before this date.
8+
- Response: toolKey, toolName, startDate, endDate, commitCount
9+
10+
TAGS "Report"
11+
12+
NODE ai_code_tracker_result
13+
DESCRIPTION >
14+
Aggregate pre-computed AI commit counts by tool and time period
15+
SQL >
16+
%
17+
SELECT
18+
toolKey,
19+
multiIf(
20+
toolKey = 'github-copilot', 'GitHub Copilot',
21+
toolKey = 'chatgpt', 'ChatGPT',
22+
toolKey = 'claude', 'Claude',
23+
toolKey = 'cursor', 'Cursor',
24+
toolKey = 'codewhisperer', 'CodeWhisperer',
25+
toolKey = 'gemini', 'Gemini',
26+
toolKey = 'codeium', 'Codeium',
27+
toolKey = 'aider', 'Aider',
28+
toolKey = 'devin', 'Devin',
29+
toolKey = 'tabnine', 'Tabnine',
30+
toolKey = 'other', 'Other AI',
31+
'Unknown'
32+
) AS toolName,
33+
formatDateTime(
34+
CASE
35+
WHEN {{ String(granularity, description="Time aggregation: monthly or yearly", required=True) }} = 'monthly'
36+
THEN monthStart
37+
ELSE toStartOfYear(monthStart)
38+
END,
39+
'%Y-%m-%d'
40+
) AS startDate,
41+
formatDateTime(
42+
CASE
43+
WHEN {{ String(granularity, description="Time aggregation: monthly or yearly", required=True) }} = 'monthly'
44+
THEN monthStart + INTERVAL 1 MONTH - INTERVAL 1 DAY
45+
ELSE toStartOfYear(monthStart) + INTERVAL 1 YEAR - INTERVAL 1 DAY
46+
END,
47+
'%Y-%m-%d'
48+
) AS endDate,
49+
sum(commitCount) AS commitCount
50+
FROM ai_code_tracker_ds
51+
WHERE
52+
toolKey != '__total__'
53+
{% if defined(startDate) %}
54+
AND monthStart >= toDate({{ DateTime(startDate, description="Filter commits after this date", required=False) }})
55+
{% end %}
56+
{% if defined(endDate) %}
57+
AND monthStart < toDate({{ DateTime(endDate, description="Filter commits before this date", required=False) }})
58+
{% end %}
59+
GROUP BY toolKey, startDate, endDate
60+
ORDER BY startDate ASC, commitCount DESC
61+
62+
TYPE endpoint

‎services/libs/tinybird/pipes/ai_code_tracker_copy.pipe‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
DESCRIPTION >
2+
- `ai_code_tracker_copy.pipe` materializes AI-assisted commit counts into `ai_code_tracker_ds`.
3+
- Runs daily to scan `activities_deduplicated_ds` for commits matching AI tool patterns.
4+
- Classifies commits by AI tool based on title, body, and attributes content.
5+
- Stores monthly aggregates per tool plus total commits per month (toolKey = '__total__').
6+
- Uses multiSearchAnyCaseInsensitive for fast single-pass pre-filtering before classification.
7+
8+
TAGS "Report"
9+
10+
NODE ai_code_tracker_copy_totals
11+
DESCRIPTION >
12+
Total commits per month - no LIKE scanning needed, just a simple count
13+
SQL >
14+
SELECT
15+
toStartOfMonth(a.timestamp) AS monthStart,
16+
'__total__' AS toolKey,
17+
count() AS commitCount
18+
FROM activities_deduplicated_ds a
19+
WHERE a.type = 'authored-commit'
20+
GROUP BY monthStart
21+
22+
NODE ai_code_tracker_copy_prefilter
23+
DESCRIPTION >
24+
Fast pre-filter: only keep commits that contain ANY AI keyword in title, body, or attributes.
25+
multiSearchAnyCaseInsensitive does a single pass instead of dozens of separate LIKE checks.
26+
SQL >
27+
SELECT
28+
toStartOfMonth(a.timestamp) AS monthStart,
29+
a.title,
30+
a.body,
31+
a.attributes
32+
FROM activities_deduplicated_ds a
33+
WHERE
34+
a.type = 'authored-commit'
35+
AND (
36+
multiSearchAnyCaseInsensitive(a.title, ['copilot', 'chatgpt', 'claude', 'cursor', 'codewhisperer', 'gemini', 'codeium', 'aider', 'devin', 'tabnine', 'ai-generated', 'ai generated']) != 0
37+
OR multiSearchAnyCaseInsensitive(a.body, ['copilot', 'chatgpt', 'claude', 'cursor', 'codewhisperer', 'gemini', 'codeium', 'aider', 'devin', 'tabnine', 'ai-generated', 'ai generated', 'co-authored-by']) != 0
38+
OR multiSearchAnyCaseInsensitive(a.attributes, ['copilot', 'ai-generated']) != 0
39+
)
40+
41+
NODE ai_code_tracker_copy_classify
42+
DESCRIPTION >
43+
Classify pre-filtered commits by AI tool
44+
SQL >
45+
SELECT
46+
monthStart,
47+
multiIf(
48+
positionCaseInsensitive(title, 'github copilot') > 0 OR positionCaseInsensitive(body, 'co-authored-by') > 0 AND positionCaseInsensitive(body, 'copilot') > 0 OR positionCaseInsensitive(attributes, 'copilot') > 0, 'github-copilot',
49+
positionCaseInsensitive(title, 'cursor') > 0 OR positionCaseInsensitive(body, 'cursor') > 0, 'cursor',
50+
positionCaseInsensitive(title, 'claude') > 0 OR positionCaseInsensitive(body, 'claude') > 0, 'claude',
51+
positionCaseInsensitive(title, 'chatgpt') > 0 OR positionCaseInsensitive(body, 'chatgpt') > 0, 'chatgpt',
52+
positionCaseInsensitive(title, 'codewhisperer') > 0 OR positionCaseInsensitive(body, 'codewhisperer') > 0, 'codewhisperer',
53+
positionCaseInsensitive(title, 'gemini') > 0 OR positionCaseInsensitive(body, 'gemini') > 0, 'gemini',
54+
positionCaseInsensitive(title, 'codeium') > 0 OR positionCaseInsensitive(body, 'codeium') > 0, 'codeium',
55+
positionCaseInsensitive(title, 'copilot') > 0 OR positionCaseInsensitive(body, 'copilot') > 0, 'github-copilot',
56+
positionCaseInsensitive(title, 'aider') > 0 OR positionCaseInsensitive(body, 'aider') > 0, 'aider',
57+
positionCaseInsensitive(title, 'devin') > 0 OR positionCaseInsensitive(body, 'devin') > 0, 'devin',
58+
positionCaseInsensitive(title, 'tabnine') > 0 OR positionCaseInsensitive(body, 'tabnine') > 0, 'tabnine',
59+
positionCaseInsensitive(title, 'ai-generated') > 0 OR positionCaseInsensitive(title, 'ai generated') > 0
60+
OR positionCaseInsensitive(body, 'ai-generated') > 0 OR positionCaseInsensitive(body, 'ai generated') > 0
61+
OR positionCaseInsensitive(attributes, 'ai-generated') > 0
62+
OR (positionCaseInsensitive(body, 'co-authored-by') > 0 AND positionCaseInsensitive(body, 'bot') > 0), 'other',
63+
'__none__'
64+
) AS toolKey
65+
FROM ai_code_tracker_copy_prefilter
66+
67+
NODE ai_code_tracker_copy_by_tool
68+
DESCRIPTION >
69+
Aggregate AI commits by month and tool
70+
SQL >
71+
SELECT
72+
monthStart,
73+
toolKey,
74+
count() AS commitCount
75+
FROM ai_code_tracker_copy_classify
76+
WHERE toolKey != '__none__'
77+
GROUP BY monthStart, toolKey
78+
79+
NODE ai_code_tracker_copy_result
80+
DESCRIPTION >
81+
Union AI tool counts and total counts
82+
SQL >
83+
SELECT monthStart, toolKey, commitCount
84+
FROM ai_code_tracker_copy_by_tool
85+
UNION ALL
86+
SELECT monthStart, toolKey, commitCount
87+
FROM ai_code_tracker_copy_totals
88+
89+
TYPE COPY
90+
TARGET_DATASOURCE ai_code_tracker_ds
91+
COPY_MODE replace
92+
COPY_SCHEDULE 0 3 * * *

‎services/libs/tinybird/pipes/ai_code_tracker_total_commits.pipe‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
DESCRIPTION >
2+
- `ai_code_tracker_total_commits.pipe` returns total commit counts per time period.
3+
- Reads from pre-computed `ai_code_tracker_ds` datasource (materialized hourly by `ai_code_tracker_copy.pipe`).
4+
- Parameters:
5+
- `granularity`: Required. Either 'monthly' or 'yearly'.
6+
- `startDate`: Optional DateTime filter for commits after this date.
7+
- `endDate`: Optional DateTime filter for commits before this date.
8+
- Response: startDate, totalCommits
9+
10+
TAGS "Report"
11+
12+
NODE ai_code_tracker_total_commits_result
13+
DESCRIPTION >
14+
Aggregate pre-computed total commit counts by time period
15+
SQL >
16+
%
17+
SELECT
18+
formatDateTime(
19+
CASE
20+
WHEN {{ String(granularity, description="Time aggregation: monthly or yearly", required=True) }} = 'monthly'
21+
THEN monthStart
22+
ELSE toStartOfYear(monthStart)
23+
END,
24+
'%Y-%m-%d'
25+
) AS startDate,
26+
sum(commitCount) AS totalCommits
27+
FROM ai_code_tracker_ds
28+
WHERE
29+
toolKey = '__total__'
30+
{% if defined(startDate) %}
31+
AND monthStart >= toDate({{ DateTime(startDate, description="Filter commits after this date", required=False) }})
32+
{% end %}
33+
{% if defined(endDate) %}
34+
AND monthStart < toDate({{ DateTime(endDate, description="Filter commits before this date", required=False) }})
35+
{% end %}
36+
GROUP BY startDate
37+
ORDER BY startDate ASC
38+
39+
TYPE endpoint

0 commit comments

Comments
 (0)