Skip to content

Commit 09b3e85

Browse files
nicosuavenicosuave
authored
Merge pull request #17 from sidequery/product-analytics
Add Sidemantic product analytics
2 parents 9509cdf + e8dda81 commit 09b3e85

17 files changed

Lines changed: 6767 additions & 2304 deletions

‎Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ regex = "1"
3535
axum = { version = "0.7", default-features = false, features = ["json", "query", "tokio", "http1"] }
3636
dotenvy = "0.15"
3737
reqwest = { version = "0.12", features = ["json", "rustls-tls"] }
38-
tokio = { version = "1.37", features = ["macros", "rt-multi-thread", "process", "net", "time"] }
38+
tokio = { version = "1.37", features = ["macros", "rt-multi-thread", "process", "net", "time", "io-util", "sync"] }
3939
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
4040

4141
[dev-dependencies]

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ flowchart TB
7171
EventsPipeline --> EventsR2["R2 Data Catalog<br/>events table"]
7272
PersonsPipeline --> PersonsR2["R2 Data Catalog<br/>persons table"]
7373
74-
ReplayUI["Replay Explorer"] -->|"R2 SQL"| EventsR2
74+
WebUI["Hogflare UI"] -->|"R2 SQL"| EventsR2
7575
Models["Semantic Models<br/>models/*.yml"] --> EventsR2
7676
Models --> PersonsR2
7777
Consumers["DuckDB / R2 SQL / BI"] --> Models

‎docs/product-analytics.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Product Analytics
2+
3+
Hogflare serves product analytics from `/analytics`. The dashboard uses the repo's Sidemantic `events`, `pageviews`, and `sessions` models through the native Sidemantic/DuckDB Iceberg bridge.
4+
5+
## API Config
6+
7+
Product analytics shares the same R2 Data Catalog warehouse credentials as replay, but uses analytics-specific names when provided:
8+
9+
| Setting | Notes |
10+
| --- | --- |
11+
| `HOGFLARE_ANALYTICS_ACCOUNT_ID` | Cloudflare account id for the R2 Data Catalog warehouse. Falls back to `HOGFLARE_REPLAY_ACCOUNT_ID`. |
12+
| `HOGFLARE_ANALYTICS_BUCKET` | R2 bucket name backing the warehouse. Falls back to `HOGFLARE_REPLAY_BUCKET`. |
13+
| `HOGFLARE_ANALYTICS_R2_SQL_TOKEN` | R2 SQL/Data Catalog token. Store as a secret. Falls back to `HOGFLARE_REPLAY_R2_SQL_TOKEN`. |
14+
| `HOGFLARE_ANALYTICS_EVENTS_TABLE` | Iceberg events table. Falls back to `HOGFLARE_REPLAY_EVENTS_TABLE`. |
15+
| `HOGFLARE_ANALYTICS_PERSONS_TABLE` | Optional Iceberg persons table. Defaults are inferred from the events table. |
16+
| `HOGFLARE_ANALYTICS_MODEL_DIR` | Optional Sidemantic model directory. Defaults to `models`. |
17+
| `HOGFLARE_ANALYTICS_SIDEMANTIC_SCRIPT` | Optional override for the native analytics worker script. |
18+
| `HOGFLARE_ANALYTICS_PREAGG` | Optional Sidemantic pre-aggregation switch. Defaults to enabled. Set to `0` to disable. |
19+
| `HOGFLARE_ANALYTICS_PREAGG_SCHEMA` | Optional DuckDB schema for Sidemantic rollup tables. Defaults to `sidemantic_preagg`. |
20+
21+
## Routes
22+
23+
- `/` serves the Hogflare app and opens product analytics by default.
24+
- `/analytics` serves the product analytics view.
25+
- `/analytics/api/charts` returns overview metrics, a focused trend, and semantic breakdowns including domains, referrers, browser, country, region, and city leaderboards.
26+
- `/replay` serves the replay feature.
27+
28+
At worker startup, Sidemantic materializes known count-based chart shapes into daily pre-aggregation rollups and automatically routes eligible queries through those tables. `metric`, `dimension`, and `granularity` choose the focused chart. `semantic_filters` carries clickable cross-filter state as a JSON object of semantic dimensions to values. Analytics leaderboards use a fixed top-10 row cap plus an Others row.

‎docs/session-replay.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
![Hogflare replay explorer](assets/replay-explorer.jpg)
44

5-
Hogflare stores replay uploads in the same events table as analytics events, then serves a read-only replay explorer from `/replay`. The UI is for product analytics workflows: browse recent sessions, search events, inspect funnel drop-offs, review computed friction signals, and follow a person journey.
5+
Hogflare stores replay uploads in the same events table as product events, then serves a read-only replay explorer from `/replay`. The replay feature is for browsing recent sessions, searching events, inspecting funnel drop-offs, reviewing computed friction signals, and following a person journey.
66

77
## Ingestion
88

@@ -27,14 +27,16 @@ Replay APIs require:
2727

2828
The token stays server-side in the Worker. The browser only calls Hogflare's replay API.
2929

30+
Product analytics has a separate first-class route and API. See [product analytics](product-analytics.md).
31+
3032
## Routes
3133

3234
- `/replay` serves the explorer UI.
3335
- `/replay/api/sessions` lists replay sessions by reading `$snapshot_items` and legacy `$snapshot` rows from Iceberg through R2 SQL.
34-
- `/replay/api/events` searches analytics events while excluding replay recording rows.
36+
- `/replay/api/events` searches product events while excluding replay recording rows.
3537
- `/replay/api/funnels` classifies sessions as converted, stuck, or dropped for an ordered `steps` list of event names.
3638
- `/replay/api/friction` computes replay-derived signals such as rage clicks, dead clicks, form thrash, long idle gaps, repeated navigation, and deep scroll without follow-up.
37-
- `/replay/api/person` joins a distinct ID's replay sessions and analytics events into one journey timeline.
39+
- `/replay/api/person` joins a distinct ID's replay sessions and product events into one journey timeline.
3840
- `/replay/api/sessions/:session_id` returns normalized rrweb events plus an activity timeline for one session.
3941

4042
## Filters

‎models/events.yml‎

Lines changed: 77 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,6 @@ models:
5454
group3,
5555
group4,
5656
group_properties,
57-
api_key,
5857
extra,
5958
coalesce(
6059
json_extract_string(properties, '$.$session_id'),
@@ -87,6 +86,14 @@ models:
8786
json_extract_string(properties, '$.$referrer'),
8887
json_extract_string(properties, '$.referrer')
8988
) as referrer,
89+
nullif(regexp_extract(
90+
coalesce(
91+
json_extract_string(properties, '$.$referrer'),
92+
json_extract_string(properties, '$.referrer')
93+
),
94+
'^(?:[a-zA-Z][a-zA-Z0-9+.-]*://)?([^/?#]+)',
95+
1
96+
), '') as referrer_domain,
9097
coalesce(
9198
json_extract_string(properties, '$.$utm_source'),
9299
json_extract_string(properties, '$.utm_source'),
@@ -188,11 +195,11 @@ models:
188195
select
189196
*,
190197
coalesce(session_id, actor_id || ':' || strftime(event_time, '%Y-%m-%d')) as session_key,
191-
event_type not in ('$identify', '$groupidentify', '$create_alias', '$engage', '$snapshot') as is_capture_event,
198+
event_type not in ('$identify', '$groupidentify', '$create_alias', '$engage', '$snapshot', '$snapshot_items') as is_capture_event,
192199
event_type in ('$pageview', 'page_view', '$screen', 'screen') as is_pageview_event,
193200
event_type in ('$identify', '$engage') as is_person_mutation_event,
194201
event_type = '$groupidentify' as is_group_event,
195-
event_type = '$snapshot' as is_session_recording_event,
202+
event_type in ('$snapshot', '$snapshot_items') as is_session_recording_event,
196203
resolved_person_id is not null as has_person,
197204
group0 is not null or group1 is not null or group2 is not null or group3 is not null or group4 is not null as has_group,
198205
session_id is not null as has_session_id
@@ -240,9 +247,6 @@ models:
240247
- name: team_id
241248
type: numeric
242249
description: Optional PostHog team id assigned by the Worker.
243-
- name: api_key
244-
type: categorical
245-
description: PostHog project API key.
246250
- name: distinct_id
247251
type: categorical
248252
description: Original PostHog distinct id.
@@ -285,6 +289,9 @@ models:
285289
- name: referrer
286290
type: categorical
287291
description: Browser referrer.
292+
- name: referrer_domain
293+
type: categorical
294+
description: Domain extracted from the browser referrer.
288295
- name: utm_source
289296
type: categorical
290297
description: UTM source.
@@ -326,15 +333,19 @@ models:
326333
description: Country code from Cloudflare enrichment or event properties.
327334
- name: geo_region
328335
type: categorical
336+
parent: geo_country_code
329337
description: Region/subdivision from Cloudflare enrichment or event properties.
330338
- name: geo_city
331339
type: categorical
340+
parent: geo_region
332341
description: City from Cloudflare enrichment or event properties.
333342
- name: geo_timezone
334343
type: categorical
344+
parent: geo_country_code
335345
description: Timezone from Cloudflare enrichment.
336346
- name: cf_colo
337347
type: categorical
348+
parent: geo_country_code
338349
description: Cloudflare colo.
339350
- name: cf_asn
340351
type: numeric
@@ -479,8 +490,8 @@ models:
479490
- name: snapshot_events
480491
agg: count
481492
filters:
482-
- "event_type = '$snapshot'"
483-
description: Session recording snapshot events.
493+
- "event_type in ('$snapshot', '$snapshot_items')"
494+
description: Session recording snapshot and normalized snapshot-item events.
484495
- name: grouped_events
485496
agg: count
486497
filters:
@@ -500,10 +511,6 @@ models:
500511
agg: count_distinct
501512
sql: coalesce(group0, group1, group2, group3, group4)
502513
description: Distinct group keys across populated group slots.
503-
- name: unique_api_keys
504-
agg: count_distinct
505-
sql: api_key
506-
description: Distinct PostHog project API keys.
507514
- name: power_users
508515
type: cohort
509516
entity: actor_id
@@ -524,6 +531,64 @@ models:
524531
agg: count
525532
description: Count of actors with at least two sessions in the query scope.
526533

534+
pre_aggregations:
535+
- name: analytics_hourly
536+
measures:
537+
- event_count
538+
- capture_events
539+
- pageviews
540+
- snapshot_events
541+
dimensions:
542+
- event_type
543+
- pathname
544+
- host
545+
- referrer_domain
546+
- referrer
547+
- browser
548+
- os
549+
- device_type
550+
- geo_country_code
551+
- geo_region
552+
- geo_city
553+
- geo_timezone
554+
- cf_asn
555+
- utm_source
556+
- utm_campaign
557+
time_dimension: event_time
558+
granularity: hour
559+
partition_granularity: day
560+
refresh_key:
561+
every: 1 hour
562+
incremental: false
563+
- name: analytics_daily
564+
measures:
565+
- event_count
566+
- capture_events
567+
- pageviews
568+
- snapshot_events
569+
dimensions:
570+
- event_type
571+
- pathname
572+
- host
573+
- referrer_domain
574+
- referrer
575+
- browser
576+
- os
577+
- device_type
578+
- geo_country_code
579+
- geo_region
580+
- geo_city
581+
- geo_timezone
582+
- cf_asn
583+
- utm_source
584+
- utm_campaign
585+
time_dimension: event_time
586+
granularity: day
587+
partition_granularity: month
588+
refresh_key:
589+
every: 1 hour
590+
incremental: false
591+
527592
segments:
528593
- name: capture_events
529594
sql: is_capture_event
@@ -546,4 +611,3 @@ models:
546611
- name: session_recordings
547612
sql: is_session_recording_event
548613
description: Session recording snapshot events.
549-

‎models/pageviews.yml‎

Lines changed: 78 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,20 +48,29 @@ models:
4848
group2,
4949
group3,
5050
group4,
51-
api_key,
5251
coalesce(json_extract_string(properties, '$.$session_id'), json_extract_string(properties, '$.session_id')) as session_id,
5352
coalesce(json_extract_string(properties, '$.$current_url'), json_extract_string(properties, '$.current_url'), json_extract_string(properties, '$.url')) as current_url,
5453
coalesce(json_extract_string(properties, '$.$pathname'), json_extract_string(properties, '$.pathname'), json_extract_string(properties, '$.path')) as pathname,
5554
coalesce(json_extract_string(properties, '$.$host'), json_extract_string(properties, '$.host')) as host,
5655
coalesce(json_extract_string(properties, '$.$title'), json_extract_string(properties, '$.title')) as page_title,
5756
coalesce(json_extract_string(properties, '$.$referrer'), json_extract_string(properties, '$.referrer')) as referrer,
57+
nullif(regexp_extract(
58+
coalesce(json_extract_string(properties, '$.$referrer'), json_extract_string(properties, '$.referrer')),
59+
'^(?:[a-zA-Z][a-zA-Z0-9+.-]*://)?([^/?#]+)',
60+
1
61+
), '') as referrer_domain,
5862
coalesce(json_extract_string(properties, '$.$utm_source'), json_extract_string(properties, '$.utm_source'), json_extract_string(properties, '$.$initial_utm_source')) as utm_source,
5963
coalesce(json_extract_string(properties, '$.$utm_medium'), json_extract_string(properties, '$.utm_medium'), json_extract_string(properties, '$.$initial_utm_medium')) as utm_medium,
6064
coalesce(json_extract_string(properties, '$.$utm_campaign'), json_extract_string(properties, '$.utm_campaign'), json_extract_string(properties, '$.$initial_utm_campaign')) as utm_campaign,
6165
coalesce(json_extract_string(properties, '$.$browser'), json_extract_string(properties, '$.browser')) as browser,
6266
coalesce(json_extract_string(properties, '$.$os'), json_extract_string(properties, '$.os')) as os,
6367
coalesce(json_extract_string(properties, '$.$device_type'), json_extract_string(properties, '$.device_type')) as device_type,
64-
coalesce(json_extract_string(properties, '$.$geoip_country_code'), json_extract_string(properties, '$.country')) as geo_country_code
68+
coalesce(json_extract_string(properties, '$.$geoip_country_code'), json_extract_string(properties, '$.country')) as geo_country_code,
69+
coalesce(json_extract_string(properties, '$.$geoip_subdivision_1_code'), json_extract_string(properties, '$.region')) as geo_region,
70+
coalesce(json_extract_string(properties, '$.$geoip_city_name'), json_extract_string(properties, '$.city')) as geo_city,
71+
json_extract_string(properties, '$.$geoip_time_zone') as geo_timezone,
72+
json_extract_string(properties, '$.cf_colo') as cf_colo,
73+
try_cast(json_extract_string(properties, '$.cf_asn') as bigint) as cf_asn
6574
from {{ events_table }}
6675
left join identity_map on distinct_id = identity_map.linked_distinct_id
6776
)
@@ -105,6 +114,8 @@ models:
105114
type: categorical
106115
- name: referrer
107116
type: categorical
117+
- name: referrer_domain
118+
type: categorical
108119
- name: utm_source
109120
type: categorical
110121
- name: utm_medium
@@ -119,6 +130,20 @@ models:
119130
type: categorical
120131
- name: geo_country_code
121132
type: categorical
133+
- name: geo_region
134+
type: categorical
135+
parent: geo_country_code
136+
- name: geo_city
137+
type: categorical
138+
parent: geo_region
139+
- name: geo_timezone
140+
type: categorical
141+
parent: geo_country_code
142+
- name: cf_colo
143+
type: categorical
144+
parent: geo_country_code
145+
- name: cf_asn
146+
type: numeric
122147
- name: event_time
123148
type: time
124149
granularity: day
@@ -140,3 +165,54 @@ models:
140165
sql: resolved_person_id
141166
description: Distinct resolved persons with pageviews.
142167

168+
pre_aggregations:
169+
- name: analytics_hourly
170+
measures:
171+
- pageviews
172+
dimensions:
173+
- event_type
174+
- pathname
175+
- host
176+
- referrer_domain
177+
- referrer
178+
- browser
179+
- os
180+
- device_type
181+
- geo_country_code
182+
- geo_region
183+
- geo_city
184+
- geo_timezone
185+
- cf_asn
186+
- utm_source
187+
- utm_campaign
188+
time_dimension: event_time
189+
granularity: hour
190+
partition_granularity: day
191+
refresh_key:
192+
every: 1 hour
193+
incremental: false
194+
- name: analytics_daily
195+
measures:
196+
- pageviews
197+
dimensions:
198+
- event_type
199+
- pathname
200+
- host
201+
- referrer_domain
202+
- referrer
203+
- browser
204+
- os
205+
- device_type
206+
- geo_country_code
207+
- geo_region
208+
- geo_city
209+
- geo_timezone
210+
- cf_asn
211+
- utm_source
212+
- utm_campaign
213+
time_dimension: event_time
214+
granularity: day
215+
partition_granularity: month
216+
refresh_key:
217+
every: 1 hour
218+
incremental: false

0 commit comments

Comments
 (0)