Feat--consistent-analytics-events by grutt · Pull Request #3239 · hatchet-dev/hatchet

grutt · 2026-03-11T15:43:18Z

Description

Improves analytics eventing consistency throughout applicaiton surface.

Type of change

Chore (changes which are not directly related to any business logic)

vercel · 2026-03-11T15:45:23Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hatchet-docs	Ready	Preview, Comment	Mar 12, 2026 6:51pm

Copilot

Pull request overview

This PR refactors and expands server-side analytics by introducing a structured analytics API (resource/action taxonomy), adding high-throughput aggregated counting, updating the PostHog implementation, and wiring analytics emission across gRPC services and HTTP API handlers.

Changes:

Introduces analytics.Resource / analytics.Action, context-based API token attribution, and a new Count aggregation path.
Adds an Aggregator to batch Count events and integrates it into the PostHog analytics emitter with lifecycle (Start/Close).
Instruments multiple gRPC/HTTP handlers to emit Enqueue/Count events and threads Analytics through service constructors.

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pkg/config/loader/loader.go	Initializes PostHog analytics with logger, starts it, and closes it during cleanup.
pkg/analytics/posthog/posthog.go	Reworks PostHog emitter to support typed events, aggregation-backed counts, logging, and Close().
pkg/analytics/analytics.go	Defines Resource/Action taxonomy, context token key, helpers, and expands Analytics interface.
pkg/analytics/aggregating.go	Adds a batching Aggregator for high-throughput `Count` calls.
pkg/analytics/aggregating_test.go	Adds unit tests covering aggregation, eviction, shutdown flush, and concurrency.
internal/services/ingestor/server.go	Adds `Count` instrumentation for event/log/stream ingestion endpoints.
internal/services/ingestor/ingestor.go	Adds Analytics dependency injection to ingestor service with NoOp default.
internal/services/grpc/middleware/rate_limit.go	Switches rate limit token lookup to `analytics.APITokenIDKey`.
internal/services/grpc/middleware/auth.go	Stores API token UUID in context under `analytics.APITokenIDKey`.
internal/services/dispatcher/v1/server.go	Adds worker register/listen `Count` instrumentation.
internal/services/dispatcher/v1/dispatcher.go	Adds Analytics dependency injection to v1 dispatcher service with NoOp default.
internal/services/dispatcher/server.go	Adds worker lifecycle `Count` instrumentation and new props collection.
internal/services/dispatcher/dispatcher.go	Adds Analytics dependency injection to dispatcher implementation with NoOp default.
internal/services/admin/v1/server.go	Adds `Count` instrumentation for task/workflow operations and adds workflow feature-flag extraction.
internal/services/admin/v1/admin.go	Adds Analytics default (NoOp) to v1 admin service options.
internal/services/admin/server.go	Adds `Count` instrumentation for workflow/run creation and rate limit creation; adds workflow opts feature flags extraction.
internal/services/admin/admin.go	Adds Analytics dependency injection to admin service with NoOp default.
cmd/hatchet-engine/engine/run.go	Wires `sc.Analytics` into dispatcher/admin/ingestor constructors (v0 and v1).
api/v1/server/handlers/workflows/delete.go	Emits workflow delete analytics event.
api/v1/server/handlers/v1/webhooks/delete.go	Emits webhook delete analytics event.
api/v1/server/handlers/v1/webhooks/create.go	Emits webhook create analytics event.
api/v1/server/handlers/users/update_login.go	Emits basic login analytics event.
api/v1/server/handlers/users/reject_invite.go	Migrates invite reject analytics to typed Enqueue signature.
api/v1/server/handlers/users/google_oauth_callback.go	Emits Google OAuth login analytics event.
api/v1/server/handlers/users/github_oauth_callback.go	Emits GitHub OAuth login analytics event.
api/v1/server/handlers/users/get_current.go	Switches from event enqueue to Identify for updating user properties.
api/v1/server/handlers/users/create.go	Emits user create analytics event.
api/v1/server/handlers/users/accept_invite.go	Migrates invite accept analytics to typed Enqueue signature.
api/v1/server/handlers/tenants/create.go	Emits tenant create analytics event.
api/v1/server/handlers/tenants/create_invite.go	Migrates invite create analytics to typed Enqueue signature.
api/v1/server/handlers/api-tokens/revoke.go	Migrates token revoke analytics to typed Enqueue signature.
api/v1/server/handlers/api-tokens/create.go	Migrates token create analytics to typed Enqueue signature.
api/v1/server/authn/middleware.go	Captures token UUID from JWT validation and stores it for downstream use.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

…t-dev/hatchet into feat--consistent-analytics-events

Copilot

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

internal/services/dispatcher/dispatcher.go:285

File contains unresolved Git merge-conflict markers (e.g. <<<<<<<, =======, >>>>>>>). This will not compile and also drops either the analytics assignment or streamEventBufferTimeout depending on the resolution. Resolve the conflict and ensure both fields are set in the DispatcherImpl struct literal.

		workflowRunBufferSize:       opts.workflowRunBufferSize,
		analytics:                   opts.analytics,
		streamEventBufferTimeout:    opts.streamEventBufferTimeout,
		version:                     opts.version,
	}, nil
}

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

Copilot reviewed 35 out of 35 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Copilot · 2026-03-11T18:20:03Z

+	u.config.Analytics.Enqueue(
+		ctx.Request().Context(),
+		analytics.User, analytics.Login,
+		existingUser.ID.String(),
+		map[string]interface{}{"provider": "basic"},
+	)


UserUpdateLogin is an unauthenticated endpoint (no auth middleware sets analytics context keys), so ctx.Request().Context() likely lacks analytics.UserIDKey/TenantIDKey/APITokenIDKey. In that case DistinctID(...) becomes empty and the PostHog capture will fail / be dropped. Consider deriving a context with analytics.UserIDKey set to existingUser.ID (and using that for Enqueue) so login events always have a distinct id.

Copilot · 2026-03-11T18:20:03Z

+	u.config.Analytics.Enqueue(
+		ctx.Request().Context(),
+		analytics.User, analytics.Create,
+		user.ID.String(),
+		map[string]interface{}{"provider": "basic"},
+	)


UserCreate is also unauthenticated (wrapper doesn’t set Cookie/Bearer auth scopes), so ctx.Request().Context() likely doesn’t include analytics.UserIDKey. This can lead to an empty PostHog DistinctId and the signup event being dropped. Set analytics.UserIDKey in a derived context (using the newly created user.ID) before calling Enqueue.

Copilot · 2026-03-11T18:20:04Z

+	u.config.Analytics.Enqueue(
+		ctx.Request().Context(),
+		analytics.User, analytics.Login,
+		user.ID.String(),
+		map[string]interface{}{"provider": "google"},
+	)


OAuth callback routes are unauthenticated in the generated wrapper, so ctx.Request().Context() likely lacks analytics.UserIDKey when calling Enqueue. That results in an empty DistinctId and the login event being dropped. Consider creating a derived context with analytics.UserIDKey set to user.ID for the analytics call.

Copilot · 2026-03-11T18:20:04Z

+	u.config.Analytics.Enqueue(
+		ctx.Request().Context(),
+		analytics.User, analytics.Login,
+		user.ID.String(),
+		map[string]interface{}{"provider": "github"},
+	)


OAuth callback routes are unauthenticated in the generated wrapper, so ctx.Request().Context() likely lacks analytics.UserIDKey when calling Enqueue. That results in an empty DistinctId and the login event being dropped. Consider creating a derived context with analytics.UserIDKey set to user.ID for the analytics call.

Copilot · 2026-03-11T18:20:04Z

 func (a *AdminServiceImpl) PutWorkflow(ctx context.Context, req *contracts.CreateWorkflowVersionRequest) (*contracts.CreateWorkflowVersionResponse, error) {
 	tenant := ctx.Value("tenant").(*sqlcv1.Tenant)
 	tenantId := tenant.ID
+	a.analytics.Count(ctx, analytics.Workflow, analytics.Create, putWorkflowFeatureFlags(req))



This method currently emits both Count(workflow:create, feature-flags) and Enqueue(workflow:create, workflow_id) for the same operation, which will double-count workflow creations in PostHog. Also, workflow CRUD is typically low-throughput, so aggregation via Count is likely unnecessary here. Consider consolidating into a single Enqueue call and include the feature-flag properties there.

Copilot · 2026-03-11T18:20:04Z

 func (a *AdminServiceImpl) CancelTasks(ctx context.Context, req *contracts.CancelTasksRequest) (*contracts.CancelTasksResponse, error) {
 	tenant := ctx.Value("tenant").(*sqlcv1.Tenant)
+	// FIXME-ANALYTICS: Count number of tasks cancelled
+	a.analytics.Count(ctx, analytics.TaskRun, analytics.Cancel)



CancelTasks can cancel many tasks, but the instrumentation increments the counter only once per request. This undercounts cancellations (the FIXME suggests this is known). Consider counting based on the actual number of tasks cancelled (e.g., derived from tasksToCancel / filter results) rather than per-request.

Copilot · 2026-03-11T18:20:05Z

 func (a *AdminServiceImpl) ReplayTasks(ctx context.Context, req *contracts.ReplayTasksRequest) (*contracts.ReplayTasksResponse, error) {
 	tenant := ctx.Value("tenant").(*sqlcv1.Tenant)
+	// FIXME-ANALYTICS: Count number of tasks replayed
+	a.analytics.Count(ctx, analytics.TaskRun, analytics.Replay)



ReplayTasks can replay many tasks, but the instrumentation increments the counter only once per request. This undercounts replays (the FIXME suggests this is known). Consider counting based on the actual number of tasks replayed (e.g., derived from filter results) rather than per-request.

Copilot · 2026-03-11T18:20:05Z

 	cleanup = func() error {
 		log.Printf("cleaning up server config")

 		if err := cleanupSchedulingPoolV1(); err != nil {
 			return fmt.Errorf("error cleaning up scheduling pool (v1): %w", err)
 		}
-
 		if err := cleanup1(); err != nil {
 			return fmt.Errorf("error cleaning up rabbitmq: %w", err)
 		}
+
+		if closeErr := analyticsEmitter.Close(); closeErr != nil {
+			l.Error().Err(closeErr).Msg("error closing analytics emitter")
+		}


In this cleanup function, returning early on cleanupSchedulingPoolV1() / cleanup1() errors will skip analyticsEmitter.Close(), potentially leaving the aggregator goroutine running and dropping buffered analytics on shutdown. Consider ensuring Close() is always attempted (e.g., defer it at the start of the cleanup or aggregate errors).

Copilot · 2026-03-11T18:20:05Z

+	t.config.Analytics.Enqueue(
+		ctx.Request().Context(),
+		analytics.Tenant, analytics.Create,
+		tenantId.String(),
+		map[string]interface{}{
+			"name": tenant.Name,
+			"slug": tenant.Slug,
+		},
+	)


TenantCreate is the one place where the tenant is being created, so the request context won’t already have analytics.TenantIDKey set. As a result, this tenant:create event won’t be associated with the tenant group in PostHog. Consider deriving a context with analytics.TenantIDKey = tenantId for the Enqueue call so the event is grouped to the newly created tenant.

mrkaye97 · 2026-03-11T19:47:36Z

+// Props builds a property map from variadic key-value pairs, keeping all
+// non-nil values regardless of type. Keys must be strings; non-string keys
+// are skipped. Nil values are omitted.
+func Props(kvs ...interface{}) map[string]interface{} {


can we use a more informative type than interface{} here?

mrkaye97

blocking because this makes me very nervous, both that we'll blow up our Posthog budget, and that we'll have a bunch of performance degradation, memory leak risk, etc. on the API, and, more importantly, on the engine from this. I think we need to discuss this more, and I also still feel pretty strongly that e.g. tracking counts of things like task runs, stream events, etc. in Posthog is not the right place to be putting them, but it's fine IMO if we want to make dashboards and stuff from them 🤷

mrkaye97

discussed sync, let's give it a shot

(still one or two comments to address but wanted to unblock)

github-actions · 2026-03-12T15:14:44Z

Benchmark results

goos: linux
goarch: amd64
pkg: github.com/hatchet-dev/hatchet/internal/msgqueue/rabbitmq
cpu: AMD Ryzen 9 7950X3D 16-Core Processor          
                              │ /tmp/new.txt │
                              │    sec/op    │
CompressPayloads_1x10KiB-8       76.99µ ± 3%
CompressPayloads_10x10KiB-8      906.5µ ± 4%
CompressPayloads_10x100KiB-8     10.97m ± 3%
CompressPayloads_Concurrent-8    61.91µ ± 4%
geomean                          466.6µ

                              │ /tmp/new.txt │
                              │     B/op     │
CompressPayloads_1x10KiB-8      11.08Ki ± 4%
CompressPayloads_10x10KiB-8     111.0Ki ± 1%
CompressPayloads_10x100KiB-8    2.916Mi ± 0%
CompressPayloads_Concurrent-8   54.34Ki ± 1%
geomean                         118.9Ki

                              │ /tmp/new.txt │
                              │  allocs/op   │
CompressPayloads_1x10KiB-8        5.000 ± 0%
CompressPayloads_10x10KiB-8       32.00 ± 0%
CompressPayloads_10x100KiB-8      63.00 ± 2%
CompressPayloads_Concurrent-8     17.00 ± 0%
geomean                           20.35

_{Compared against main (66b0d93)}

mrkaye97

🚀

grutt added 3 commits March 10, 2026 14:50

chore: log when enqueue errors

778a3e1

feat: standardize and improve actor model

7c7459f

feat: track entire surface

0c89a7c

Copilot AI review requested due to automatic review settings March 11, 2026 15:43

Copilot started reviewing on behalf of grutt March 11, 2026 15:43 View session

vercel Bot deployed to Preview March 11, 2026 15:46 View deployment

chore: remove heartbeats

710387a

vercel Bot deployed to Preview March 11, 2026 15:50 View deployment

Copilot AI reviewed Mar 11, 2026

View reviewed changes

Comment thread internal/services/dispatcher/server.go Outdated

Comment thread api/v1/server/authn/middleware.go

Comment thread internal/services/grpc/middleware/rate_limit.go Outdated

Comment thread pkg/config/loader/loader.go

Comment thread pkg/analytics/aggregating_test.go

grutt added 2 commits March 11, 2026 13:42

cleanup

7a187ee

Merge branch 'main' into feat--consistent-analytics-events

301b0fa

grutt requested a review from Copilot March 11, 2026 17:43

Merge branch 'feat--consistent-analytics-events' of github.com:hatche…

eebacd9

…t-dev/hatchet into feat--consistent-analytics-events

Copilot started reviewing on behalf of grutt March 11, 2026 17:45 View session

grutt added 2 commits March 11, 2026 13:45

fix: merge

bb0d87b

Merge branch 'main' into feat--consistent-analytics-events

953d68e

vercel Bot deployed to Preview March 11, 2026 17:50 View deployment

Copilot AI reviewed Mar 11, 2026

View reviewed changes

grutt added 2 commits March 11, 2026 14:04

chore: cleanup and AI

01ac87b

fix: getter

e5fcdeb

vercel Bot deployed to Preview March 11, 2026 18:08 View deployment

grutt requested a review from Copilot March 11, 2026 18:12

Copilot started reviewing on behalf of grutt March 11, 2026 18:12 View session

vercel Bot deployed to Preview March 11, 2026 18:13 View deployment

Copilot AI reviewed Mar 11, 2026

View reviewed changes

mrkaye97 reviewed Mar 11, 2026

View reviewed changes

Comment thread pkg/analytics/analytics.go Outdated

mrkaye97 reviewed Mar 11, 2026

View reviewed changes

mrkaye97 requested changes Mar 11, 2026

View reviewed changes

mrkaye97 approved these changes Mar 11, 2026

View reviewed changes

chore: feedback

10e25cd

vercel Bot deployed to Preview March 12, 2026 15:19 View deployment

flush mu

efe0a98

vercel Bot deployed to Preview March 12, 2026 16:43 View deployment

grutt added 2 commits March 12, 2026 12:55

feat: source attribute

74fd373

feat: cli source

8ade384

vercel Bot deployed to Preview March 12, 2026 17:02 View deployment

fix: webhook

3851d22

mrkaye97 approved these changes Mar 12, 2026

View reviewed changes

vercel Bot deployed to Preview March 12, 2026 18:00 View deployment

feat: add server url

de12c60

vercel Bot deployed to Preview March 12, 2026 18:51 View deployment

grutt merged commit 6419ad3 into main Mar 12, 2026
52 checks passed

grutt deleted the feat--consistent-analytics-events branch March 12, 2026 19:02

Uh oh!

Conversation

grutt commented Mar 11, 2026

Description

Type of change

Uh oh!

vercel Bot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mrkaye97 Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

mrkaye97 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrkaye97 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark results

Uh oh!

mrkaye97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

vercel Bot commented Mar 11, 2026 •

edited

Loading

mrkaye97 left a comment •

edited

Loading

github-actions Bot commented Mar 12, 2026 •

edited

Loading