Skip to content

Commit 8349ddb

Browse files
harryalbertharryalbert
authored
add detailed telemetry for the TUI (#14547)
## Description Adds detailed product telemetry for the headless TUI while preserving the existing GUI behavior and shared telemetry envelope. Covered flows include: - **TUI identification and startup:** distinguishes TUI events through `client_id=warp-tui`, records release channel/version, and reports sanitized `TERM_PROGRAM` plus terminal multiplexer context on `TUI.Startup`. - **Launch and meaningful-use signals:** keeps `TUI.Startup` as a launch metric and records prompt submission, completed commands, and agent exchanges as separate interaction signals. This PR does not connect TUI interaction to the shared `Active App Usage` heartbeat. It also does not add the GUI retention check-in: the GUI calls `/client_version/daily`, which emits the server-side `Checked For Updated Client Version` event used for retention, while the TUI updater continues to call the non-instrumented `/client_version` endpoint. Consequently, this PR does not establish GUI-parity DAU/WAU/MAU measurement. - **Agent and prompt lifecycle:** records completed, cancelled, and failed responses, response latency, request errors/retries, and defensive failed-submission paths. - **Terminal use:** covers PTY/bootstrap lifecycle, completed user command blocks, background blocks, and Local/Dev input-classification details. It intentionally does not emit GUI `Tab Creation` telemetry. - **Handoff and orchestration:** covers handoff initiation/snapshot/failure, orchestration entry and card decisions, local-vs-remote configuration, supported child navigation, and final requested/launched/failed child counts per parent conversation. - **Conversation menu:** records menu opens, item selections, and local/server restore started/succeeded/failed/cancelled outcomes through TUI-native event names. - **Voice input:** matches the GUI recording lifecycle with start, stop, cancel, duration, and current TUI input-mode metadata, without audio or transcript content. - **Slash commands, models, and credentials:** records slash-menu/command use, model selection, natural-language detection changes, and provider credential add/remove transitions without secret values. - **Shared agent tools:** attributes existing grep, file glob, code edit, MCP, computer-use, and autoexecution telemetry to the TUI through the common client identifier. - **Updater health:** records transition-deduplicated TUI update-check outcomes and failures. These are updater diagnostics, not equivalents of the GUI retention check-in. ## Testing - [x] I have manually tested my changes locally with `./script/run` Also ran w/ the `warpui_core/log_named_telemetry_events` feature and confirmed events were being emitted correctly. ## Agent Mode - [x] Warp Agent Mode - This PR was created via Warp's AI Agent Mode
1 parent ddfe0b4 commit 8349ddb

36 files changed

Lines changed: 1459 additions & 289 deletions

‎app/src/ai/blocklist/action_model.rs‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ use crate::ai::agent::{
6161
RequestCommandOutputResult,
6262
};
6363
use crate::ai::blocklist::action_model::execute::suggest_new_conversation::SuggestNewConversationExecutor;
64+
use crate::ai::blocklist::telemetry::send_run_agents_completed_telemetry;
6465
use crate::ai::document::ai_document_model::AIDocumentModel;
6566
use crate::ai::get_relevant_files::controller::GetRelevantFilesController;
6667
use crate::terminal::TerminalModel;
@@ -734,12 +735,15 @@ impl BlocklistAIActionModel {
734735
);
735736
return;
736737
};
738+
let result =
739+
AIAgentActionResultType::RunAgents(ai::agent::action_result::RunAgentsResult::Denied {
740+
reason,
741+
});
742+
send_run_agents_completed_telemetry(conversation_id, &action.action, &result, ctx);
737743
let result = Arc::new(AIAgentActionResult {
738744
id: action.id,
739745
task_id: action.task_id,
740-
result: AIAgentActionResultType::RunAgents(
741-
ai::agent::action_result::RunAgentsResult::Denied { reason },
742-
),
746+
result,
743747
});
744748
self.handle_action_result(conversation_id, result, None, ctx);
745749
}
@@ -1217,10 +1221,17 @@ impl BlocklistAIActionModel {
12171221
);
12181222
}
12191223

1224+
let cancelled_result = pending_action.action.cancelled_result();
1225+
send_run_agents_completed_telemetry(
1226+
conversation_id,
1227+
&pending_action.action,
1228+
&cancelled_result,
1229+
ctx,
1230+
);
12201231
let result = Arc::new(AIAgentActionResult {
12211232
id: pending_action.id,
12221233
task_id: pending_action.task_id,
1223-
result: pending_action.action.cancelled_result(),
1234+
result: cancelled_result,
12241235
});
12251236
self.handle_action_result(conversation_id, result, reason, ctx);
12261237
}

‎app/src/ai/blocklist/action_model/execute.rs‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ use crate::ai::agent::{
9393
};
9494
use crate::ai::ambient_agents::AmbientAgentTaskId;
9595
use crate::ai::blocklist::action_model::recording_controller::RecordingController;
96+
use crate::ai::blocklist::telemetry::send_run_agents_completed_telemetry;
9697
use crate::ai::get_relevant_files::controller::GetRelevantFilesController;
9798
#[cfg(feature = "local_fs")]
9899
use crate::ai::{agent::AnyFileContent, paths::host_native_absolute_path};
@@ -895,11 +896,18 @@ impl BlocklistAIActionExecutor {
895896
executor.cancel_execution(&tool_call_id);
896897
});
897898
}
899+
let result = running.action.action.cancelled_result();
900+
send_run_agents_completed_telemetry(
901+
running.conversation_id,
902+
&running.action.action,
903+
&result,
904+
ctx,
905+
);
898906
ctx.emit(BlocklistAIActionExecutorEvent::FinishedAction {
899907
result: Arc::new(AIAgentActionResult {
900908
id: running.action.id.clone(),
901909
task_id: running.action.task_id,
902-
result: running.action.action.cancelled_result(),
910+
result,
903911
}),
904912
conversation_id: running.conversation_id,
905913
cancellation_reason: reason,

‎app/src/ai/blocklist/action_model/execute/run_agents.rs‎

Lines changed: 27 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ use futures::FutureExt;
1616
use futures::future::BoxFuture;
1717
use warp_cli::agent::Harness;
1818
use warp_core::execution_mode::AppExecutionMode;
19+
use warp_core::telemetry::TelemetryEvent as _;
20+
use warp_core::{send_telemetry_from_app_ctx, send_telemetry_from_ctx};
1921
use warpui::{Entity, EntityId, ModelContext, ModelHandle, SingletonEntity};
2022

2123
use super::start_agent::{StartAgentExecutor, StartAgentOutcome};
@@ -25,6 +27,9 @@ use crate::ai::agent::{
2527
AIAgentAction, AIAgentActionId, AIAgentActionResultType, AIAgentActionType, AIAgentInput,
2628
StartAgentExecutionMode,
2729
};
30+
use crate::ai::blocklist::telemetry::{
31+
BlocklistOrchestrationTelemetryEvent, run_agents_completed_event,
32+
};
2833
use crate::ai::blocklist::{BlocklistAIHistoryModel, BlocklistAIPermissions};
2934
use crate::ai::document::plan_publication::{
3035
prepare_plan_publications, wait_for_plan_publications,
@@ -390,21 +395,33 @@ impl RunAgentsExecutor {
390395
&self.launched_agents,
391396
ctx,
392397
) {
393-
return ActionExecution::Sync(AIAgentActionResultType::RunAgents(
394-
RunAgentsResult::Denied { reason },
395-
));
398+
let result = RunAgentsResult::Denied { reason };
399+
send_telemetry_from_ctx!(
400+
BlocklistOrchestrationTelemetryEvent::RunAgentsCompleted(
401+
run_agents_completed_event(parent_conversation_id, &request, &result)
402+
),
403+
ctx
404+
);
405+
return ActionExecution::Sync(AIAgentActionResultType::RunAgents(result));
396406
}
407+
let telemetry_request = request.clone();
397408

398409
let receiver =
399410
self.dispatch_prepared_run_agents(action_id, request, parent_conversation_id, ctx);
400411

401-
ActionExecution::new_async(
402-
async move { receiver.recv().await },
403-
|result, _| match result {
404-
Ok(r) => AIAgentActionResultType::RunAgents(r),
405-
Err(_) => AIAgentActionResultType::RunAgents(RunAgentsResult::Cancelled),
406-
},
407-
)
412+
ActionExecution::new_async(async move { receiver.recv().await }, move |result, ctx| {
413+
let result = match result {
414+
Ok(result) => result,
415+
Err(_) => RunAgentsResult::Cancelled,
416+
};
417+
send_telemetry_from_app_ctx!(
418+
BlocklistOrchestrationTelemetryEvent::RunAgentsCompleted(
419+
run_agents_completed_event(parent_conversation_id, &telemetry_request, &result,)
420+
),
421+
ctx
422+
);
423+
AIAgentActionResultType::RunAgents(result)
424+
})
408425
}
409426

410427
pub(super) fn should_autoexecute(

‎app/src/ai/blocklist/action_model/execute/run_agents_tests.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,7 @@ fn execute_denies_duplicate_launched_agent() {
168168

169169
fn initialize_run_agents_test(app: &mut App, mode: ExecutionMode) -> RunAgentsTestState {
170170
initialize_settings_for_tests_with_mode(app, mode, false);
171+
app.update(warp_core::telemetry::testing::MockTelemetryContextProvider::register);
171172
let global_resource_handles = GlobalResourceHandles::mock(app);
172173
app.add_singleton_model(|_| GlobalResourceHandlesProvider::new(global_resource_handles));
173174
let history = app.add_singleton_model(|_| BlocklistAIHistoryModel::new(vec![], vec![], &[]));

‎app/src/ai/blocklist/handoff/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ pub use pipeline::{
3636
HandoffCommitFailure, HandoffCommitOutcome, HandoffCreated, HandoffPrepareError,
3737
HandoffPrepareInput, HandoffPresentationSnapshot, HandoffRestoration,
3838
HandoffTargetMaterialization, MaterializeHandoffTarget, PendingHandoff, execute_handoff,
39-
prepare_handoff,
39+
handoff_dispatch_error, prepare_handoff,
4040
};
4141
#[cfg(feature = "local_fs")]
4242
#[cfg_attr(not(feature = "tui"), allow(unused_imports))]

‎app/src/ai/blocklist/handoff/pipeline.rs‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,8 +48,9 @@ use crate::ai::cloud_environments::CloudAmbientAgentEnvironment;
4848
use crate::ai::execution_profiles::resolve_cloud_agent_computer_use_state;
4949
use crate::ai::llms::{LLMId, LLMPreferences};
5050
use crate::ai::orchestration::{
51-
CloudAgentStartupIssue, classify_cloud_agent_startup_error, oz_run_url,
52-
resolve_default_environment_id, resolve_default_host_slug, should_disable_snapshot,
51+
CloudAgentStartupBlocker, CloudAgentStartupFailure, CloudAgentStartupIssue,
52+
classify_cloud_agent_startup_error, oz_run_url, resolve_default_environment_id,
53+
resolve_default_host_slug, should_disable_snapshot,
5354
};
5455
use crate::cloud_object::CloudObjectLookup as _;
5556
use crate::server::ids::{ServerId, SyncId};
@@ -639,6 +640,21 @@ pub enum HandoffCommitOutcome {
639640
Created(HandoffCreated),
640641
}
641642

643+
pub fn handoff_dispatch_error(issue: &CloudAgentStartupIssue) -> String {
644+
match issue {
645+
CloudAgentStartupIssue::Blocked(CloudAgentStartupBlocker::GitHubAuthRequired {
646+
message,
647+
..
648+
})
649+
| CloudAgentStartupIssue::Failed(
650+
CloudAgentStartupFailure::Capacity { message }
651+
| CloudAgentStartupFailure::OutOfCredits { message }
652+
| CloudAgentStartupFailure::ServerOverloaded { message }
653+
| CloudAgentStartupFailure::Other { message },
654+
) => message.clone(),
655+
}
656+
}
657+
642658
/// State after selecting or creating the server-side conversation fork.
643659
struct ForkedHandoff {
644660
pending: PendingHandoff,

‎app/src/ai/blocklist/inline_action/run_agents_card_view.rs‎

Lines changed: 9 additions & 139 deletions
Original file line numberDiff line numberDiff line change
@@ -47,9 +47,8 @@ use crate::ai::blocklist::inline_action::requested_action::{
4747
CTRL_C_KEYSTROKE, ENTER_KEYSTROKE, render_requested_action_row_for_text,
4848
};
4949
use crate::ai::blocklist::telemetry::{
50-
BlocklistOrchestrationTelemetryEvent, OrchestrationApprovalStatus, OrchestrationEnteredEvent,
51-
OrchestrationEntrySource, OrchestrationExecutionModeKind, OrchestrationHarnessKind,
52-
RunAgentsCardDecision, RunAgentsCardDecisionEvent, orchestration_modified_field,
50+
BlocklistOrchestrationTelemetryEvent, OrchestrationEnteredEvent, OrchestrationEntrySource,
51+
RunAgentsCardDecision, run_agents_card_decision_event,
5352
};
5453
use crate::ai::connected_self_hosted_workers::{
5554
ConnectedSelfHostedWorkersEvent, ConnectedSelfHostedWorkersModel,
@@ -759,57 +758,17 @@ impl RunAgentsCardView {
759758
let Some(conversation_id) = self.block_model.conversation_id(ctx) else {
760759
return;
761760
};
762-
let modified_fields_from_tool_call = diverged_orch_fields(
761+
let event = run_agents_card_decision_event(
762+
conversation_id,
763+
(!self.card.plan_id.is_empty()).then(|| self.card.plan_id.clone()),
764+
decision,
765+
self.card.agent_run_configs.len(),
763766
&self.orchestration_edit_state.orchestration_config_state,
764767
&self.original_tool_call_request,
768+
self.active_config.as_ref(),
765769
);
766-
let (had_active_config, active_config_status, modified_fields_from_active_config) =
767-
match &self.active_config {
768-
Some((cfg, status)) => {
769-
let status_enum = if status.is_approved() {
770-
Some(OrchestrationApprovalStatus::Approved)
771-
} else if status.is_disapproved() {
772-
Some(OrchestrationApprovalStatus::Disapproved)
773-
} else {
774-
None
775-
};
776-
let diff = if status.is_approved() {
777-
diverged_orch_fields_against_config(
778-
&self.orchestration_edit_state.orchestration_config_state,
779-
cfg,
780-
)
781-
} else {
782-
Vec::new()
783-
};
784-
(true, status_enum, diff)
785-
}
786-
None => (false, None, Vec::new()),
787-
};
788770
send_telemetry_from_ctx!(
789-
BlocklistOrchestrationTelemetryEvent::RunAgentsCardDecision(
790-
RunAgentsCardDecisionEvent {
791-
conversation_id,
792-
plan_id: (!self.card.plan_id.is_empty()).then(|| self.card.plan_id.clone()),
793-
decision,
794-
agent_count: self.card.agent_run_configs.len(),
795-
harness: OrchestrationHarnessKind::from_str(
796-
&self
797-
.orchestration_edit_state
798-
.orchestration_config_state
799-
.harness_type
800-
),
801-
execution_mode: OrchestrationExecutionModeKind::from_run_agents(
802-
&self
803-
.orchestration_edit_state
804-
.orchestration_config_state
805-
.execution_mode,
806-
),
807-
modified_fields_from_tool_call,
808-
modified_fields_from_active_config,
809-
had_active_config,
810-
active_config_status,
811-
}
812-
),
771+
BlocklistOrchestrationTelemetryEvent::RunAgentsCardDecision(event),
813772
ctx
814773
);
815774
}
@@ -1485,95 +1444,6 @@ impl TypedActionView for RunAgentsCardView {
14851444
}
14861445
}
14871446

1488-
/// Field names from [`orchestration_modified_field`] that differ
1489-
/// between the user-edited `state` and the LLM's original
1490-
/// `RunAgentsRequest`.
1491-
fn diverged_orch_fields(
1492-
state: &oc::OrchestrationConfigState,
1493-
original: &RunAgentsRequest,
1494-
) -> Vec<&'static str> {
1495-
let mut fields = Vec::new();
1496-
if state.model_id != original.model_id {
1497-
fields.push(orchestration_modified_field::MODEL_ID);
1498-
}
1499-
if state.harness_type != original.harness_type {
1500-
fields.push(orchestration_modified_field::HARNESS);
1501-
}
1502-
let state_remote = state.execution_mode.is_remote();
1503-
let original_remote = original.execution_mode.is_remote();
1504-
if state_remote != original_remote {
1505-
fields.push(orchestration_modified_field::EXECUTION_MODE);
1506-
} else if let (
1507-
RunAgentsExecutionMode::Remote {
1508-
environment_id: state_env,
1509-
worker_host: state_host,
1510-
..
1511-
},
1512-
RunAgentsExecutionMode::Remote {
1513-
environment_id: orig_env,
1514-
worker_host: orig_host,
1515-
..
1516-
},
1517-
) = (&state.execution_mode, &original.execution_mode)
1518-
{
1519-
if state_env != orig_env {
1520-
fields.push(orchestration_modified_field::ENVIRONMENT_ID);
1521-
}
1522-
if state_host != orig_host {
1523-
fields.push(orchestration_modified_field::WORKER_HOST);
1524-
}
1525-
}
1526-
if state.auth_secret_name() != original.harness_auth_secret_name.as_deref() {
1527-
fields.push(orchestration_modified_field::AUTH_SECRET);
1528-
}
1529-
fields
1530-
}
1531-
1532-
/// Same shape as [`diverged_orch_fields`] but compares against an
1533-
/// approved `OrchestrationConfig`. auth_secret is omitted: managed
1534-
/// secrets are per-user, not stored on the config.
1535-
fn diverged_orch_fields_against_config(
1536-
state: &oc::OrchestrationConfigState,
1537-
config: &OrchestrationConfig,
1538-
) -> Vec<&'static str> {
1539-
use ai::agent::orchestration_config::OrchestrationExecutionMode;
1540-
let mut fields = Vec::new();
1541-
if state.model_id != config.model_id {
1542-
fields.push(orchestration_modified_field::MODEL_ID);
1543-
}
1544-
if state.harness_type != config.harness_type {
1545-
fields.push(orchestration_modified_field::HARNESS);
1546-
}
1547-
let state_remote = state.execution_mode.is_remote();
1548-
let config_remote = matches!(
1549-
config.execution_mode,
1550-
OrchestrationExecutionMode::Remote { .. }
1551-
);
1552-
if state_remote != config_remote {
1553-
fields.push(orchestration_modified_field::EXECUTION_MODE);
1554-
} else if let (
1555-
RunAgentsExecutionMode::Remote {
1556-
environment_id: state_env,
1557-
worker_host: state_host,
1558-
..
1559-
},
1560-
OrchestrationExecutionMode::Remote {
1561-
environment_id: cfg_env,
1562-
worker_host: cfg_host,
1563-
..
1564-
},
1565-
) = (&state.execution_mode, &config.execution_mode)
1566-
{
1567-
if state_env != cfg_env {
1568-
fields.push(orchestration_modified_field::ENVIRONMENT_ID);
1569-
}
1570-
if state_host != cfg_host {
1571-
fields.push(orchestration_modified_field::WORKER_HOST);
1572-
}
1573-
}
1574-
fields
1575-
}
1576-
15771447
fn render_confirmation_card(
15781448
orchestration_config_state: &OrchestrationConfigState,
15791449
card: &RunAgentsCardFields,

0 commit comments

Comments
 (0)