feat(tasks): inject agent OTLP telemetry into cloud sandboxes

[tatoalo]

Problem

Cloud task agent telemetry needs an opt-in path from the sandbox to PostHog Logs and Tracing. The backend did not have safe configuration plumbing for the OTLP endpoints and project token, and fresh runs and snapshot resumes needed identical behavior.

Changes

Before:

flowchart LR
    Agent{{Cloud agent}} --> API[Run log API]
    API --> Session[(Session log)]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;
    class Agent phBlue;
    class API phRed;
    class Session phGray;

Loading

After:

flowchart LR
    Settings[Backend settings] --> Agent{{Cloud agent}}
    Agent --> API[Run log API]
    API --> Session[(Session log)]
    Agent --> Logs[OTLP logs]
    Agent --> Traces[OTLP traces]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;
    class Agent phBlue;
    class API,Logs,Traces phRed;
    class Settings phYellow;
    class Session phGray;

Loading

• Add optional sandbox settings for the OTLP logs URL, traces URL, and project token. Telemetry remains disabled unless the logs URL and token are both configured.
• Inject the agent-specific environment variables into fresh and snapshot-resumed sandboxes.
• Reserve the variables so task-provided sandbox configuration cannot override them.
• Support local agentsh firewall allowlisting and Docker host URL rewriting.
• Document the local configuration and add coverage for gating, reservation, and both sandbox creation paths.

Also carried on this branch: scout run log mirror (#71094)

This branch now incorporates the scout run log mirror from #71094 (credit @andrewm4894), squashed from that PR's final state so both delivery paths ship together:

• TaskRun.append_log emits one structured stdout line per persisted entry (event=task_run_log); the per-cluster OTel collector that already ships container stdout delivers them into the region's internal project's Logs, with request_id (the run uuid) becoming the trace id.
• Scoped by TASK_RUN_LOGS_MIRROR_ORIGIN_PRODUCTS (default: signals_scout only; empty disables). Mirror failures are logged and never break the run's log write.
• On top of feat(tasks): mirror scout run logs into posthog logs #71094, this branch adds a direct OTLP delivery leg (TASK_RUN_LOGS_MIRROR_OTLP_URL/_TOKEN, default unset). The token pins the destination to PostHog's own internal logs project — scout runs execute for customer teams, and their mirrored transcripts must never land in (or bill) a customer's project. It is also the only delivery path in local dev, where append_log runs in the host Django process whose stdout the dev collector (docker containers only) never tails.
• Complementary postures: the mirror carries readable bodies for PostHog-authored scout runs into the internal project, while the OTLP path above stays metadata-only and covers customer-driven cloud runs.
• Protocol setup envelopes and metadata-only updates are mirrored as type labels rather than raw payloads, so configuration headers never enter Logs.

If #71094 merges separately first, the duplicated commit here should be dropped on the next rebase.

[github-actions]

🤖 CI report

✅ Playwright — all passed

All tests passed.

View test results →

⚠️ Backend snapshots — 11 updated (11 modified, 0 added, 0 deleted)

Query snapshots: Backend query snapshots updated

Changes: 11 snapshots (11 modified, 0 added, 0 deleted)

What this means:

• Query snapshots have been automatically updated to match current output
• These changes reflect modifications to database queries or schema

Next steps:

• Review the query changes to ensure they're intentional
• If unexpected, investigate what caused the query to change

Review snapshot changes →

⚠️ Backend coverage — 82.0% of changed backend lines covered — 76 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ████████████████░░░░ 82.0% (371 / 447)

File	Patch	Uncovered changed lines
products/tasks/backend/temporal/process_task/tests/test_utils.py	0.0%	3, 1072–1073, 1077, 1081–1082, 1087, 1089–1091, 1093, 1097, 1101–1102, 1107, 1109, 1111, 1115, 1119–1120, 1125, 1127
products/tasks/backend/temporal/process_task/tests/test_provision_sandbox.py	0.0%	202–205, 226, 229, 231, 236, 238, 241–245, 247, 252, 254
products/tasks/backend/temporal/process_task/activities/provision_sandbox.py	0.0%	348–349
products/tasks/backend/temporal/process_task/activities/tests/test_get_task_processing_context.py	0.0%	42–43, 54–56
products/tasks/backend/temporal/process_task/utils.py	11.1%	1152–1153, 1155, 1165–1167, 1171–1172
products/tasks/backend/temporal/process_task/activities/get_task_processing_context.py	25.0%	381–382, 384–386, 388–389, 394, 795
products/tasks/backend/feature_flags.py	72.7%	46–48
products/tasks/backend/temporal/client.py	88.9%	124, 152–153
products/tasks/backend/logic/services/run_log_mirror.py	94.9%	190–192, 198, 208–209, 281

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 30028222937 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)

Product	Coverage	Lines
platform_features	██░░░░░░░░░░░░░░░░░░ 12.1%	7 / 58
batch_exports	████████░░░░░░░░░░░░ 39.6%	8,414 / 21,249
demo	███████████░░░░░░░░░ 56.2%	1,497 / 2,663
warehouse_sources_queue	████████████░░░░░░░░ 59.2%	148 / 250
tasks	██████████████░░░░░░ 69.4%	30,000 / 43,235
data_tools	██████████████░░░░░░ 70.0%	63 / 90
ai_gateway	███████████████░░░░░ 75.0%	9 / 12
signals	████████████████░░░░ 79.8%	21,628 / 27,098
cdp	████████████████░░░░ 81.0%	3,144 / 3,881
data_modeling	█████████████████░░░ 82.7%	5,547 / 6,708
wizard	█████████████████░░░ 84.8%	935 / 1,102
notebooks	█████████████████░░░ 85.3%	7,275 / 8,531
agent_platform	█████████████████░░░ 86.4%	3,807 / 4,405
actions	█████████████████░░░ 86.6%	717 / 828
cohorts	██████████████████░░ 87.8%	4,488 / 5,114
product_tours	██████████████████░░ 87.9%	1,303 / 1,482
exports	██████████████████░░ 88.4%	6,949 / 7,861
data_warehouse	██████████████████░░ 88.9%	11,887 / 13,378
business_knowledge	██████████████████░░ 89.0%	4,391 / 4,936
conversations	██████████████████░░ 89.3%	16,820 / 18,834
engineering_analytics	██████████████████░░ 89.4%	6,371 / 7,130
dashboards	██████████████████░░ 89.4%	5,983 / 6,693
visual_review	██████████████████░░ 89.4%	5,818 / 6,505
error_tracking	██████████████████░░ 89.7%	10,174 / 11,342
alerts	██████████████████░░ 89.9%	4,054 / 4,508
early_access_features	██████████████████░░ 90.1%	1,031 / 1,144
mcp_analytics	██████████████████░░ 90.1%	2,763 / 3,065
streamlit_apps	██████████████████░░ 90.4%	2,501 / 2,767
links	██████████████████░░ 90.6%	183 / 202
slack_app	██████████████████░░ 90.7%	9,028 / 9,951
marketing_analytics	██████████████████░░ 91.0%	11,792 / 12,964
stamphog	██████████████████░░ 91.1%	4,056 / 4,450
mcp_store	██████████████████░░ 91.9%	4,257 / 4,634
notifications	███████████████████░ 92.6%	1,011 / 1,092
product_analytics	███████████████████░ 92.7%	5,811 / 6,268
ai_observability	███████████████████░ 92.7%	14,995 / 16,168
surveys	███████████████████░ 93.0%	5,730 / 6,163
managed_migrations	███████████████████░ 93.1%	1,424 / 1,530
web_analytics	███████████████████░ 93.1%	14,398 / 15,461
posthog_ai	███████████████████░ 93.2%	1,326 / 1,422
approvals	███████████████████░ 93.3%	3,437 / 3,682
reminders	███████████████████░ 93.4%	468 / 501
workflows	███████████████████░ 93.6%	6,390 / 6,826
legal_documents	███████████████████░ 94.1%	1,568 / 1,667
endpoints	███████████████████░ 94.1%	8,640 / 9,177
messaging	███████████████████░ 94.3%	2,682 / 2,845
tracing	███████████████████░ 94.5%	2,670 / 2,826
revenue_analytics	███████████████████░ 94.6%	3,651 / 3,858
skills	███████████████████░ 94.6%	3,035 / 3,207
review_hog	███████████████████░ 94.7%	6,814 / 7,199
logs	███████████████████░ 95.4%	9,935 / 10,416
experiments	███████████████████░ 95.7%	25,036 / 26,159
growth	███████████████████░ 96.1%	3,245 / 3,376
annotations	███████████████████░ 96.2%	732 / 761
replay_vision	███████████████████░ 96.2%	15,094 / 15,691
feature_flags	███████████████████░ 96.5%	17,110 / 17,734
user_interviews	███████████████████░ 96.5%	2,638 / 2,734
access_control	███████████████████░ 96.8%	851 / 879
customer_analytics	███████████████████░ 97.1%	9,047 / 9,317
warehouse_sources	███████████████████░ 97.1%	329,165 / 338,906
data_catalog	███████████████████░ 97.4%	2,367 / 2,429
analytics_platform	████████████████████ 98.0%	2,153 / 2,197
metrics	████████████████████ 98.2%	2,491 / 2,536
pulse	████████████████████ 98.4%	2,017 / 2,049
live_debugger	████████████████████ 99.2%	613 / 618
field_notes	████████████████████ 99.4%	158 / 159

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

ℹ️ Docs preview — preview build triggered

Docs from this PR will be published at posthog.com.

Project	Preview	Updated (UTC)
posthog.com	Open preview	Jul 23, 2026, 5:10 PM

The preview should be ready in about 10 minutes. Open the preview at /handbook/engineering/.

[greptile-apps]

Security Review

Customer-controlled sandbox subprocesses inherit the reusable internal telemetry project key, allowing arbitrary writes to that project's logs and traces.

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
products/tasks/backend/temporal/process_task/utils.py:1167-1168
**Telemetry credential reaches user processes**

When telemetry is configured, this helper places its project API key in the sandbox environment inherited by customer-controlled commands, allowing those commands to exfiltrate the key and submit arbitrary logs or traces that corrupt internal telemetry and consume its ingestion allowance.

### Issue 2 of 2
products/tasks/backend/logic/services/run_log_mirror.py:175-180
**OTLP request blocks log appends**

When the configured OTLP endpoint is slow or unavailable, this synchronous request blocks the Django worker handling `append_log` for the connect and read timeout, delaying or timing out the agent's persisted log-delivery path during telemetry-ingest degradation.

_{Reviews (1): Last reviewed commit: "refactor(tasks): simplify run log mirror..." | Re-trigger Greptile}

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 4 · PR risk: 0/10

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-23 18:41 UTC	Run
prod-us	✅ Deployed	2026-07-23 19:10 UTC	Run
prod-eu	✅ Deployed	2026-07-23 19:14 UTC	Run