[docs] Overview page redesign plan for agents#5429
Conversation
- Introduced README.md outlining the redesign goals and structure. - Created context.md detailing the current problems and scope of the Overview page. - Added design.md specifying views, empty states, and onboarding processes. - Developed plan.md for the phased implementation strategy. - Included research.md summarizing current data sources and their usage. - Established status.md to track the planning stage and decisions made.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe PR adds planning documentation for an Agent Overview redesign, covering scope, verified data sources, view and empty-state behavior, phased frontend delivery, backend attribution work, feature-flag rollout, locked decisions, and open implementation questions. ChangesAgent Overview redesign
Estimated code review effort: 2 (Simple) | ~15 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ashrafchowdury
changed the title
|
I have not reviewed yet |
|
We can show, in natural language:
The goal is to keep the surface simple and informative and avoid making it overcomplicated. |
…rics and context usage details
…ntegration and cost/token attribution details
…guidance and onboarding experience
dosubot
Bot
added
size:XL
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 31423d2b-d441-4253-9ac2-93b0d7770e59
📒 Files selected for processing (6)
docs/design/agent-workflows/projects/agent-overview-redesign/README.mddocs/design/agent-workflows/projects/agent-overview-redesign/context.mddocs/design/agent-workflows/projects/agent-overview-redesign/design.mddocs/design/agent-workflows/projects/agent-overview-redesign/plan.mddocs/design/agent-workflows/projects/agent-overview-redesign/research.mddocs/design/agent-workflows/projects/agent-overview-redesign/status.md
| - **Success rate** — share of recent runs that completed — *tracing aggregate*. | ||
| - **Avg cost / run** — typical spend per run — *tracing aggregate (`gen_ai.usage.cost`)*. | ||
| - **Run now** — manual trigger — *triggers (manual)*. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Make cost availability behavior consistent across the design and delivery contract.
Agent cost is currently unavailable until attribution is fixed, so the design must specify an unavailable/hidden state rather than an unconditional aggregate. The Slice 1 exit criteria must likewise avoid requiring real cost before that dependency lands.
docs/design/agent-workflows/projects/agent-overview-redesign/design.md#L27-L29: mark Avg cost/run as attribution-gated and define its empty state.docs/design/agent-workflows/projects/agent-overview-redesign/plan.md#L33-L40: remove “real cost” from the pre-attribution exit criteria or explicitly make it conditional.
📍 Affects 2 files
docs/design/agent-workflows/projects/agent-overview-redesign/design.md#L27-L29(this comment)docs/design/agent-workflows/projects/agent-overview-redesign/plan.md#L33-L40
| 1. Needs-you: count + list of `pending` session interactions, each linking to where it's | ||
| answered (reuse `ApprovalDock` where possible). | ||
| 2. Connections: list with health from `TriggerConnectionStatus`; Reconnect for NEEDS_AUTH. | ||
| 3. Triggers: active schedules / subscriptions + next scheduled run. | ||
| - **Exit:** a blocked agent shows its pending items and they resolve from Overview; a | ||
| connection needing reauth shows Reconnect; schedules show next run. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Align Slice 3 with the Phase 1 interaction-resolution decision.
The exit criterion says pending items resolve directly from Overview, but status.md Q4 recommends linking out in Phase 1 and embedding resolution later. Define the exit criterion as link-out, or update the status decision before implementation.
| 2. Connections: list with health from `TriggerConnectionStatus`; Reconnect for NEEDS_AUTH. | ||
| 3. Triggers: active schedules / subscriptions + next scheduled run. | ||
| - **Exit:** a blocked agent shows its pending items and they resolve from Overview; a | ||
| connection needing reauth shows Reconnect; schedules show next run. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the referenced docs
git ls-files 'docs/design/agent-workflows/projects/agent-overview-redesign/*' | sed -n '1,20p'
echo '--- plan.md (around lines 1-140) ---'
nl -ba docs/design/agent-workflows/projects/agent-overview-redesign/plan.md | sed -n '1,140p'
echo '--- research.md search for NEEDS_* ---'
rg -n 'NEEDS_(AUTH|INPUT|READY)|TriggerConnectionStatus|Reconnect|input|configuration' docs/design/agent-workflows/projects/agent-overview-redesign/research.md docs/design/agent-workflows/projects/agent-overview-redesign/plan.mdRepository: Agenta-AI/agenta
Length of output: 658
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the surrounding design docs for any explicit handling of NEEDS_INPUT
fd -a --type f 'research.md|plan.md' docs/design/agent-workflows/projects/agent-overview-redesign
echo '--- research.md (full, if small) ---'
wc -l docs/design/agent-workflows/projects/agent-overview-redesign/research.md
nl -ba docs/design/agent-workflows/projects/agent-overview-redesign/research.md | sed -n '1,220p'Repository: Agenta-AI/agenta
Length of output: 488
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Check if the plan already includes any handling for NEEDS_INPUT in nearby sections
nl -ba docs/design/agent-workflows/projects/agent-overview-redesign/plan.md | sed -n '1,220p' | rg -n 'NEEDS_INPUT|NEEDS_AUTH|READY|Reconnect|input|configuration|Exit|Connections|Triggers'Repository: Agenta-AI/agenta
Length of output: 190
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the exact snippet context with line numbers
nl -ba docs/design/agent-workflows/projects/agent-overview-redesign/plan.md | sed -n '54,70p'Repository: Agenta-AI/agenta
Length of output: 194
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- plan.md excerpt ---'
sed -n '54,70p' docs/design/agent-workflows/projects/agent-overview-redesign/plan.md
echo '--- research.md NEEDS_* matches ---'
grep -nE 'NEEDS_(AUTH|INPUT|READY)|TriggerConnectionStatus|Reconnect|input|configuration' \
docs/design/agent-workflows/projects/agent-overview-redesign/research.md \
docs/design/agent-workflows/projects/agent-overview-redesign/plan.md || true
echo '--- research.md excerpt around NEEDS_INPUT if present ---'
grep -n 'NEEDS_INPUT' docs/design/agent-workflows/projects/agent-overview-redesign/research.md || trueRepository: Agenta-AI/agenta
Length of output: 3122
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the exact section in research.md where TriggerConnectionStatus is described
grep -n 'TriggerConnectionStatus' docs/design/agent-workflows/projects/agent-overview-redesign/research.md || true
grep -n -A6 -B6 'TriggerConnectionStatus' docs/design/agent-workflows/projects/agent-overview-redesign/research.md || trueRepository: Agenta-AI/agenta
Length of output: 906
Define NEEDS_INPUT behavior. research.md includes READY, NEEDS_AUTH, and NEEDS_INPUT, but this slice only defines Reconnect for NEEDS_AUTH. Add the matching input/configuration action and include it in the exit criteria so NEEDS_INPUT isn’t left undefined.
| 1. **Agent cost/token attribution** (unblocks the Cost/Token views, research.md §6). Pick | ||
| one: (a) map `gen_ai.usage.cost` in semconv and treat agent/workflow root spans as | ||
| cost-bearing (or bridge the harness's separate OTLP batch so its LLM spans roll onto the | ||
| root); or (b) accept the FE reading the agent's reported per-run usage from the trace and | ||
| aggregating client-side. Either makes the root cost/token paths the gateway reads | ||
| non-empty for agents. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Separate the client fallback from gateway attribution.
Option (b) reads reported usage in the frontend; it cannot make the root analytics paths non-empty or unblock /analytics/query. Split the alternatives and state that only the ingest/roll-up option fixes gateway data, while the frontend path remains a fallback.
| 3. **Aggregation freshness budget** — is live per-request aggregation acceptable for | ||
| Phase 1, and what staleness is tolerable before Slice 5 becomes required? *Recommend:* | ||
| ship Phase 1 live; set a first-paint latency threshold that triggers Slice 6. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Correct the slice reference in the freshness question.
Aggregation performance concerns should trigger Slice 6, which contains the materialized aggregates and fan-out reduction work—not Slice 5, which only covers empty states and never-run guidance.
mmabrouk
left a comment
mmabrouk
left a comment
There was a problem hiding this comment.
Thank you, Ashraf. I think we should start simple and iterate. I think this is overly complex. Let's do baby steps and use only things that we have right now.
| - **Where it's failing** — failures grouped by cause — *`ag.exception.type`; child-span data, | ||
| same root-only caveat as tools.* | ||
|
|
||
| ### Resource usage & cost (the "what is it consuming" group) |
There was a problem hiding this comment.
Basically, there is a part which is analytics, which we start with what we have and improve our time.
| why the live dashboard shows `-`. Depends on the backend fix (plan.md Slice 4 note + | ||
| Slice 6); until then, degrade to the run-level token count the agent reports, not a zero. | ||
|
|
||
| ### Reliability (drill-down, not forced) |
There was a problem hiding this comment.
Analytics which we improve over time
| - **Run meta** — trigger source · duration · cost · tools-used count — *tracing + | ||
| triggers*. | ||
|
|
||
| ### Metrics (charts) — from the analytics gateway |
There was a problem hiding this comment.
Analytics which we improve over time
| agent is blocked on, each linking to where it's answered — *session interactions, | ||
| `status = pending`*. | ||
|
|
||
| ### What your agent produced (outcomes feed) |
There was a problem hiding this comment.
This is overly complex. Feels to me that this can be swapped by just a list of sessions.
| Plain-language work stays on top for the operator; usage/efficiency detail sits lower and | ||
| behind clicks for the builder and budget reader. | ||
|
|
||
| ### Connections & triggers |
There was a problem hiding this comment.
Triggers make sense. Connection a little bit less, since connections are at a project level or on certain revisions.
There was a problem hiding this comment.
Reading this, a simplified version of this would be:
- one section for analytics
- one section for sessions
- one section for triggers
Redesign overview page
Onboarding - frash agentI have intentionally kept it simple to avoid making it overcomplicated by adding steps and anything else 
|
2 participants
Context
The agent Overview page is inherited wholesale from the prompt-management era. For an agent workflow it renders deployment cards, a variant list, and evaluation runs, which describe a prompt asset being promoted through dev/staging/prod. They do not describe an agent's work. Opening an agent's Overview today, you cannot see what it produced, whether it needs you, or whether it is healthy. A brand-new agent lands on the same panels rendering zeros, with no path to first value.
This PR adds a design workspace only. No product code changes, and no implementation has started.
What this adds
A planning folder under
docs/design/agent-workflows/projects/agent-overview-redesign/that reframes the page around three questions, in priority order:Six documents, each grounded in verified backend sources:
README.md— the redesign at a glance: decisions, deliverables, intended outcome.context.md— problem, scope, non-goals, product language (Run, Outcome, Artifact, Trigger, Needs-you item, Connection), success criteria.research.md— what the code gives us today, withfile:line. Maps each proposed view to a real data source: tracing (ag.meta.final_result, tool calls, token/cost/duration, exceptions), session mounts for produced files, session interactions for needs-you items, triggers/connections for schedules and reconnect state.design.md— the view catalog for an active agent, the three distinct zero states, and the fresh-agent onboarding flow. Names views and data only; layout and visuals stay with design.plan.md— a sliced implementation (Slice 0–5) behind workflow-kind gating and a feature flag, each slice leaving the tree working.status.md— living source of truth: locked decisions plus five open questions with recommendations.Key decisions captured