Skip to content

Commit 1416147

Browse files
abossardCopilot
andauthored
Agent workbench v2 (#19)
* Extract agent builder into extensible module with tests Create backend/agent_builder/ as a standalone, deeply layered module following Grokking Simplicity (data/calculations/actions separation) and A Philosophy of Software Design (deep modules). Structure: - models/: Pure data (Pydantic/SQLModel) - agent, run, evaluation, chat - tools/: ToolRegistry, schema converter, MCP adapter - engine/: Unified ReAct runner, callbacks, prompt builder - evaluator.py: Success criteria evaluation (mostly calculations) - persistence/: DB engine setup + repository pattern - service.py: WorkbenchService (deep module facade) - chat_service.py: ChatService using shared ReAct engine - routes.py: Quart Blueprint replacing 200+ lines from app.py - tests/: 107 tests (unit + integration + E2E) Key improvements: - Eliminated duplicate ReAct agent building (was in both agents.py and agent_workbench/service.py) - DRY error handling in routes via Blueprint - Repository pattern isolates DB from business logic - Pure calculation modules (prompt_builder, schema_converter, evaluator) are independently testable - Backward-compatible: agent_workbench/__init__.py shims to new module Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add per-agent LLM config: model, temperature, recursion_limit, max_tokens, output_instructions Each AgentDefinition now stores configurable LLM parameters: - model: override service default (e.g. gpt-4o vs gpt-4o-mini) - temperature: 0.0-2.0 (deterministic to creative) - recursion_limit: 1-100 max ReAct loop iterations - max_tokens: cap response length (0 = unlimited) - output_instructions: custom formatting (replaces default markdown) Changes: - models/agent.py: 5 new fields with validation (ge/le bounds) - persistence/database.py: migrations for existing DBs - engine/react_runner.py: build_llm accepts temperature+max_tokens - engine/prompt_builder.py: append_output_instructions for custom formatting - service.py: _resolve_llm_for_agent builds per-agent LLM when config differs - routes.py: ui-config v2 exposes llm_config_fields and defaults - 12 new tests (model validation, CRUD, E2E roundtrip via REST) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add output_schema for type-safe structured output, fix defaults Changes: - recursion_limit default: 10 → 3 (most agents finish in 1-3 tool calls) - max_tokens default: 0 → 4096 (sensible cap instead of unlimited) - New field: output_schema (JSON Schema stored as JSON in DB) output_schema is config, not code. You define the expected response shape as a JSON Schema: {"type":"object","properties":{"breaches":{"type":"array",...}}} At runtime this does two things: 1. Injected into system prompt so the LLM knows the expected structure 2. Takes priority over output_instructions and default markdown Priority chain for output formatting: output_schema (strict JSON) > output_instructions (free text) > default markdown 128 tests pass (9 new tests for schema handling). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add suggest-schema endpoint and UI button New endpoint: POST /api/workbench/suggest-schema Takes agent name, description, system_prompt and asks the LLM to propose a JSON Schema for the agent's structured output. Backend: - service.py: suggest_schema() method - builds a prompt, calls LLM, parses JSON response (handles markdown fences), falls back to generic schema on parse failure - routes.py: POST /api/workbench/suggest-schema route Frontend: - api.js: suggestOutputSchema() function - WorkbenchPage.jsx: output schema textarea + Suggest Schema button in the create form. Schema is editable JSON, sent as output_schema on agent creation. Button disabled until name or prompt is filled. 129 tests pass (1 new E2E test for suggest-schema endpoint). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Wire output_schema to LangGraph response_format for SDK-level enforcement When an agent has output_schema configured, it now does TWO things: 1. Prompt injection (existing) — schema is described in the system prompt so the LLM understands the expected structure 2. SDK enforcement (new) — schema is passed as response_format to create_react_agent(), which uses LangGraph's built-in structured output mechanism (provider-native or tool-based) At runtime, structured_response from the LangGraph result takes priority over raw message content. If the agent has no output_schema, behavior is unchanged (markdown output from final message). The output pipeline: output_schema defined → response_format=schema → structured_response → JSON no output_schema → final message content → markdown (default) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Always use structured_response with default schema Every agent now always returns structured output via LangGraph's response_format — no more untyped markdown strings. Default schema (when no custom output_schema is set): { "message": "string (markdown)", "referenced_tickets": ["string"] } This means: - Plain agents → get {message: '...markdown...', referenced_tickets: [...]} - Custom schema agents → get whatever schema they define - Both enforced at SDK level via response_format, not just prompt Changes: - prompt_builder.py: DEFAULT_OUTPUT_SCHEMA, resolve_output_schema() - service.py: always passes effective schema to create_react_agent - routes.py: ui-config exposes default_output_schema for frontend - Tests updated (132 pass) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add comprehensive docs with mermaid diagrams, clean up stale docs New: docs/AGENT_BUILDER.md — full architecture documentation with: - Architecture diagram (module layers + data flow) - Sequence diagram (agent run lifecycle) - Structured output pipeline flowchart - ER diagram (DB schema) - Data/Calculations/Actions separation diagram - Deep modules table - Extensibility flowchart - API endpoint reference - Testing commands Updated: - AGENTS_IMPLEMENTATION.md — replaced stale content with summary + pointer - docs/AGENTS.md — replaced stale architecture with mermaid + pointer - docs/PROJECT_STRUCTURE.md — added agent_builder/ to tree Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Docs overhaul + remove ~1800 lines of dead code/stale docs Documentation: - README.md: Complete rewrite with features table, screenshots, mermaid architecture diagram, agent builder section, correct tech stack - PROJECT_STRUCTURE.md: Full rewrite matching actual codebase - AGENTS.md: Fixed AgentService→WorkbenchService, updated examples - LEARNING.md: Fixed broken link Deleted stale docs: - AGENTS_IMPLEMENTATION.md (was a 3-line redirect stub) - docs/RULES.md (empty file) - docs/SQLMODEL_MIGRATION.md (historical, migration complete) Dead code removed from agents.py (~250 lines): - MCP client stubs (_mcp_tool_to_langchain, _ensure_ticket_mcp_connection, close) - Schema helpers only used by dead MCP code (_json_type_to_python, _schema_to_pydantic) - OpenAI logging callback (duplicated in agent_builder/engine/callbacks.py) - _build_state_graph learning example (dead code) - Unused imports (get_langchain_tools, MCPClient, create_model) Deleted old agent_workbench/ source files (~1030 lines): - models.py, service.py, evaluator.py, tool_registry.py - Only __init__.py shim remains for backward compatibility 132 backend tests + 15 Playwright tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Playwright tests for suggest-schema and agent chat New E2E tests in workbench.spec.js: - 'creates agent with output schema via suggest button' — mocks /api/workbench/suggest-schema, clicks Suggest Schema, verifies schema populates textarea, creates agent, deletes it - 'sends message and displays mocked response' (Agent Chat UI) — mocks /api/agents/run, types message, clicks send, verifies markdown heading and tool badge render 17 Playwright tests pass (was 15, +2 new). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add VPN agent and failure handling Playwright tests New Agent Fabric E2E tests: - 'runs VPN troubleshooting agent and verifies structured output' Creates agent with VPN analysis prompt, runs it (mocked), verifies structured JSON output with ticket IDs (INC-101, INC-312), referenced_tickets field, and VPN content in rendered output - 'handles agent run failure gracefully' Creates agent, runs it with mocked failure response, verifies UI doesn't crash and shows completion state 19 Playwright tests pass (was 17, +2 new). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix structured output rendering in Agent Fabric UI The output is now always structured JSON ({message, referenced_tickets}). The UI now parses it and renders each part appropriately: - message → rendered as GitHub-flavored Markdown (ReactMarkdown) - referenced_tickets → rendered as monospace badges below the output - Extra custom schema fields → rendered as formatted JSON in a pre block - Button preview → shows message text, not raw JSON Also handles non-JSON output gracefully (falls back to raw markdown). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add MCP App technical documentation New: docs/MCP_APP.md — comprehensive guide on how this project works as an MCP application: - What an MCP App is (app that exposes business logic via MCP protocol) - Architecture diagrams: consumers (Claude, Copilot, agents) → MCP endpoint - Full protocol sequence diagram (initialize → tools/list → tools/call) - The @operation decorator: single source of truth for REST + MCP + LangChain - How to connect clients (Claude Desktop, Python, curl examples) - 4-layer architecture diagram (business logic → operations → adapters → consumers) - Extension roadmap: Resources, Prompts, SSE streaming - Security considerations table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add SchemaRenderer + visual SchemaEditor with x-ui widget system SchemaRenderer (frontend/src/features/workbench/SchemaRenderer.jsx): - Generic component: takes {data, schema} and renders each property using x-ui widget annotations - Widgets: markdown, table, badge-list, stat-card, bar-chart (Nivo), pie-chart (Nivo), json, hidden - Auto-detection when no x-ui: string→markdown, integer→stat-card, array of objects→table, array of strings→badge-list, object→json - Console debug logging, data-testid per field for E2E testing SchemaEditor (frontend/src/features/workbench/SchemaEditor.jsx): - Visual property list editor (no raw JSON editing needed) - Add/remove properties, set name/type/description - Widget picker dropdown with all available widgets - Context-sensitive options (columns for table, label for stat-card, indexBy/keys for bar-chart) - Syncs with suggest-schema: LLM suggestion populates visual editor - Outputs valid JSON Schema with x-ui annotations Backend: - DEFAULT_OUTPUT_SCHEMA now has x-ui annotations (markdown + badge-list) - suggest_schema prompt updated to suggest x-ui widgets per property Wiring: - WorkbenchPage uses SchemaRenderer for run output (replaces hardcoded) - WorkbenchPage uses SchemaEditor for create form (replaces textarea) 20 Playwright tests pass (including new SchemaRenderer widget test). 132 backend tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve suggest-schema prompt with full data domain + widget docs The suggest-schema LLM prompt now includes: - Ticket data domain (all field names, types, enum values, example cities) - Available tools with descriptions (csv_list_tickets, csv_search_tickets, etc.) - Full widget documentation with use-cases and options for each: markdown, table (columns), badge-list, stat-card (label), bar-chart (indexBy, keys), pie-chart, json, hidden - Explicit rules: always include message+referenced_tickets, match widget to data shape, use snake_case names This gives the LLM enough context to suggest schemas that actually match the ticket data (e.g. status distribution → pie-chart, ticket list → table with incident_id/summary/status columns). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix latency issues: schema title bug + recursion_limit headroom Investigation found 3 root causes for slow AI calls: 1. gpt-5-nano is a REASONING model — burns 192-832 reasoning tokens per LLM call (invisible chain-of-thought), taking 2-8s each. A simple 'say hello' costs 8.4s with 832 reasoning tokens. 2. response_format adds a 3rd LLM call — LangGraph's generate_structured_response makes a separate LLM call to format the output as JSON after the ReAct loop finishes. Without: 4.7s (2 calls). With: 13s (3 calls). 3. Missing 'title' in output_schema crashed with_structured_output. OpenAI's API requires a top-level 'title' in the JSON Schema. Fixes applied: - resolve_output_schema() now auto-adds 'title': 'AgentOutput' when missing (both default and custom schemas) - DEFAULT_OUTPUT_SCHEMA has explicit 'title' field - recursion_limit: user's setting (default 3) is now multiplied by 4 for the actual LangGraph graph, with a floor of 10. This prevents GraphRecursionError when response_format adds extra graph steps. Note: The main latency driver (reasoning tokens) is inherent to the model choice. Users can switch to gpt-4o-mini via per-agent 'model' field for ~10x faster non-reasoning responses. 133 backend + 20 Playwright tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix agent tool token bloat: compact fields + lower default limits Root cause: csv_list_tickets tool returned full Ticket objects with ALL fields (notes, description, resolution, work logs) — ~65K tokens for 100 tickets. The LLM had to process all of this, causing 30-60s per step with a reasoning model. Changes to operations.py: - csv_list_tickets: returns compact dicts (10 fields, not 30+), default limit 25 (was 100), max limit 100 (was 500) - csv_search_tickets: same compact treatment, limit 25 (was 50) - csv_get_ticket: now accepts optional 'fields' parameter for selective detail drill-down, returns dict (was full Ticket) - Tool descriptions updated to guide agents: 'use csv_get_ticket for full details' pattern Token impact per tool call: Before: 100 tickets × ~400 tokens = ~65,000 tokens After: 25 tickets × ~60 tokens = ~1,500 tokens (97% reduction) Expected latency improvement: Before: ~13s per tool call (65K token input processing) After: ~3-5s per tool call (1.5K token input) 153 tests pass (133 backend + 20 Playwright). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Drop response_format to eliminate extra LLM call LangGraph 1.0.8 implements response_format via a SEPARATE LLM call (generate_structured_response) — adding 5-10s latency per run. The refactor to inline tool-based structured output (github.com/ langchain-ai/langgraph/issues/5872) hasn't shipped yet. Fix: remove response_format from create_react_agent. The system prompt already instructs the LLM to produce JSON matching the schema (via append_output_instructions). The frontend's SchemaRenderer handles both parsed JSON and raw text gracefully. Latency impact: Before: 3 LLM calls (decide tool + answer + format JSON) ~13s After: 2 LLM calls (decide tool + answer as JSON) ~5s When LangGraph ships inline structured output, we can re-enable response_format with zero code changes (just pass it back to build_react_agent). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Enable OpenAI JSON mode for guaranteed valid JSON output Adds response_format: {type: 'json_object'} to the ChatOpenAI constructor via model_kwargs. This is a model-level setting that constrains token generation to valid JSON — no extra LLM call, no post-processing, just guaranteed JSON from every response. This is different from LangGraph's response_format parameter (which adds a separate LLM call). This is OpenAI's native JSON mode applied at the API level during the same call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert JSON mode — incompatible with non-strict tool schemas OpenAI's response_format: json_object requires all tools to have strict schemas. Our tools (from @operation decorator) don't set strict=True, causing: 'csv_search_tickets is not strict. Only strict function tools can be auto-parsed'. Reverting to prompt-only JSON enforcement, which tested at 3/3 reliability with gpt-5-nano. The frontend fallback (wraps non-JSON as {message: raw_text}) provides additional safety. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add widget E2E tests + strict tools + Agent Chat JSON mode New Playwright tests (23 total, +3): - 'renders bar-chart and pie-chart from x-ui annotations' — injects mock agent with output_schema containing x-ui widgets, verifies SVG rendering for pie/bar charts, stat-card with label, badges - 'renders raw JSON for object data' — verifies auto-detection: objects render as formatted JSON in pre blocks - 'falls back gracefully for non-JSON output' — verifies plain markdown string wraps as {message: text} and renders correctly Agent Chat (agents.py) fixes: - Added JSON output mode (response_format: json_object) - Added strict=True tool binding for compatibility - Matches the same pattern as agent_builder Strict tool binding (react_runner.py): - build_react_agent pre-binds tools with strict=True - Required for OpenAI JSON mode (response_format: json_object) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix NameError: OpenAICallLoggingCallback was removed but still referenced The class was deleted in the dead code cleanup but agents.py still used it. Replaced with make_llm_logging_callback from agent_builder. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add 'Show in Menu' — agents appear as tabs in navigation When an agent has show_in_menu=true, it appears as a tab in the main navigation bar. Clicking it opens a dedicated run page with just the input field, run button, and SchemaRenderer output. Backend: - AgentDefinition: new show_in_menu bool field (default false) - AgentDefinitionCreate/Update: show_in_menu parameter - Migration for existing DBs - Service wires it through create/update Frontend: - WorkbenchPage: 'Show in menu' checkbox in create form - App.jsx: fetches agents with show_in_menu=true, injects as tabs - AgentRunPage.jsx: simple standalone run page (title, description, optional input, run button, SchemaRenderer output) - Dynamic routes: /agent-run/{agentId} E2E test: - Creates agent via API with show_in_menu=true - Verifies tab appears in navigation with agent name - Clicks tab, verifies AgentRunPage renders - Runs agent (mocked), verifies output with SchemaRenderer 24 Playwright + 133 backend = 157 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add missing tools to chat agent: csv_sla_breach_tickets, csv_ticket_stats The SLA Breach page was slow because the chat agent (agents.py) didn't have the csv_sla_breach_tickets tool. The prompt said 'call csv_sla_breach_tickets' but the tool didn't exist, so the LLM tried to replicate SLA breach logic manually using csv_list_tickets — fetching many tickets and reasoning over them. Now the chat agent has all 6 CSV tools matching the operations: - csv_list_tickets (existing) - csv_get_ticket (existing) - csv_search_tickets (existing) - csv_ticket_fields (existing) - csv_sla_breach_tickets (NEW — pre-computed, ~1000 tokens) - csv_ticket_stats (NEW — aggregated stats, ~350 bytes) Expected improvement: 1 tool call (~1000 tokens) instead of multiple list calls + manual reasoning (~30-60K tokens). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add ticket detail modal and enhance CSV ticket table functionality Signed-off-by: Andre Bossard <anbossar@microsoft.com> * Refactor CSVTicketTable component: reorder DialogActions import for consistency Signed-off-by: Andre Bossard <anbossar@microsoft.com> * Add reasoning_effort config + new tools for major speed improvement Performance: - reasoning_effort='low' as default — reduces gpt-5-nano from 512 reasoning tokens (~7s) to 0-192 tokens (~1-3s) per LLM call - Configurable per agent: low (fast), medium, high (deep), default - Both agent_builder and legacy chat agent use reasoning_effort='low' New tools: - csv_count_tickets: count matching tickets WITHOUT returning data. Lets the LLM check 'how many VPN tickets?' (~50 tokens) before deciding to fetch details (~5000 tokens) - csv_search_tickets_with_details: search + return full details (notes, resolution, description) in ONE call. Eliminates the N × csv_get_ticket drill-down pattern that caused the 'Ticket Knowledgebase Creator' to make 5+ sequential LLM calls Impact on 'Ticket Knowledgebase Creator' agent: Before: search(compact) → get_ticket × N → generate = 5+ LLM calls × ~5s = 25s+ After: search_with_details(query, limit=10) → generate = 2 LLM calls × ~2s = 4s Also fixed: removed stale response_format: json_object from build_llm (was causing strict tool errors). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update incident details in FALL_2_HARDWARE_PERIPHERIE and FALL_3_ZUGRIFF_BERECHTIGUNG documentation for consistency and clarity Signed-off-by: Andre Bossard <anbossar@microsoft.com> * Fix: all E2E tests now clean up created agents Two tests were creating agents via the UI but not deleting them, leaving orphans in the DB after each test run: - 'runs an agent and appends output to run button' - 'requires and forwards configured run input' Added Delete button clicks at the end of both tests. All 10 agent-creating tests now verified to clean up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rewrite workbench e2e tests for tabbed UI - Add helpers: goToCreateTab, goToAgentsTab, createAgent, createAgentViaAPI, deleteAgentViaAPI, mockEmptyRuns - Update 'creates and deletes' to use Create Agent tab and agent cards - Update 'runs an agent' to verify output in RunsSidePanel - Update 'requires input' to use card inline input field + Go button - Update 'suggest schema' to navigate to Create tab first - Update 'failure handling' to check error in run detail panel - Refactor SchemaRenderer tests to use setupSchemaTest helper (API-created agents, run output in side panel) - Keep Agent Chat UI and Show in Menu tests unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: redesign workbench with agent cards, runs side panel, and tabbed layout - Split WorkbenchPage into tabbed UI: Agents (cards grid) + Create Agent - AgentCardsPanel: icon cards with Run/Edit/Delete buttons per agent - RunsSidePanel: scrollable run history with click-to-view output - AgentEditDialog: edit existing agents via dialog - AgentCreateForm: extracted creation form (reusable for create + edit) - Added API functions: updateWorkbenchAgent, listAllRuns, getRun - All 47 Playwright tests pass (12 workbench tests updated for new UI) - Removed Ollama references from setup.sh and package.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: LiteLLM fallback in agent_builder + add live lifecycle test - Fixed agent_builder/engine/react_runner.py: ChatLiteLLM when no API key - Fixed agent_builder/service.py: removed hard OpenAI key requirement - Fixed agent_builder/chat_service.py: same - Fixed RunsSidePanel output parsing for raw string output - Added full lifecycle e2e test (live LLM): create → run → edit → re-run → verify history → delete Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: suggest schema & tools, default no tools, pure function refactor - 'Suggest Schema & Tools' button: LLM suggests output schema AND tool selection - Backend: _build_suggest_prompt and _parse_suggest_response as pure functions - Frontend: tools default to empty, populated by suggest response - RunsSidePanel: pure calculations extracted (buildAgentMap, sortRunsNewestFirst, resolveOutputSchema, resolveAgentName, parseRunOutput, formatRelativeTime) - All 49 Playwright tests pass (2 live LLM tests included) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: result dialog, chart rendering, markdown fence parsing - Run results now open in a large Dialog (900px wide, 85vh max) - Fixed parseRunOutput: strips markdown code fences from LLM output - Fixed PieChartWidget: filters non-numeric values, formats labels - Fixed BarChartWidget: accepts object {key: number} in addition to arrays - Chart containers: 300px height, 600px max-width - Tests: close dialog before cleanup (dialog blocks pointer events) - All 49 Playwright tests pass Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: all-live Playwright tests, result dialog fix, runs panel fix - Rewrote workbench tests: ZERO mocks, all 8 tests use live LLM - Fixed RunsSidePanel: min-height for layout, runs visible on load - Fixed parseRunOutput: strips markdown fences from LLM output - Fixed chart widgets: pie/bar handle non-numeric values, proper sizing - Fixed dialog close: tests use X button (in viewport) not Close (scrolled) - Total: 43 tests, all passing, all live (1.1 min) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: extract shared parseRunOutput, add delete-all-runs - Extracted parseRunOutput (fence-stripping + JSON parsing) into outputUtils.js — shared by RunsSidePanel and AgentRunPage - Fixed AgentRunPage (show_in_menu): renders markdown instead of raw JSON - Added DELETE /api/workbench/runs endpoint + trash button in Runs panel - Runs panel: min-height 500px so content is visible on load Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add SSE activity monitor, settings page, agent templates & run history - Agent Activity page with real-time SSE event stream (tool calls, LLM thinking, run lifecycle), filterable by run_id via URL query param - EventBus pub/sub + StreamingCallbackHandler wired into ReAct engine - Settings page: drag-and-drop tab reorder, hide/show toggles, icon picker (57 FluentUI icons), persisted to localStorage - Agent templates dropdown (KBA from tickets, worklog stats, next step advisor) pre-fills the create agent form - AgentRunPage now shows filtered run history with detail dialog and link to Activity page filtered by run_id - 19 new Playwright E2E tests (8 activity + 11 settings) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add Support Workflow canvas page with interactive editor Purely browser-side workflow visualization using HTML Canvas: - 5 node types: Start, End, Action, Decision, Wait (each with distinct shape and color) - Drag-and-drop to reposition nodes - Shift+drag to create connections between nodes - Double-click to rename nodes inline - Animate button shows flowing dots along edges - Toolbar to add/delete nodes, reset to default workflow - Default workflow: Ticket Created → Auto-Classify → Priority decision → L1/L2 paths → Resolved decision → Close/Reopen - 9 Playwright E2E tests with screenshots Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: metro-map workflow with presets, color picker, agent assignment Rewrite WorkflowPage as metro-map style inspired by Incident & Problem Solving methodology: - 3 workflow presets: Incident Solving, Problem Solving, Change Mgmt - Metro station circle nodes with thick colored edge lines - Edge color inherited from outgoing node - Click node → dialog with color picker (8 colors) and agent selector (10 agent presets) - Agent indicator dot on nodes with assigned agents - Color legend auto-generated from used colors - 12 Playwright E2E tests covering presets, node config, animation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: friendlier workflow editor — connect mode, double-click add, dialog edges - Connect Mode toggle button: click source node then target to draw edge (no shift key needed). Crosshair cursor + green '+' hint on target. - Double-click empty canvas area to add a node at that position - Node dialog now has 'Connect to…' section with buttons for each unconnected node — draw edges without touching the canvas - Add Node button opens config dialog immediately for the new node - Dynamic help text updates based on current mode - Escape key exits connect mode - Updated Playwright tests for new UX Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add 'Improve my Prompt' button to Agent Fabric LLM-powered prompt improvement following 2025 best practices: - Backend: /api/workbench/improve-prompt endpoint + service method that rewrites prompts with clear role, goals, numbered steps, tool references, output format, and constraints - Frontend: '✨ Improve my Prompt' button below the system prompt textarea, disabled when empty, replaces prompt with improved version - 4 Playwright E2E tests with before/after screenshots Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: prompt improvement skips output format (handled by schema) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve-prompt uses selected tools, not all available Pass tool_names from frontend form state so the LLM only references tools the user actually selected for this agent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: remove maxHeight on tools list to avoid scrolling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: replace worklog template with Topic & Product Analysis Worklog columns in data.csv are all empty/zero. New template analyzes topics, products, services, priority distribution, and group workload using data that actually exists in the CSV. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Signed-off-by: Andre Bossard <anbossar@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 7ac9bb4 commit 1416147

78 files changed

Lines changed: 10664 additions & 2397 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

AGENTS_IMPLEMENTATION.md

Lines changed: 0 additions & 159 deletions
This file was deleted.

CLAUDE.md

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Commands
6+
7+
### Setup
8+
```bash
9+
./setup.sh # Bootstrap: creates .venv, installs deps, installs Playwright
10+
source .venv/bin/activate
11+
```
12+
13+
### Run (Development)
14+
```bash
15+
./start-dev.sh # Starts backend (port 5001) + frontend (port 3001) together
16+
17+
# Or manually:
18+
# Terminal 1 - Backend:
19+
source .venv/bin/activate && cd backend && python app.py
20+
21+
# Terminal 2 - Frontend:
22+
cd frontend && npm run dev
23+
```
24+
25+
### Run (Production / Docker)
26+
```bash
27+
docker build -t quart-react-demo .
28+
docker run -p 5001:5001 quart-react-demo
29+
# Quart serves both API + built frontend on port 5001
30+
```
31+
32+
### Tests
33+
```bash
34+
# E2E tests (Playwright - requires both servers running or auto-started)
35+
npm run test:e2e # Run all E2E tests (chromium, firefox, webkit)
36+
npm run test:e2e:ui # Interactive UI mode
37+
npm run test:e2e:report # View last HTML report
38+
npx playwright test tests/e2e/workbench.spec.js --project=chromium # Single file + browser
39+
40+
# Backend unit tests (pytest)
41+
cd backend
42+
pytest # All backend tests
43+
pytest agent_builder/tests/ # 132 agent framework tests
44+
pytest test_agents.py -v # Single file, verbose
45+
```
46+
47+
### Frontend Build
48+
```bash
49+
cd frontend && npm install && npm run build # Outputs to frontend/dist/
50+
```
51+
52+
## Architecture
53+
54+
### Overview
55+
Full-stack teaching app: **Quart** (async Python ASGI, port 5001) + **React/Vite** (port 3001 in dev). In development, Vite proxies `/api/*` to the Quart backend. In production, Quart serves the built frontend statically.
56+
57+
### Backend (`backend/`)
58+
59+
**Key pattern — Unified Operation System** (`api_decorators.py`):
60+
```python
61+
@operation("create_task", "Create a task", http_method="POST", mcp_enabled=True)
62+
def op_create_task(data: TaskCreate) -> Task:
63+
return TaskService.create_task(data)
64+
```
65+
The `@operation` decorator auto-generates: REST endpoint, MCP JSON-RPC method, JSON schema, and LangChain `StructuredTool`. Define once, used everywhere. Operations are registered in `operations.py`.
66+
67+
**Deep module pattern**: Business logic consolidated into service classes (`TaskService`, `WorkbenchService`, `CSVTicketService`, `KBAService`) rather than scattered thin functions.
68+
69+
**Key files:**
70+
- `app.py` — Quart app, route/blueprint registration, SSE streams, LLM config
71+
- `api_decorators.py``@operation` decorator implementation
72+
- `operations.py` — All operation definitions
73+
- `tasks.py` — SQLModel task models + `TaskService` (SQLite via `data/tasks.db`)
74+
- `csv_data.py``CSVTicketService`: loads CSV (`CSV/data.csv`), date parsing (German format `DD.MM.YYYY HH:MM:SS`), filtering, pagination
75+
- `tickets.py` — Pydantic ticket models + enums (Status, Priority, Urgency, Impact)
76+
- `agent_builder/` — Full agent lifecycle: `service.py` (WorkbenchService), `engine/` (ReAct loop), `persistence/` (SQLite repos), `tools/` (ToolRegistry, MCP adapters), `routes.py` (Blueprint at `/api/workbench/*`)
77+
- `kba_service.py` — Knowledge Base Article generation pipeline
78+
79+
**LLM config**: LiteLLM is default (works with GitHub Copilot, no key needed); OpenAI is optional override. Config via `.env`.
80+
81+
**MCP**: Backend exposes both REST and MCP JSON-RPC from the same `@operation` handlers. `mcp_handler.py` routes JSON-RPC to operations.
82+
83+
### Frontend (`frontend/src/`)
84+
85+
**Feature-first structure**: `features/<name>/` contains page component, subcomponents, utils, and local state. All backend calls go through `services/api.js`.
86+
87+
**Key features:**
88+
- `workbench/` — Agent Fabric: create/run/inspect ReAct agents with live schema editor
89+
- `csvtickets/` — Browse/filter/sort/paginate the CSV ticket dataset
90+
- `tickets/` — Nivo chart visualizations (bar, sankey, stream)
91+
- `usecase-demo/` — Pre-built demo agent runs with custom result renderers
92+
- `kba-drafter/` — KBA generation UI (draft list, editor, audit trail)
93+
- `dashboard/` — Real-time updates via Server-Sent Events (`/api/time-stream`)
94+
- `agent/` — One-shot agent chat interface
95+
96+
**UI**: FluentUI v9 (`@fluentui/react-components`) throughout. Charts via `@nivo/*`.
97+
98+
### Testing
99+
100+
E2E tests (`tests/e2e/`) use Playwright. `playwright.config.js` auto-starts backend + frontend via `webServer`. Tests use `data-testid` attributes. Three browser projects: chromium, firefox, webkit.
101+
102+
Backend tests use pytest. `agent_builder/tests/` has 132 tests covering the agent framework.
103+
104+
## Environment
105+
106+
Copy `.env.example` to `.env`. Key variables:
107+
```
108+
# LiteLLM (default — no key required if using GitHub Copilot)
109+
LITELLM_MODEL=github_copilot/gpt-4o
110+
111+
# OpenAI (optional override)
112+
OPENAI_API_KEY=sk-proj-...
113+
OPENAI_MODEL=gpt-4o-mini
114+
115+
# KBA publishing path
116+
KB_FILE_BASE_PATH=./kb_published
117+
```
118+
119+
## Ports
120+
- Backend (Quart): `http://localhost:5001`
121+
- Frontend dev (Vite): `http://localhost:3001`
122+
- In production: single port `5001` (Quart serves everything)
123+
124+
## Docs
125+
- `docs/QUICKSTART.md` — 5-minute setup
126+
- `docs/AGENT_BUILDER.md` — Agent framework architecture with diagrams
127+
- `docs/UNIFIED_ARCHITECTURE.md` — REST + MCP integration patterns
128+
- `docs/LEARNING.md` — Design principles (Grokking Simplicity, deep modules)
129+
- `docs/TROUBLESHOOTING.md` — Common issues

backend/agent_builder/__init__.py

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
"""
2+
Agent Builder — Public API
3+
4+
Import everything you need from this package:
5+
6+
from agent_builder import WorkbenchService, ChatService, ToolRegistry
7+
"""
8+
9+
from .chat_service import ChatService
10+
from .evaluator import compute_score, evaluate_run
11+
from .models import (
12+
AgentDefinition,
13+
AgentDefinitionCreate,
14+
AgentDefinitionUpdate,
15+
AgentEvaluation,
16+
AgentRequest,
17+
AgentResponse,
18+
AgentRun,
19+
AgentRunCreate,
20+
CriteriaResult,
21+
CriteriaType,
22+
RunStatus,
23+
SuccessCriteria,
24+
)
25+
from .service import WorkbenchService
26+
from .tools import ToolRegistry
27+
28+
__all__ = [
29+
# Services
30+
"ChatService",
31+
"WorkbenchService",
32+
"ToolRegistry",
33+
# Models
34+
"AgentDefinition",
35+
"AgentDefinitionCreate",
36+
"AgentDefinitionUpdate",
37+
"AgentEvaluation",
38+
"AgentRequest",
39+
"AgentResponse",
40+
"AgentRun",
41+
"AgentRunCreate",
42+
"CriteriaResult",
43+
"CriteriaType",
44+
"RunStatus",
45+
"SuccessCriteria",
46+
# Evaluator helpers
47+
"compute_score",
48+
"evaluate_run",
49+
]

0 commit comments

Comments
 (0)