Skip to content

Commit 3c6cf26

Browse files
goergenjCopilot
andauthored
docs: reconcile backlog/limitation status with shipped code (#134)
* docs: reconcile backlog/limitation status with shipped code Audited open backlog and "Known Limitations" entries against the actual codebase in both solutions and corrected three stale items: - ARCHITECTURE.md: mark "Voice Live Agent Mode Integration" complete. Implemented in both harness (--agent-name/--project-name, is_agent_mode, build_agent_config, agent connect()) and Container App (AgentConfig, is_agent_mode/build_agent_config in config.py, agent-mode branch in voicelive_client.py). Validated via live agent-mode E2E on both. - evaluation_harness/README.md: limitation #10 (batch-processor shared-file race) marked Fixed — per-process eval files with post-completion aggregate_evaluation_files(), matching ARCHITECTURE.md #18. - container-app/README.md: agent mode listed as a shipped feature instead of "prepared for future". No code changes; documentation only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: clarify voice/VAD are agent-owned-by-default but client-overridable in agent mode Addresses Copilot review feedback on PR #134: the Container App sends no instructions/tools in agent mode (agent owns them), but voice and turn_detection are conditionally overridden from session_config when non-default (voicelive_client.py:297-323). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 74ba362 commit 3c6cf26

3 files changed

Lines changed: 3 additions & 3 deletions

File tree

evaluation_agent/ARCHITECTURE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1348,7 +1348,7 @@ Auto-registration in `check_voicelive_job_status` attempts Foundry dataset uploa
13481348
- [x] **Media Dataset Support** ✅ — Both harness and Container App support Foundry `input_audio` media format (base64 data-URI and blob storage URLs) alongside legacy `WavPath`. Container App downloads Foundry datasets directly via `foundry_dataset` parameter. PR #43, #54.
13491349

13501350
### High Priority
1351-
- [ ] **Voice Live Agent Mode Integration** — Add support for VoiceLive Agent Mode in both the evaluation harness and the Container App. Agent Mode enables the VoiceLive service to act as an autonomous agent with tool calling, system prompts, and multi-turn conversation handling server-side, rather than requiring the client to orchestrate each turn. This changes the evaluation flow: instead of sending audio turn-by-turn and collecting responses, the client establishes an agent session and the service handles the full conversation. Requires: new session config options for agent mode, updated VoiceLive client patterns, evaluation data format adaptation for agent-driven conversations, and alignment between harness and container app implementations.
1351+
- [x] **Voice Live Agent Mode Integration** ✅ — VoiceLive Agent Mode is implemented in **both** the evaluation harness and the Container App. Agent Mode enables the VoiceLive service to act as an autonomous agent with tool calling, system prompts, and multi-turn conversation handling server-side, rather than requiring the client to orchestrate each turn. The client establishes an agent session and the service handles the full conversation. **Harness**: `--agent-name`/`--project-name`/`--agent-version` CLI options, `SessionConfig.is_agent_mode`, `build_agent_config()`, agent `connect()` in `voice_agent_audio_input_evaluation.py`. **Container App**: `AgentConfig` + `is_agent_mode`/`build_agent_config()` in `app/config.py`, agent-mode connect branch in `app/voicelive_client.py`. Validated via live agent-mode E2E on both solutions.
13521352
- [ ] **Container App: Upload Results to Foundry** — When the input was a Foundry dataset (`foundry_dataset` param), the Container App should also upload evaluation-ready results directly to Foundry Data Store, bypassing the blob intermediary. Currently the Function App handles this during `check_voicelive_job_status` polling.
13531353
- [ ] **Blob Output Cleanup** — Endpoint to delete old VoiceLive output blobs from the `outputs` container (Foundry dataset and eval group deletion already implemented)
13541354
- [ ] **Record Response Audio in Container App** — The Container App (`voicelive_client.py`) currently only records `first_audio_response_time` from `RESPONSE_AUDIO_DELTA` events but does not collect or save the actual audio chunks. The evaluation harness saves response audio as WAV files per turn, but the cloud agent pipeline does not. Adding audio output recording would enable: (a) audio quality evaluation via Foundry's `input_audio` media format, (b) playback of agent responses in the portal, (c) TTS quality assessment. Requires: collecting `RESPONSE_AUDIO_DELTA` chunks in `voicelive_client.py`, saving WAV files to blob storage, and including audio URLs or base64 in evaluation datasets.

evaluation_agent/deploy/container-app/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ This Container App runs VoiceLive audio tests on datasets and generates evaluati
2424
- **Foundry Dataset Integration**: Downloads datasets directly from Foundry Data Store via `foundry_dataset` parameter (independent of Function App — no blob staging required)
2525
- **Media Dataset Support**: Accepts Foundry media format (`input_audio` via base64 data-URI or blob URL) alongside legacy `WavPath`
2626
- **Dynamic Session Config**: Configurable via API or environment
27-
- **Extensible Design**: Prepared for future Voice Live Agent mode
27+
- **Voice Live Agent Mode**: Connects to a Foundry Agent (`agent` section in `session_config` / `AGENT_NAME`+`PROJECT_NAME`) instead of a bare model deployment; the agent owns its instructions and tools, while voice and turn detection are agent-owned by default but can be overridden via `session_config` (Entra ID auth required)
2828

2929
## API Endpoints
3030

evaluation_harness/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -546,7 +546,7 @@ Evaluation scores range from **1-5** (GPT-judge evaluators) or **0/1** (binary e
546546
7. ~~**No barge-in / interruption handling**~~**Implemented** (v1.2.0b4+): auto-truncation enabled by default (`--enable-barge-in`). Tracks `was_truncated`, `response_full`, and `barge_in` in evaluation output.
547547
8. **Evaluation polling has no timeout**`voice_agent_evaluation.py` polls indefinitely for eval run completion with no maximum attempt cap; a stuck run will block the process.
548548
9. **Evaluation output is pretty-printed JSON** — the `*_eval_output.jsonl` files use `indent=4` formatting, so each record spans multiple lines (not strict one-record-per-line JSONL).
549-
10. **Batch processor shared file writes**parallel subprocess workers append to a shared aggregate JSONL file without inter-process locking; unlikely but possible write contention under high parallelism.
549+
10. ~~**Batch processor shared file writes**~~**Fixed**: each subprocess worker writes its own per-process eval JSONL file (no shared file); results are concatenated post-completion via `aggregate_evaluation_files()` in `batch_processor.py`. No inter-process write contention.
550550

551551
## Preparing Test Datasets
552552

0 commit comments

Comments
 (0)