Skip to content

Commit 2dfc20e

Browse files
xkd9xkd9
committed
docs: refresh README with Langfuse inference metrics and model capabilities
Made-with: Cursor
1 parent abc2454 commit 2dfc20e

1 file changed

Lines changed: 115 additions & 54 deletions

File tree

‎README.md‎

Lines changed: 115 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,12 @@ An AI-powered application that converts patient–doctor conversations into stru
3434
- [Usage Guide](#usage-guide)
3535
- [Performance Tips](#performance-tips)
3636
- [Processing Benchmarks](#processing-benchmarks)
37+
- [Inference Metrics (Langfuse)](#inference-metrics-langfuse)
3738
- [Model Capabilities](#model-capabilities)
38-
- [Whisper-1](#whisper-1)
3939
- [GPT-4o](#gpt-4o)
40+
- [Qwen2.5 (Ollama, local chat)](#qwen25-ollama-local-chat)
41+
- [Whisper-1 (OpenAI STT)](#whisper-1-openai-stt)
42+
- [Systran / faster-whisper (HF, local STT)](#systran--faster-whisper-hf-local-stt)
4043
- [Comparison Summary](#comparison-summary)
4144
- [Model Configuration](#model-configuration)
4245
- [Swapping the STT Model](#swapping-the-stt-model)
@@ -434,77 +437,135 @@ MediScriptAI/
434437

435438
## Processing Benchmarks
436439

437-
The table below shows approximate end-to-end processing times for the full pipeline (Whisper STT + GPT-4o diarization/SOAP) across different audio lengths. Times were measured on a standard broadband connection (100 Mbps upload) and reflect typical OpenAI API response times.
440+
The table below shows approximate **end-to-end UI processing times** for the full clinical pipeline (Whisper STT + chat diarization/SOAP) across different audio lengths. Times were measured on a standard broadband connection (100 Mbps upload) and reflect typical OpenAI API response times.
438441

439-
| Audio Length | File Size (MP3) | Whisper Time | GPT-4o Time | Total (approx.) |
440-
|--------------|-----------------|--------------|-------------|-----------------|
441-
| 1 minute | ~1 MB | 3–5 s | 5–8 s | 8–13 s |
442-
| 3 minutes | ~3 MB | 6–10 s | 6–10 s | 12–20 s |
443-
| 5 minutes | ~5 MB | 10–18 s | 7–12 s | 17–30 s |
444-
| 10 minutes | ~10 MB | 20–35 s | 8–15 s | 28–50 s |
442+
| Audio Length | File Size (MP3) | Whisper Time | Chat (e.g. GPT-4o) Time | Total (approx.) |
443+
|--------------|-----------------|--------------|-------------------------|-----------------|
444+
| 1 minute | ~1 MB | 3–5 s | 5–8 s | 8–13 s |
445+
| 3 minutes | ~3 MB | 6–10 s | 6–10 s | 12–20 s |
446+
| 5 minutes | ~5 MB | 10–18 s | 7–12 s | 17–30 s |
447+
| 10 minutes | ~10 MB | 20–35 s | 8–15 s | 28–50 s |
445448

446449
> **Notes:**
447450
>
448-
> - Whisper processing time scales primarily with audio file size (upload bandwidth + transcription compute). GPT-4o time scales with the number of transcript segments (input tokens), which grows more slowly than raw audio length.
449-
> - Times shown use `whisper-1` with `verbose_json` and `gpt-4o` with `json_object` response format. Switching to `gpt-4o-mini` reduces GPT-4o time by approximately 30–50% at the cost of slightly reduced diarization accuracy on short or ambiguous conversations.
451+
> - Whisper processing time scales primarily with audio file size (upload bandwidth + transcription compute). Chat time scales with the number of transcript segments (input tokens), which grows more slowly than raw audio length.
452+
> - Times shown use `whisper-1` with `verbose_json` and `gpt-4o` with `json_object` response format. Switching to `gpt-4o-mini` reduces chat time by approximately 30–50% at the cost of slightly reduced diarization accuracy on short or ambiguous conversations.
450453
> - Billing code generation (`/api/generate-billing`) is a separate lightweight call — typically 2–5 seconds regardless of original audio length, since it processes only the SOAP note text.
451454
> - OpenAI API latency varies with platform load. During peak hours, add 5–15 seconds to all estimates above. Check [status.openai.com](https://status.openai.com) if latency appears consistently elevated.
452455
453456
---
454457

455-
## Model Capabilities
458+
## Inference Metrics (Langfuse)
459+
460+
Values below are **Langfuse numeric scores** from the **MediScriptAI** project on traces named **`mediscript-benchmark`**, from **`POST /benchmark`** with `LANGFUSE_ENABLED=true` (score names: `avg_input_tokens`, `avg_output_tokens`, `avg_total_tokens_per_request`, `p50_latency_ms`, `p95_latency_ms`). Reproduce or refresh with **`BENCHMARKING_GUIDE.md`** and:
461+
462+
```bash
463+
python3 benchmarks/run_benchmark.py --url http://localhost:8000 --payload benchmarks/default_inputs.json
464+
python3 benchmarks/run_benchmark.py --url http://localhost:8000 --payload benchmarks/langfuse_smoke_inputs.json --quick
465+
```
466+
467+
Audio + STT benchmarks (`POST /benchmark/audio`) also emit Langfuse traces; add a row here when you have captured scores for that workload.
468+
469+
### Billing LLM benchmark (`POST /benchmark`)
456470

457-
### Whisper-1
471+
Equivalent to **`POST /api/generate-billing`**: SOAP JSON → CPT/ICD-10. This path exercises **chat / LLM** only (no Whisper).
458472

459-
OpenAI's production speech-to-text model, trained on 680,000 hours of multilingual audio.
473+
| Provider | Model | Deployment | Context window (effective) | Avg input tokens | Avg output tokens | Avg tokens / request | P50 latency (ms) | P95 latency (ms) |
474+
| :------- | :---- | :--------- | :------------------------- | ---------------: | ----------------: | -------------------: | ---------------: | ---------------: |
475+
| OpenAI (Cloud) | `gpt-4o` | API (Cloud) | 128K (`OPENAI_CHAT_MODEL`) | 444.84 | 307.67 | 752.5 | 6,090 | 10,590 |
476+
| Ollama (local) | `qwen2.5:3b` | Docker (`docker-compose.qwen.yml`) | 8K–32K† | 235.33 | 165.33 | 400.67 | 84,410 | 201,070 |
460477

461-
| Attribute | Details |
462-
|-------------------------|--------------------------------------------------------------------------------------------|
463-
| **Task** | Speech-to-text transcription |
464-
| **Response Format** | `verbose_json` — returns text, language, duration, and a `segments` array with timestamps |
465-
| **Languages** | 99 languages; strongest performance in English, Spanish, French, German, and Japanese |
466-
| **Audio Formats** | MP3, MP4, MPEG, MPGA, M4A, WAV, WebM |
467-
| **Max File Size** | 25 MB per request |
468-
| **Speaker Diarization** | Not native — segment timestamps are used; GPT-4o assigns speaker roles in Stage 2 |
469-
| **Noise Robustness** | Strong on clear speech; degrades with heavy background noise or strongly overlapping speech |
470-
| **Pricing** | $0.006 / minute of audio |
471-
| **Deployment** | Cloud-only — OpenAI API. No self-hosted or on-prem option |
472-
| **Configurable via** | `OPENAI_WHISPER_MODEL` environment variable |
478+
† **Context window** for Ollama is the runtime context configured for that tag; it is **not** always the model’s theoretical maximum.
479+
480+
> **Notes:**
481+
>
482+
> - **Model scope:** **`gpt-4o`** and **`qwen2.5:3b`** were used **only** for **keyword chat**—the LLM chat completions that drive **medical keyword** extraction. **Whisper** (`whisper-1`) and local **faster-whisper** handle **speech-to-text** only.
483+
> - **Methodology:** `POST /benchmark` aggregates over all `inputs[]` in the payload. Latency scores are **end-to-end per run** (wall clock) as recorded on the trace.
484+
485+
---
486+
487+
## Model Capabilities
473488

474489
### GPT-4o
475490

476-
OpenAI's flagship model, used for both clinical reasoning (Stage 2) and billing code extraction (Stage 3).
477-
478-
| Attribute | Details |
479-
|-----------------------------|----------------------------------------------------------------------------------|
480-
| **Parameters** | Not publicly disclosed |
481-
| **Architecture** | Multimodal Transformer (text + image input, text output) |
482-
| **Context Window** | 128,000 tokens input / 16,384 tokens max output |
483-
| **Structured Output** | `json_object` and strict JSON schema mode supported |
484-
| **Tool / Function Calling** | Supported; parallel function calling |
485-
| **Medical Knowledge** | Strong — trained on clinical literature, medical coding standards (CPT, ICD-10) |
486-
| **Multilingual** | Broad multilingual support across 50+ languages |
487-
| **Pricing** | $2.50 / 1M input tokens, $10.00 / 1M output tokens |
488-
| **Fine-Tuning** | Supervised fine-tuning available via OpenAI API |
489-
| **License** | Proprietary (OpenAI Terms of Use) |
490-
| **Deployment** | Cloud-only — OpenAI API or Azure OpenAI Service |
491-
| **Knowledge Cutoff** | April 2024 |
492-
| **Configurable via** | `OPENAI_CHAT_MODEL` environment variable |
491+
OpenAI’s flagship multimodal model, used in MediScript for **contextual diarization** (Doctor/Patient labels from Whisper segments), **SOAP** and **keywords**, and **billing JSON** (`json_object`). Default when `VLLM_CHAT_URL` is unset.
492+
493+
| Attribute | Details |
494+
| :-------- | :------ |
495+
| **Parameters** | Not publicly disclosed |
496+
| **Architecture** | Multimodal Transformer (text + image input, text output) |
497+
| **Context window** | 128,000 tokens input / 16,384 tokens max output |
498+
| **Reasoning mode** | Standard chat completion (no separate “thinking” toggle in our integration) |
499+
| **Tool / function calling** | Supported in general; MediScript uses **structured JSON** responses for clinical outputs |
500+
| **Structured output** | `json_object` for SOAP, utterances, keywords, and billing |
501+
| **Multilingual** | Broad multilingual support |
502+
| **Medical use** | Strong general clinical and coding knowledge; all outputs require human review |
503+
| **Pricing** | $2.50 / 1M input tokens, $10.00 / 1M output tokens (see OpenAI pricing page for current rates) |
504+
| **Fine-tuning** | Supervised fine-tuning via OpenAI API |
505+
| **License** | Proprietary (OpenAI Terms of Use) |
506+
| **Deployment** | Cloud — OpenAI API or Azure OpenAI Service |
507+
| **Knowledge cutoff** | April 2024 (per OpenAI model card) |
508+
| **Configurable via** | `OPENAI_CHAT_MODEL` |
509+
510+
### Qwen2.5 (Ollama, local chat)
511+
512+
Open-weight **chat** alternative for billing and (if configured) SOAP stages when `VLLM_CHAT_URL` points at Ollama’s OpenAI-compatible API. Default in **`docker-compose.qwen.yml`**: `qwen2.5:3b` (smaller RAM footprint in Docker).
513+
514+
| Attribute | Details |
515+
| :-------- | :------ |
516+
| **Role in MediScript** | Same OpenAI SDK path as cloud chat; **`VLLM_CHAT_MODEL`** selects the Ollama tag |
517+
| **Open weights** | Yes — Qwen2.5 family (license per model card on Hugging Face) |
518+
| **Architecture** | Dense decoder-only Transformer (Instruct-tuned) |
519+
| **Context window** | Depends on Ollama model and server settings (often 8K–32K in practice) |
520+
| **Structured output** | `json_object` / JSON billing schema (quality varies vs GPT-4o; validate outputs) |
521+
| **Quantization / edge** | Ollama serves GGUF-style bundles; suitable for **on-prem** and **air-gapped** flows |
522+
| **Multimodal (image)** | Not used in MediScript pipeline |
523+
| **Deployment** | Local — **`docker compose -f docker-compose.yml -f docker-compose.qwen.yml`** |
524+
| **Configurable via** | `VLLM_CHAT_URL`, `VLLM_CHAT_MODEL`, `VLLM_API_KEY` |
525+
526+
### Whisper-1 (OpenAI STT)
527+
528+
OpenAI’s production **speech-to-text** model for **`/api/process-audio`** when `VLLM_STT_URL` is unset.
529+
530+
| Attribute | Details |
531+
| :-------- | :------ |
532+
| **Task** | Speech-to-text transcription |
533+
| **Response format** | `verbose_json` — text, language, duration, **`segments`** with timestamps |
534+
| **Languages** | 99+ languages; strongest in high-resource locales |
535+
| **Audio formats** | MP3, MP4, MPEG, MPGA, M4A, WAV, WebM |
536+
| **Max file size** | 25 MB per request (align with `MAX_FILE_SIZE_MB`) |
537+
| **Speaker diarization** | **Not native** — MediScript uses **timestamped segments + chat model** for Doctor/Patient labels |
538+
| **Pricing** | Per-minute audio (see OpenAI pricing) |
539+
| **Deployment** | Cloud-only |
540+
| **Configurable via** | `OPENAI_WHISPER_MODEL` |
541+
542+
### Systran / faster-whisper (HF, local STT)
543+
544+
**CTranslate2** weights on the **Hugging Face Hub** (e.g. **`Systran/faster-whisper-large-v3`**) served by the repo’s **`stt-service`** when using **`docker-compose.whisper-hf.yml`**. Implements **`POST /v1/audio/transcriptions`** compatible with MediScript’s `VLLM_STT_URL` client.
545+
546+
| Attribute | Details |
547+
| :-------- | :------ |
548+
| **Task** | Local speech-to-text with **`verbose_json`-style segments** for the same downstream prompt as Whisper-1 |
549+
| **Open weights** | Yes — Hub model id (e.g. Systran/faster-whisper-*) |
550+
| **Deployment** | Docker — **`stt-whisper-hf`** service; weights cached under volume **`stt_whisper_hf_cache`** |
551+
| **Hardware** | CPU (`int8`) by default; GPU optional if you extend the image |
552+
| **Speaker diarization** | Same as Whisper-1: **segments only**; roles from **chat** model |
553+
| **Configurable via** | `WHISPER_HF_MODEL`, `VLLM_STT_URL`, `VLLM_STT_MODEL` |
493554

494555
### Comparison Summary
495556

496-
| Capability | Whisper-1 | GPT-4o |
497-
|-------------------------------|--------------------------------------|-------------------------------------------------|
498-
| Role in pipeline | Stage 1 — audio to text | Stage 2 & 3 — reasoning and coding |
499-
| Input type | Audio file (MP3, WAV, etc.) | Text (transcript segments or SOAP notes) |
500-
| Output type | Timestamped transcript segments | Structured JSON (utterances, SOAP, billing codes)|
501-
| Speaker diarization | No (timestamp segments only) | Yes (context-based role assignment) |
502-
| Medical terminology accuracy | High (trained on diverse audio) | High (clinical training data) |
503-
| Structured JSON output | No | Yes (`json_object` mode) |
504-
| Open weights / self-hosted | No | No |
505-
| Cost per typical 5-min visit | ~$0.03 | ~$0.02–$0.05 (scales with transcript length) |
506-
507-
> Both models are cloud-only and require an active OpenAI API key. Whisper handles all audio physics (timestamps, noise filtering); GPT-4o handles all clinical logic (who said what, what it means, how to code it). This separation keeps each model doing what it does best and makes it straightforward to swap either independently via environment variable.
557+
| Capability | Qwen2.5 (`qwen2.5:3b` Ollama) | GPT-4o | Whisper-1 (STT) | Systran / faster-whisper (STT) |
558+
| :--------- | :--------------------------- | :----- | :-------------- | :----------------------------- |
559+
| **JSON / structured billing** | Yes (local inference) | Yes (cloud) | N/A | N/A |
560+
| **Clinical SOAP + diarization chat** | Optional (`VLLM_CHAT_URL`) | Default cloud path | N/A | N/A |
561+
| **Timestamped STT segments** | N/A | N/A | Yes | Yes |
562+
| **On-prem / air-gapped** | Yes (chat) | No | No | Yes (STT) |
563+
| **Data sovereignty (weights local)** | Yes | No | No | Yes |
564+
| **Open weights** | Yes | No | No | Yes |
565+
| **Multimodal (image in chat)** | No | Yes (API capability; not used in default UI) | N/A | N/A |
566+
| **Typical latency (MediScript benchmarks)** | High on CPU Docker | Lower (cloud) | Cloud STT latency | CPU-bound; model-size dependent |
567+
568+
> **Summary:** MediScript’s default **product demo** path uses **OpenAI `whisper-1` + `gpt-4o`** — maximum convenience and strong JSON quality, with audio leaving the org for STT. **Qwen via Ollama** and **faster-whisper via Hugging Face** exist for **local chat and/or local STT**, which helps **regulated, air-gapped, or cost-sensitive** deployments at the expense of **latency** (especially on CPU) and sometimes **JSON robustness** compared to GPT-4o. All four can be benchmarked consistently via **`POST /benchmark`** and **`POST /benchmark/audio`** with **Langfuse** scores on trace **`mediscript-benchmark`**.
508569
509570
---
510571

0 commit comments

Comments
 (0)