Skip to content

Commit 5069277

Browse files
rui-renruiren_microsoftCopilotkunal-vaishnavi
authored and
Prathik Rao
committed
Add live audio transcription streaming support to Foundry Local JS SDK (#486)
Here's the corrected PR description with all names aligned to the actual code: --- Adds real-time audio streaming support to the Foundry Local JS SDK, enabling live microphone-to-text transcription via ONNX Runtime GenAI ASR. The existing AudioClient only supports file-based transcription. This PR introduces `LiveAudioTranscriptionSession` that accepts continuous PCM audio chunks (e.g., from a microphone) and returns partial/final transcription results as an async iterable. - `src/openai/liveAudioTranscriptionClient.ts` — Streaming session with `start()`, `append()`, `getTranscriptionStream()`, `stop()`, `dispose()` - `src/openai/liveAudioTranscriptionTypes.ts` — `LiveAudioTranscriptionResponse` and `CoreErrorResponse` interfaces, `tryParseCoreError()` helper - `src/detail/coreInterop.ts` — Added `executeCommandWithBinary()` method and `StreamingRequestBuffer` struct for binary PCM data transport - app.js — E2E example with microphone capture (naudiodon2) and synthetic audio fallback - `test/openai/liveAudioTranscription.test.ts` — Unit tests for types/settings and E2E test with synthetic PCM audio - `src/imodel.ts` — Added `createLiveTranscriptionSession()` to interface - `src/model.ts` — Delegates to `selectedVariant.createLiveTranscriptionSession()` - `src/modelVariant.ts` — Implementation (creates new `LiveAudioTranscriptionSession(modelId, coreInterop)`) - `src/index.ts` — Exports `LiveAudioTranscriptionSession`, `LiveAudioTranscriptionOptions`, `LiveAudioTranscriptionResponse`, `TranscriptionContentPart` ```js const session = model.createLiveTranscriptionSession(); session.settings.sampleRate = 16000; session.settings.channels = 1; session.settings.language = "en"; await session.start(); // Push audio from microphone callback await session.append(pcmBytes); // Read results as async iterable for await (const result of session.getTranscriptionStream()) { console.log(result.content[0].text); } await session.stop(); ``` - **Internal async push queue** — Bounded `AsyncQueue<T>` serializes audio pushes from any context (safe for mic callbacks) and provides backpressure via FIFO resolver queue. Mirrors C#'s `Channel<T>` pattern. - **Binary data transport** — `executeCommandWithBinary()` sends PCM bytes alongside JSON params via `StreamingRequestBuffer`, with transcription results parsed from push responses. - **Settings freeze** — Audio format settings are snapshot-copied and `Object.freeze()`d at `start()`, immutable during the session - **Buffer copy** — `append()` copies the input `Uint8Array` before queueing, safe when caller reuses buffers - **Drain-on-stop** — `stop()` completes the push queue, waits for the push loop to drain, parses final transcription from stop response, then completes the output stream - **Error propagation** — `start()` failures are propagated to `outputQueue` so `getTranscriptionStream()` consumers see the error; `tryParseCoreError()` handles both raw JSON and CoreInterop-prefixed error messages - **Dispose safety** — `dispose()` wraps `stop()` in try/catch, never throws This PR adds the JS SDK surface. The 3 native commands (`audio_stream_start`, `audio_stream_push`, `audio_stream_stop`) are routed through `execute_command` and the new `execute_command_with_binary` exports. The code compiles with zero TypeScript errors without the native library. - ✅ TypeScript compilation — 0 errors across all source files - ✅ Unit tests for `parseTranscriptionResult()`, `tryParseCoreError()`, `LiveAudioTranscriptionOptions` - ✅ E2E test with synthetic PCM audio (skips gracefully if native core unavailable) This implementation mirrors the C# `LiveAudioTranscriptionSession` with identical logic: - Same session lifecycle: `start` → `append` → `getStream` → `stop` - Same push loop with error handling and binary data transport - Same settings freeze and buffer copy semantics - Same drain-before-stop ordering with final result parsing - Same E2E test pattern (synthetic 440Hz sine wave, 100ms chunks, ConversationItem-shaped response validation) - Same renamed types: `LiveAudioTranscription*` (matching C# rename) --- Changes from the original: | Old (incorrect) | New (matches code) | |---|---| | `LiveAudioTranscriptionClient` | `LiveAudioTranscriptionSession` | | `LiveAudioTranscriptionSettings` | `LiveAudioTranscriptionOptions` | | `LiveAudioTranscriptionResult` | `LiveAudioTranscriptionResponse` | | `createLiveTranscriptionClient()` | `createLiveTranscriptionSession()` | --------- Co-authored-by: ruiren_microsoft <ruiren@microsoft.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: Kunal Vaishnavi <kvaishnavi@microsoft.com>
1 parent 8bbccc5 commit 5069277

11 files changed

Lines changed: 969 additions & 0 deletions

File tree

‎samples/js/live-audio-transcription-example/README.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Live Audio Transcription Example
2+
3+
Real-time microphone-to-text transcription using the Foundry Local JS SDK with Nemotron ASR.
4+
5+
## Prerequisites
6+
7+
- [Foundry Local](https://github.com/microsoft/Foundry-Local) installed
8+
- Node.js 18+
9+
- A microphone (optional — falls back to synthetic audio)
10+
11+
## Setup
12+
13+
```bash
14+
npm install foundry-local-sdk naudiodon2
15+
```
16+
17+
> **Note:** `naudiodon2` is optional — provides cross-platform microphone capture. Without it, the example falls back to synthetic audio for testing.
18+
19+
## Run
20+
21+
```bash
22+
node app.js
23+
```
24+
25+
Speak into your microphone. Transcription appears in real-time. Press `Ctrl+C` to stop.
26+
27+
## How it works
28+
29+
1. Initializes the Foundry Local SDK and loads the Nemotron ASR model
30+
2. Creates a `LiveAudioTranscriptionSession` with 16kHz/16-bit/mono PCM settings
31+
3. Captures microphone audio via `naudiodon2` (or generates synthetic audio as fallback)
32+
4. Pushes PCM chunks to the SDK via `session.append()`
33+
5. Reads transcription results via `for await (const result of session.getTranscriptionStream())`
34+
6. Access text via `result.content[0].text` (OpenAI Realtime ConversationItem pattern)
35+
36+
## API
37+
38+
```javascript
39+
const audioClient = model.createAudioClient();
40+
const session = audioClient.createLiveTranscriptionSession();
41+
session.settings.sampleRate = 16000;
42+
session.settings.channels = 1;
43+
session.settings.language = 'en';
44+
45+
await session.start();
46+
47+
// Push audio
48+
await session.append(pcmBytes);
49+
50+
// Read results
51+
for await (const result of session.getTranscriptionStream()) {
52+
console.log(result.content[0].text); // transcribed text
53+
console.log(result.content[0].transcript); // alias (OpenAI compat)
54+
console.log(result.is_final); // true for final results
55+
}
56+
57+
await session.stop();
58+
```

‎samples/js/live-audio-transcription-example/app.js‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
// Live Audio Transcription Example — Foundry Local JS SDK
2+
//
3+
// Demonstrates real-time microphone-to-text using the JS SDK.
4+
// Requires: npm install foundry-local-sdk naudiodon2
5+
//
6+
// Usage: node app.js
7+
8+
import { FoundryLocalManager } from 'foundry-local-sdk';
9+
10+
console.log('╔══════════════════════════════════════════════════════════╗');
11+
console.log('║ Foundry Local — Live Audio Transcription (JS SDK) ║');
12+
console.log('╚══════════════════════════════════════════════════════════╝');
13+
console.log();
14+
15+
// Initialize the Foundry Local SDK
16+
console.log('Initializing Foundry Local SDK...');
17+
const manager = FoundryLocalManager.create({
18+
appName: 'foundry_local_live_audio',
19+
logLevel: 'info'
20+
});
21+
console.log('✓ SDK initialized');
22+
23+
// Get and load the nemotron model
24+
const modelAlias = 'nemotron';
25+
let model = await manager.catalog.getModel(modelAlias);
26+
if (!model) {
27+
console.error(`ERROR: Model "${modelAlias}" not found in catalog.`);
28+
process.exit(1);
29+
}
30+
31+
console.log(`Found model: ${model.id}`);
32+
console.log('Downloading model (if needed)...');
33+
await model.download((progress) => {
34+
process.stdout.write(`\rDownloading... ${progress.toFixed(2)}%`);
35+
});
36+
console.log('\n✓ Model downloaded');
37+
38+
console.log('Loading model...');
39+
await model.load();
40+
console.log('✓ Model loaded');
41+
42+
// Create live transcription session
43+
const audioClient = model.createAudioClient();
44+
const session = audioClient.createLiveTranscriptionSession();
45+
session.settings.sampleRate = 16000; // Default is 16000; shown here for clarity
46+
session.settings.channels = 1;
47+
session.settings.bitsPerSample = 16;
48+
session.settings.language = 'en';
49+
50+
console.log('Starting streaming session...');
51+
await session.start();
52+
console.log('✓ Session started');
53+
54+
// Read transcription results in background
55+
const readPromise = (async () => {
56+
try {
57+
for await (const result of session.getTranscriptionStream()) {
58+
const text = result.content?.[0]?.text;
59+
if (result.is_final) {
60+
console.log();
61+
console.log(` [FINAL] ${text}`);
62+
} else if (text) {
63+
process.stdout.write(text);
64+
}
65+
}
66+
} catch (err) {
67+
if (err.name !== 'AbortError') {
68+
console.error('Stream error:', err.message);
69+
}
70+
}
71+
})();
72+
73+
// --- Microphone capture ---
74+
// This example uses naudiodon2 for cross-platform audio capture.
75+
// Install with: npm install naudiodon2
76+
//
77+
// If you prefer a different audio library, just push PCM bytes
78+
// (16-bit signed LE, mono, 16kHz) via session.append().
79+
80+
let audioInput;
81+
try {
82+
const { default: portAudio } = await import('naudiodon2');
83+
84+
audioInput = portAudio.AudioIO({
85+
inOptions: {
86+
channelCount: session.settings.channels,
87+
sampleFormat: session.settings.bitsPerSample === 16
88+
? portAudio.SampleFormat16Bit
89+
: portAudio.SampleFormat32Bit,
90+
sampleRate: session.settings.sampleRate,
91+
framesPerBuffer: 1600, // 100ms chunks
92+
maxQueue: 15 // buffer during event-loop blocks from sync FFI calls
93+
}
94+
});
95+
96+
let appendPending = false;
97+
audioInput.on('data', (buffer) => {
98+
if (appendPending) return; // drop frame while backpressured
99+
const pcm = new Uint8Array(buffer);
100+
appendPending = true;
101+
session.append(pcm).then(() => {
102+
appendPending = false;
103+
}).catch((err) => {
104+
appendPending = false;
105+
console.error('append error:', err.message);
106+
});
107+
});
108+
109+
console.log();
110+
console.log('════════════════════════════════════════════════════════════');
111+
console.log(' LIVE TRANSCRIPTION ACTIVE');
112+
console.log(' Speak into your microphone.');
113+
console.log(' Press Ctrl+C to stop.');
114+
console.log('════════════════════════════════════════════════════════════');
115+
console.log();
116+
117+
audioInput.start();
118+
} catch (err) {
119+
console.warn('⚠ Could not initialize microphone (naudiodon2 may not be installed).');
120+
console.warn(' Install with: npm install naudiodon2');
121+
console.warn(' Falling back to synthetic audio test...');
122+
console.warn();
123+
124+
// Fallback: push 2 seconds of synthetic PCM (440Hz sine wave)
125+
const sampleRate = session.settings.sampleRate;
126+
const duration = 2;
127+
const totalSamples = sampleRate * duration;
128+
const pcmBytes = new Uint8Array(totalSamples * 2);
129+
for (let i = 0; i < totalSamples; i++) {
130+
const t = i / sampleRate;
131+
const sample = Math.round(32767 * 0.5 * Math.sin(2 * Math.PI * 440 * t));
132+
pcmBytes[i * 2] = sample & 0xFF;
133+
pcmBytes[i * 2 + 1] = (sample >> 8) & 0xFF;
134+
}
135+
136+
// Push in 100ms chunks
137+
const chunkSize = (sampleRate / 10) * 2;
138+
for (let offset = 0; offset < pcmBytes.length; offset += chunkSize) {
139+
const len = Math.min(chunkSize, pcmBytes.length - offset);
140+
await session.append(pcmBytes.slice(offset, offset + len));
141+
}
142+
143+
console.log('✓ Synthetic audio pushed');
144+
}
145+
146+
// Handle graceful shutdown
147+
process.on('SIGINT', async () => {
148+
console.log('\n\nStopping...');
149+
if (audioInput) {
150+
audioInput.quit();
151+
}
152+
await session.stop();
153+
await readPromise;
154+
await model.unload();
155+
console.log('✓ Done');
156+
process.exit(0);
157+
});

‎sdk/js/src/detail/coreInterop.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,16 @@ koffi.struct('ResponseBuffer', {
1919
ErrorLength: 'int32_t',
2020
});
2121

22+
// Extended request struct for binary data (audio streaming)
23+
koffi.struct('StreamingRequestBuffer', {
24+
Command: 'char*',
25+
CommandLength: 'int32_t',
26+
Data: 'char*', // JSON params
27+
DataLength: 'int32_t',
28+
BinaryData: 'void*', // raw PCM audio bytes
29+
BinaryDataLength: 'int32_t',
30+
});
31+
2232
const CallbackType = koffi.proto('void CallbackType(void *data, int32_t length, void *userData)');
2333

2434
const __filename = fileURLToPath(import.meta.url);
@@ -28,6 +38,7 @@ export class CoreInterop {
2838
private lib: any;
2939
private execute_command: any;
3040
private execute_command_with_callback: any;
41+
private execute_command_with_binary: any = null;
3142

3243
private static _getLibraryExtension(): string {
3344
const platform = process.platform;
@@ -93,6 +104,7 @@ export class CoreInterop {
93104

94105
this.execute_command = this.lib.func('void execute_command(RequestBuffer *request, _Inout_ ResponseBuffer *response)');
95106
this.execute_command_with_callback = this.lib.func('void execute_command_with_callback(RequestBuffer *request, _Inout_ ResponseBuffer *response, CallbackType *callback, void *userData)');
107+
this.execute_command_with_binary = this.lib.func('void execute_command_with_binary(StreamingRequestBuffer *request, _Inout_ ResponseBuffer *response)');
96108
}
97109

98110
public executeCommand(command: string, params?: any): string {

‎sdk/js/src/detail/model.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,14 @@ export class Model implements IModel {
177177
return this.selectedVariant.createAudioClient();
178178
}
179179

180+
/**
181+
* Creates a LiveAudioTranscriptionSession for real-time audio streaming ASR.
182+
* @returns A LiveAudioTranscriptionSession instance.
183+
*/
184+
public createLiveTranscriptionSession(): LiveAudioTranscriptionSession {
185+
return this.selectedVariant.createLiveTranscriptionSession();
186+
}
187+
180188
/**
181189
* Creates a ResponsesClient for interacting with the model via the Responses API.
182190
* @param baseUrl - The base URL of the Foundry Local web service.

‎sdk/js/src/detail/modelVariant.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,14 @@ export class ModelVariant implements IModel {
170170
return new AudioClient(this._modelInfo.id, this.coreInterop);
171171
}
172172

173+
/**
174+
* Creates a LiveAudioTranscriptionSession for real-time audio streaming ASR.
175+
* @returns A LiveAudioTranscriptionSession instance.
176+
*/
177+
public createLiveTranscriptionSession(): LiveAudioTranscriptionSession {
178+
return new LiveAudioTranscriptionSession(this._modelInfo.id, this.coreInterop);
179+
}
180+
173181
/**
174182
* Creates a ResponsesClient for interacting with the model via the Responses API.
175183
* @param baseUrl - The base URL of the Foundry Local web service.

‎sdk/js/src/imodel.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { ChatClient } from './openai/chatClient.js';
22
import { AudioClient } from './openai/audioClient.js';
3+
import { LiveAudioTranscriptionSession } from './openai/liveAudioTranscriptionClient.js';
34
import { ResponsesClient } from './openai/responsesClient.js';
45
import { ModelInfo } from './types.js';
56

@@ -24,6 +25,13 @@ export interface IModel {
2425

2526
createChatClient(): ChatClient;
2627
createAudioClient(): AudioClient;
28+
29+
/**
30+
* Creates a LiveAudioTranscriptionSession for real-time audio streaming ASR.
31+
* The model must be loaded before calling this method.
32+
* @returns A LiveAudioTranscriptionSession instance.
33+
*/
34+
createLiveTranscriptionSession(): LiveAudioTranscriptionSession;
2735
/**
2836
* Creates a ResponsesClient for interacting with the model via the Responses API.
2937
* Unlike createChatClient/createAudioClient (which use FFI), the Responses API

‎sdk/js/src/index.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ export { ModelVariant } from './detail/modelVariant.js';
88
export type { IModel } from './imodel.js';
99
export { ChatClient, ChatClientSettings } from './openai/chatClient.js';
1010
export { AudioClient, AudioClientSettings } from './openai/audioClient.js';
11+
export { LiveAudioTranscriptionSession, LiveAudioTranscriptionOptions } from './openai/liveAudioTranscriptionClient.js';
12+
export type { LiveAudioTranscriptionResponse, TranscriptionContentPart } from './openai/liveAudioTranscriptionTypes.js';
1113
export { ResponsesClient, ResponsesClientSettings, getOutputText } from './openai/responsesClient.js';
1214
export { ModelLoadManager } from './detail/modelLoadManager.js';
1315
/** @internal */

‎sdk/js/src/openai/audioClient.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import { CoreInterop } from '../detail/coreInterop.js';
2+
import { LiveAudioTranscriptionSession } from './liveAudioTranscriptionClient.js';
23

34
export class AudioClientSettings {
45
language?: string;
@@ -56,6 +57,14 @@ export class AudioClient {
5657
this.coreInterop = coreInterop;
5758
}
5859

60+
/**
61+
* Creates a LiveAudioTranscriptionSession for real-time audio streaming ASR.
62+
* @returns A LiveAudioTranscriptionSession instance.
63+
*/
64+
public createLiveTranscriptionSession(): LiveAudioTranscriptionSession {
65+
return new LiveAudioTranscriptionSession(this.modelId, this.coreInterop);
66+
}
67+
5968
/**
6069
* Validates that the audio file path is a non-empty string.
6170
* @internal

0 commit comments

Comments
 (0)