Skip to content

Commit ab7b692

Browse files
apartsinclaude
apartsin
and
claude
committed
Comprehensive audit: fix 18 implementation issues, expand test coverage to 1581 tests
CRITICAL fixes: - Fix error_rate calculation using rolling window (state_manager.py) - Fix HTTP streaming to read line-by-line instead of buffering (base_provider.py) - Add context manager protocol to _StreamIterator (mesh_client.py) - Fix shutdown() to properly close async provider connectors (mesh.py) - Sync src/python/pyproject.toml version to 0.1.1 HIGH fixes: - Add missing AudioSpeech/AudioTranscription exports (interfaces/__init__.py) - Replace deprecated gpt-3.5-turbo with gpt-4o-mini (openai_provider.py) - Add type annotation to dynamic require() in mesh.ts - Add `implements ProviderConnector` to BrowserBaseProvider - Fix _modelsByid typo to _modelsById (browser-provider.ts) MEDIUM fixes: - Fix Optional[dict] type annotations (observability.py) Docs fixes: - Fix pip install modelmesh → modelmesh-lite (DeveloperGuide.md) - Fix Python version 3.10 → 3.11 (DeveloperGuide.md) - Fix install path src/python[yaml] → .[yaml] (ProxyGuide.md) - Mark unimplemented connectors and strategies as (Planned) (ConnectorCatalogue.md) Sample fixes: - Fix snake_case → camelCase in all TypeScript connector samples Test coverage: - Add 82 new Python tests for 5 previously untested modules (937 total) - Expand TypeScript tests from 97 to 644 across 13 test suites Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 5c7c0de commit ab7b692

File tree

24 files changed

+2945
-290
lines changed

24 files changed

+2945
-290
lines changed

‎docs/ConnectorCatalogue.md‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -301,13 +301,13 @@ Interface: [ConnectorInterfaces.md — Rotation Policy](ConnectorInterfaces.html
301301
| Policy | Description |
302302
| --- | --- |
303303
| **`rotation.modelmesh.stick-until-failure.v1`** | Use the current model until it fails, then rotate. Default policy. |
304-
| **`rotation.modelmesh.priority-selection.v1`** | Follow an ordered model/provider preference list; fall back on exhaust. |
305-
| **`rotation.modelmesh.round-robin.v1`** | Cycle through active models in sequence. |
306-
| **`rotation.modelmesh.cost-first.v1`** | Select the cheapest active model for each request. |
307-
| **`rotation.modelmesh.latency-first.v1`** | Select the model with the lowest observed latency. |
308-
| **`rotation.modelmesh.session-stickiness.v1`** | Route all requests in a session to the same model. |
309-
| **`rotation.modelmesh.rate-limit-aware.v1`** | Switch models preemptively before hitting rate limits. |
310-
| **`rotation.modelmesh.load-balanced.v1`** | Distribute requests proportionally to each model's rate-limit headroom. |
304+
| **`rotation.modelmesh.priority-selection.v1`** | *(Planned)* Follow an ordered model/provider preference list; fall back on exhaust. |
305+
| **`rotation.modelmesh.round-robin.v1`** | *(Planned)* Cycle through active models in sequence. |
306+
| **`rotation.modelmesh.cost-first.v1`** | *(Planned)* Select the cheapest active model for each request. |
307+
| **`rotation.modelmesh.latency-first.v1`** | *(Planned)* Select the model with the lowest observed latency. |
308+
| **`rotation.modelmesh.session-stickiness.v1`** | *(Planned)* Route all requests in a session to the same model. |
309+
| **`rotation.modelmesh.rate-limit-aware.v1`** | *(Planned)* Switch models preemptively before hitting rate limits. |
310+
| **`rotation.modelmesh.load-balanced.v1`** | *(Planned)* Distribute requests proportionally to each model's rate-limit headroom. |
311311

312312
---
313313

@@ -319,10 +319,10 @@ Interface: [ConnectorInterfaces.md — Secret Store](ConnectorInterfaces.html#se
319319
| --- | --- | --- | --- |
320320
| **`secret-store.modelmesh.env.v1`** | Reads secrets from environment variables. Default store. | Built-in | - |
321321
| **`secret-store.modelmesh.dotenv.v1`** | Loads secrets from `.env` files. Ideal for local development. | Built-in | - |
322-
| **`secret-store.aws.secrets-manager.v1`** | Managed secret storage with automatic rotation and IAM integration | 30-day trial; then $0.40/secret/month + $0.05/10K calls | [aws.amazon.com/secrets-manager](https://aws.amazon.com/secrets-manager) |
323-
| **`secret-store.google.secret-manager.v1`** | Google Cloud managed secrets with IAM and audit logging | 6 active versions free; 10K access ops/month free | [cloud.google.com/secret-manager](https://cloud.google.com/secret-manager) |
324-
| **`secret-store.microsoft.key-vault.v1`** | Microsoft cloud secret, key, and certificate management | 10K operations/month free (Standard tier) | [azure.microsoft.com/en-us/products/key-vault](https://azure.microsoft.com/en-us/products/key-vault) |
325-
| **`secret-store.1password.connect.v1`** | Secrets Automation API for CI/CD and server-side use | No free API tier; requires Business or Enterprise plan | [developer.1password.com](https://developer.1password.com) |
322+
| **`secret-store.aws.secrets-manager.v1`** | *(Planned)* Managed secret storage with automatic rotation and IAM integration | 30-day trial; then $0.40/secret/month + $0.05/10K calls | [aws.amazon.com/secrets-manager](https://aws.amazon.com/secrets-manager) |
323+
| **`secret-store.google.secret-manager.v1`** | *(Planned)* Google Cloud managed secrets with IAM and audit logging | 6 active versions free; 10K access ops/month free | [cloud.google.com/secret-manager](https://cloud.google.com/secret-manager) |
324+
| **`secret-store.microsoft.key-vault.v1`** | *(Planned)* Microsoft cloud secret, key, and certificate management | 10K operations/month free (Standard tier) | [azure.microsoft.com/en-us/products/key-vault](https://azure.microsoft.com/en-us/products/key-vault) |
325+
| **`secret-store.1password.connect.v1`** | *(Planned)* Secrets Automation API for CI/CD and server-side use | No free API tier; requires Business or Enterprise plan | [developer.1password.com](https://developer.1password.com) |
326326
| **`secret-store.modelmesh.json-secrets.v1`** | Reads secrets from a local JSON file. Keys are top-level object keys; values are strings. Supports dot-notation for nested keys. | Built-in | - |
327327
| **`secret-store.modelmesh.memory-secrets.v1`** | Holds secrets in an in-memory dictionary. Ideal for testing, scripting, and user-provided keys. Supports runtime add/remove via SecretManagement interface. | Built-in | - |
328328
| **`secret-store.modelmesh.encrypted-file.v1`** | AES-256-GCM encrypted JSON file. Secrets are decrypted at initialization using a passphrase (PBKDF2) or raw key. Supports save/load round-trips. | Built-in | - |
@@ -435,9 +435,9 @@ Interface: [ConnectorInterfaces.md — Storage](ConnectorInterfaces.html#storage
435435
| Connector | Backend | Concurrency | Free Tier | Best For | Docs |
436436
| --- | --- | --- | --- | --- | --- |
437437
| **`storage.modelmesh.local-file.v1`** | local disk | single-process only | Built-in | development, single-instance deploys | - |
438-
| **`storage.aws.s3.v1`** | AWS S3 | conditional writes | 5 GB, 20K GET, 2K PUT/month (12 months) | multi-instance, serverless | [aws.amazon.com/s3](https://aws.amazon.com/s3) |
439-
| **`storage.google.drive.v1`** | Google Drive | revision-based | 15 GB free (shared across Google services) | shared team state, client-side apps | [developers.google.com/drive](https://developers.google.com/drive) |
440-
| **`storage.redis.redis.v1`** | Redis | atomic operations | Redis Cloud 30 MB free; self-hosted open-source | low-latency multi-instance sync | [redis.io](https://redis.io) |
438+
| **`storage.aws.s3.v1`** | *(Planned)* AWS S3 | conditional writes | 5 GB, 20K GET, 2K PUT/month (12 months) | multi-instance, serverless | [aws.amazon.com/s3](https://aws.amazon.com/s3) |
439+
| **`storage.google.drive.v1`** | *(Planned)* Google Drive | revision-based | 15 GB free (shared across Google services) | shared team state, client-side apps | [developers.google.com/drive](https://developers.google.com/drive) |
440+
| **`storage.redis.redis.v1`** | *(Planned)* Redis | atomic operations | Redis Cloud 30 MB free; self-hosted open-source | low-latency multi-instance sync | [redis.io](https://redis.io) |
441441
| **`storage.modelmesh.sqlite.v1`** | SQLite | single-process only | Built-in | structured local storage, queryable state | - |
442442
| **`storage.modelmesh.memory.v1`** | in-memory | single-process only | Built-in | testing, ephemeral workloads, no persistence | - |
443443
| **`storage.modelmesh.localstorage.v1`** | browser localStorage | single-tab | Built-in (TS only) | browser apps, small state (~5-10 MB) | - |

‎docs/cdk/DeveloperGuide.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1590,7 +1590,7 @@ A chatbot that answers questions about anything. You type a question, the chatbo
15901590

15911591
### What you need
15921592

1593-
- **Python 3.10 or later** -- check by running `python --version` in your terminal
1593+
- **Python 3.11 or later** -- check by running `python --version` in your terminal
15941594
- **A free API key** from one of these providers:
15951595
- [OpenAI](https://platform.openai.com/api-keys) -- sign up and create a key starting with `sk-`
15961596
- [Groq](https://console.groq.com/keys) -- sign up for free and create a key starting with `gsk_`
@@ -1600,10 +1600,10 @@ A chatbot that answers questions about anything. You type a question, the chatbo
16001600
Open your terminal (Command Prompt on Windows, Terminal on Mac/Linux) and run:
16011601

16021602
```bash
1603-
pip install modelmesh
1603+
pip install modelmesh-lite
16041604
```
16051605

1606-
If that does not work, try `pip3 install modelmesh` or `python -m pip install modelmesh`.
1606+
If that does not work, try `pip3 install modelmesh-lite` or `python -m pip install modelmesh-lite`.
16071607

16081608
### Step 2: Set your API key
16091609

‎docs/guides/ProxyGuide.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ pip install modelmesh-lite[yaml]
4242
```bash
4343
git clone https://github.com/ApartsinProjects/ModelMesh.git
4444
cd ModelMesh
45-
pip install -e "src/python[yaml]"
45+
pip install -e ".[yaml]"
4646
```
4747

4848
### 2. Set API Keys

‎samples/connectors/typescript/customDiscovery.ts‎

Lines changed: 24 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -40,37 +40,37 @@ enum DeprecationAction {
4040

4141
/** Outcome of a registry synchronization run. */
4242
interface SyncResult {
43-
new_models: string[];
44-
deprecated_models: string[];
45-
updated_models: string[];
43+
newModels: string[];
44+
deprecatedModels: string[];
45+
updatedModels: string[];
4646
errors: string[];
4747
}
4848

4949
/** Current status of the registry synchronization process. */
5050
interface SyncStatus {
51-
last_sync?: Date;
52-
next_sync?: Date;
53-
models_synced: number;
51+
lastSync?: Date;
52+
nextSync?: Date;
53+
modelsSynced: number;
5454
status: string;
5555
}
5656

5757
/** Health assessment for a single provider over a monitoring window. */
5858
interface HealthReport {
59-
provider_id: string;
59+
providerId: string;
6060
available: boolean;
61-
latency_ms?: number;
62-
status_code?: number;
61+
latencyMs?: number;
62+
statusCode?: number;
6363
error?: string;
64-
availability_score: number;
64+
availabilityScore: number;
6565
timestamp: Date;
6666
}
6767

6868
/** Result of a single health probe against a provider. */
6969
interface ProbeResult {
70-
provider_id: string;
70+
providerId: string;
7171
success: boolean;
72-
latency_ms?: number;
73-
status_code?: number;
72+
latencyMs?: number;
73+
statusCode?: number;
7474
error?: string;
7575
}
7676

@@ -115,14 +115,14 @@ interface YamlModelEntry {
115115
name: string;
116116
provider: string;
117117
capabilities: string[];
118-
context_window: number;
119-
max_output_tokens: number;
118+
contextWindow: number;
119+
maxOutputTokens: number;
120120
status: "active" | "deprecated" | "preview";
121-
health_endpoint?: string;
121+
healthEndpoint?: string;
122122
pricing?: {
123-
input_per_1k_tokens: number;
124-
output_per_1k_tokens: number;
125-
per_request?: number;
123+
inputPer1kTokens: number;
124+
outputPer1kTokens: number;
125+
perRequest?: number;
126126
};
127127
metadata?: Record<string, unknown>;
128128
}
@@ -131,10 +131,10 @@ interface YamlModelEntry {
131131
interface YamlProviderEntry {
132132
id: string;
133133
name: string;
134-
base_url: string;
135-
health_endpoint: string;
136-
health_method?: string;
137-
health_expected_status?: number;
134+
baseUrl: string;
135+
healthEndpoint: string;
136+
healthMethod?: string;
137+
healthExpectedStatus?: number;
138138
models: string[];
139139
}
140140

@@ -191,7 +191,7 @@ class YamlDiscoveryConnector implements DiscoveryConnector {
191191

192192
/** Sync status tracking. */
193193
private syncStatus: SyncStatus = {
194-
models_synced: 0,
194+
modelsSynced: 0,
195195
status: "idle",
196196
};
197197

‎samples/connectors/typescript/customObservability.ts‎

Lines changed: 53 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -42,41 +42,41 @@ enum LogLevel {
4242

4343
/** A routing state-change event. */
4444
interface RoutingEvent {
45-
event_type: EventType;
45+
eventType: EventType;
4646
timestamp: Date;
47-
model_id?: string;
48-
provider_id?: string;
49-
pool_id?: string;
47+
modelId?: string;
48+
providerId?: string;
49+
poolId?: string;
5050
metadata: Record<string, unknown>;
5151
}
5252

5353
/** A single request/response log record. */
5454
interface RequestLogEntry {
5555
timestamp: Date;
56-
model_id: string;
57-
provider_id: string;
56+
modelId: string;
57+
providerId: string;
5858
capability: string;
59-
delivery_mode: string;
60-
latency_ms: number;
61-
status_code: number;
62-
tokens_in: number;
63-
tokens_out: number;
59+
deliveryMode: string;
60+
latencyMs: number;
61+
statusCode: number;
62+
tokensIn: number;
63+
tokensOut: number;
6464
cost?: number;
6565
error?: string;
6666
}
6767

6868
/** Aggregate metrics for a single model, provider, or pool over a time window. */
6969
interface AggregateStats {
70-
requests_total: number;
71-
requests_success: number;
72-
requests_failed: number;
73-
tokens_in: number;
74-
tokens_out: number;
75-
cost_total: number;
76-
latency_avg: number;
77-
latency_p95: number;
78-
downtime_total: number;
79-
rotation_events: number;
70+
requestsTotal: number;
71+
requestsSuccess: number;
72+
requestsFailed: number;
73+
tokensIn: number;
74+
tokensOut: number;
75+
costTotal: number;
76+
latencyAvg: number;
77+
latencyP95: number;
78+
downtimeTotal: number;
79+
rotationEvents: number;
8080
}
8181

8282
// ---------------------------------------------------------------------------
@@ -193,19 +193,19 @@ class SlackJsonObservabilityConnector implements ObservabilityConnector {
193193
// Check event filter.
194194
if (
195195
this.config.slackEventFilter &&
196-
!this.config.slackEventFilter.includes(event.event_type)
196+
!this.config.slackEventFilter.includes(event.eventType)
197197
) {
198198
return;
199199
}
200200

201201
// Also write the event to the JSON log for completeness.
202202
this.appendToLogFile({
203203
type: "event",
204-
event_type: event.event_type,
204+
eventType: event.eventType,
205205
timestamp: event.timestamp.toISOString(),
206-
model_id: event.model_id,
207-
provider_id: event.provider_id,
208-
pool_id: event.pool_id,
206+
modelId: event.modelId,
207+
providerId: event.providerId,
208+
poolId: event.poolId,
209209
metadata: event.metadata,
210210
});
211211

@@ -232,14 +232,14 @@ class SlackJsonObservabilityConnector implements ObservabilityConnector {
232232
const logRecord: Record<string, unknown> = {
233233
type: "request",
234234
timestamp: entry.timestamp.toISOString(),
235-
model_id: entry.model_id,
236-
provider_id: entry.provider_id,
235+
modelId: entry.modelId,
236+
providerId: entry.providerId,
237237
capability: entry.capability,
238-
delivery_mode: entry.delivery_mode,
239-
latency_ms: entry.latency_ms,
240-
status_code: entry.status_code,
241-
tokens_in: entry.tokens_in,
242-
tokens_out: entry.tokens_out,
238+
deliveryMode: entry.deliveryMode,
239+
latencyMs: entry.latencyMs,
240+
statusCode: entry.statusCode,
241+
tokensIn: entry.tokensIn,
242+
tokensOut: entry.tokensOut,
243243
};
244244

245245
// Include cost if present.
@@ -257,7 +257,7 @@ class SlackJsonObservabilityConnector implements ObservabilityConnector {
257257
// Alert on slow requests if threshold is configured.
258258
if (
259259
this.config.slowRequestThresholdMs &&
260-
entry.latency_ms > this.config.slowRequestThresholdMs
260+
entry.latencyMs > this.config.slowRequestThresholdMs
261261
) {
262262
this.sendSlowRequestAlert(entry);
263263
}
@@ -305,9 +305,9 @@ class SlackJsonObservabilityConnector implements ObservabilityConnector {
305305

306306
/** Format a routing event into a rich Slack Block Kit message. */
307307
private formatEventForSlack(event: RoutingEvent): SlackMessage {
308-
const emoji = this.getEventEmoji(event.event_type);
309-
const severity = this.getEventSeverity(event.event_type);
310-
const title = this.formatEventTitle(event.event_type);
308+
const emoji = this.getEventEmoji(event.eventType);
309+
const severity = this.getEventSeverity(event.eventType);
310+
const title = this.formatEventTitle(event.eventType);
311311

312312
const blocks: SlackBlock[] = [
313313
{
@@ -323,7 +323,7 @@ class SlackJsonObservabilityConnector implements ObservabilityConnector {
323323
fields: [
324324
{
325325
type: "mrkdwn",
326-
text: `*Event:*\n${event.event_type}`,
326+
text: `*Event:*\n${event.eventType}`,
327327
},
328328
{
329329
type: "mrkdwn",
@@ -335,22 +335,22 @@ class SlackJsonObservabilityConnector implements ObservabilityConnector {
335335

336336
// Add model/provider/pool info if present.
337337
const infoFields: Array<{ type: string; text: string }> = [];
338-
if (event.model_id) {
338+
if (event.modelId) {
339339
infoFields.push({
340340
type: "mrkdwn",
341-
text: `*Model:*\n\`${event.model_id}\``,
341+
text: `*Model:*\n\`${event.modelId}\``,
342342
});
343343
}
344-
if (event.provider_id) {
344+
if (event.providerId) {
345345
infoFields.push({
346346
type: "mrkdwn",
347-
text: `*Provider:*\n\`${event.provider_id}\``,
347+
text: `*Provider:*\n\`${event.providerId}\``,
348348
});
349349
}
350-
if (event.pool_id) {
350+
if (event.poolId) {
351351
infoFields.push({
352352
type: "mrkdwn",
353-
text: `*Pool:*\n\`${event.pool_id}\``,
353+
text: `*Pool:*\n\`${event.poolId}\``,
354354
});
355355
}
356356
if (infoFields.length > 0) {
@@ -408,19 +408,19 @@ class SlackJsonObservabilityConnector implements ObservabilityConnector {
408408
// Add a section for each scope.
409409
for (const [scopeId, scopeStats] of Object.entries(stats)) {
410410
const successRate =
411-
scopeStats.requests_total > 0
412-
? ((scopeStats.requests_success / scopeStats.requests_total) * 100).toFixed(1)
411+
scopeStats.requestsTotal > 0
412+
? ((scopeStats.requestsSuccess / scopeStats.requestsTotal) * 100).toFixed(1)
413413
: "N/A";
414414

415415
blocks.push({
416416
type: "section",
417417
fields: [
418418
{ type: "mrkdwn", text: `*Scope:*\n\`${scopeId}\`` },
419-
{ type: "mrkdwn", text: `*Requests:*\n${scopeStats.requests_total}` },
419+
{ type: "mrkdwn", text: `*Requests:*\n${scopeStats.requestsTotal}` },
420420
{ type: "mrkdwn", text: `*Success Rate:*\n${successRate}%` },
421-
{ type: "mrkdwn", text: `*Avg Latency:*\n${scopeStats.latency_avg.toFixed(0)}ms` },
422-
{ type: "mrkdwn", text: `*P95 Latency:*\n${scopeStats.latency_p95.toFixed(0)}ms` },
423-
{ type: "mrkdwn", text: `*Cost:*\n$${scopeStats.cost_total.toFixed(4)}` },
421+
{ type: "mrkdwn", text: `*Avg Latency:*\n${scopeStats.latencyAvg.toFixed(0)}ms` },
422+
{ type: "mrkdwn", text: `*P95 Latency:*\n${scopeStats.latencyP95.toFixed(0)}ms` },
423+
{ type: "mrkdwn", text: `*Cost:*\n$${scopeStats.costTotal.toFixed(4)}` },
424424
],
425425
});
426426
}
@@ -446,15 +446,15 @@ class SlackJsonObservabilityConnector implements ObservabilityConnector {
446446
type: "section",
447447
text: {
448448
type: "mrkdwn",
449-
text: `*Slow Request Alert*\nModel \`${entry.model_id}\` via \`${entry.provider_id}\` took *${entry.latency_ms.toFixed(0)}ms* (threshold: ${this.config.slowRequestThresholdMs}ms)`,
449+
text: `*Slow Request Alert*\nModel \`${entry.modelId}\` via \`${entry.providerId}\` took *${entry.latencyMs.toFixed(0)}ms* (threshold: ${this.config.slowRequestThresholdMs}ms)`,
450450
},
451451
},
452452
{
453453
type: "section",
454454
fields: [
455455
{ type: "mrkdwn", text: `*Capability:*\n${entry.capability}` },
456-
{ type: "mrkdwn", text: `*Status:*\n${entry.status_code}` },
457-
{ type: "mrkdwn", text: `*Tokens:*\n${entry.tokens_in} in / ${entry.tokens_out} out` },
456+
{ type: "mrkdwn", text: `*Status:*\n${entry.statusCode}` },
457+
{ type: "mrkdwn", text: `*Tokens:*\n${entry.tokensIn} in / ${entry.tokensOut} out` },
458458
],
459459
},
460460
],

0 commit comments

Comments
 (0)