Skip to content

Commit 07444ad

Browse files
vishalveerareddy123vishal veerareddyclaude
authored
feat(routing): tier-aware fallback, output format guard, task decomposition (#77)
* feat(routing): tier-aware fallback, output format guard, task decomposition Three opt-in/always-on improvements to routing resilience and output quality, driven by real T3 Code + Lynkr integration testing. 1. Tier-aware fallback (escalate-then-demote) — src/routing/tier-fallback.js On provider failure, climb to a MORE capable tier first (toward REASONING); only if every higher tier is unavailable fall downward to SIMPLE/local. Skips circuit-OPEN providers, dedups provider:model, never re-tries the failed model. Surfaced via X-Lynkr-Fallback / X-Lynkr-Served-Tier headers + warn logs (never silent). Always on. Integrated in invokeModel's catch block. 2. Output format guard — src/context/output-format-guard.js Injects a markdown formatting instruction for non-Claude backends so weaker models avoid ASCII/Unicode box-drawing diagrams. Keyed off the routing- resolved provider/model; skips Claude-family backends. Always on. 3. Task decomposition — src/agents/decomposition/ Opt-in (TASK_DECOMPOSITION_ENABLED). Cost-aware gate decides when to split a task into isolated-context subtasks (parallel where independent), then synthesizes; confidence-scored with monolithic fallback; shadow mode + net-savings telemetry. Exposed as the DecomposeTask tool. Tests: 47 new unit tests (gate/planner/dispatcher/synthesizer, format guard, tier-fallback chain). No regressions in existing suites. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(release): 9.6.0 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: vishal veerareddy <vishalveera.reddy@servicenow.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 09977a5 commit 07444ad

22 files changed

Lines changed: 1867 additions & 5 deletions

.env.example

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -269,6 +269,18 @@ AGENTS_DEFAULT_MODEL=haiku
269269
AGENTS_MAX_STEPS=15
270270
AGENTS_TIMEOUT=300000
271271

272+
# ==============================================================================
273+
# Task Decomposition (opt-in; requires AGENTS_ENABLED=true)
274+
# ==============================================================================
275+
# Breaks complex, divisible tasks into focused subtasks run with isolated
276+
# context (parallel where independent), then synthesizes the result. A cost-aware
277+
# gate decides WHEN to decompose — decomposition can cost MORE than it saves on
278+
# small/indivisible tasks, so it only triggers on complex, large, divisible work.
279+
# Exposed as the DecomposeTask tool. All other settings (models, gate thresholds,
280+
# shadow mode) are hardcoded in src/config/index.js.
281+
282+
TASK_DECOMPOSITION_ENABLED=false
283+
272284
# ==============================================================================
273285
# MCP Sandbox Configuration
274286
# ==============================================================================

Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ FROM node:24-alpine AS runtime
2323

2424
ARG VCS_REF
2525
ARG BUILD_DATE
26-
ARG VERSION=9.5.0
26+
ARG VERSION=9.6.0
2727

2828
LABEL org.opencontainers.image.title="Lynkr" \
2929
org.opencontainers.image.description="Universal LLM proxy for Claude Code, Cursor, and AI coding tools" \

docker-compose.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ services:
33
lynkr:
44
build: .
55
container_name: lynkr
6-
image: lynkr:9.5.0
6+
image: lynkr:9.6.0
77
ports:
88
- "8081:8081"
99
extra_hosts:
@@ -329,7 +329,7 @@ services:
329329
retries: 3
330330
start_period: 40s
331331
labels:
332-
- "com.lynkr.version=9.5.0"
332+
- "com.lynkr.version=9.6.0"
333333
- "com.lynkr.description=Claude Code proxy with multi-provider support"
334334
# Uncomment to set resource limits
335335
# deploy:

package.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "lynkr",
3-
"version": "9.5.0",
3+
"version": "9.6.0",
44
"description": "Self-hosted LLM gateway and tier-routing proxy for Claude Code, Cursor, and Codex. Routes across Ollama, AWS Bedrock, OpenRouter, Databricks, Azure OpenAI, llama.cpp, and LM Studio with prompt caching, MCP tools, and 60-80% cost savings.",
55
"main": "index.js",
66
"bin": {
@@ -16,7 +16,7 @@
1616
"dev": "nodemon index.js",
1717
"lint": "eslint src index.js",
1818
"test": "npm run test:unit && npm run test:performance",
19-
"test:unit": "DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node --test test/routing.test.js test/hybrid-routing-integration.test.js test/web-tools.test.js test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js test/azure-openai-config.test.js test/azure-openai-format-conversion.test.js test/azure-openai-routing.test.js test/azure-openai-streaming.test.js test/azure-openai-error-resilience.test.js test/azure-openai-integration.test.js test/openai-integration.test.js test/toon-compression.test.js test/llamacpp-integration.test.js test/resilience.test.js test/telemetry-routing.test.js test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js test/distill.test.js test/large-payload.test.js test/code-mode.test.js test/prompt-cache-injection.test.js test/risk-analyzer.test.js test/interaction-block.test.js test/preflight.test.js test/token-reduction.test.js test/session-affinity.test.js test/model-registry-cost.test.js",
19+
"test:unit": "DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node --test test/routing.test.js test/hybrid-routing-integration.test.js test/web-tools.test.js test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js test/azure-openai-config.test.js test/azure-openai-format-conversion.test.js test/azure-openai-routing.test.js test/azure-openai-streaming.test.js test/azure-openai-error-resilience.test.js test/azure-openai-integration.test.js test/openai-integration.test.js test/toon-compression.test.js test/llamacpp-integration.test.js test/resilience.test.js test/telemetry-routing.test.js test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js test/distill.test.js test/large-payload.test.js test/code-mode.test.js test/prompt-cache-injection.test.js test/risk-analyzer.test.js test/interaction-block.test.js test/preflight.test.js test/token-reduction.test.js test/session-affinity.test.js test/model-registry-cost.test.js test/task-decomposition.test.js test/output-format-guard.test.js test/tier-fallback.test.js",
2020
"test:memory": "DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node --test test/memory/store.test.js test/memory/surprise.test.js test/memory/extractor.test.js test/memory/search.test.js test/memory/retriever.test.js",
2121
"test:new-features": "DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node --test test/passthrough-mode.test.js test/openrouter-error-resilience.test.js test/format-conversion.test.js",
2222
"test:performance": "DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node test/hybrid-routing-performance.test.js && DATABRICKS_API_KEY=test-key DATABRICKS_API_BASE=http://test.com node test/performance-tests.js",
Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
/**
2+
* Subtask dispatcher (Phase 3).
3+
*
4+
* Executes a validated plan respecting its dependency DAG:
5+
* - subtasks are grouped into topological "levels" (Kahn's algorithm)
6+
* - subtasks in the same level have no dependency on each other → run in
7+
* parallel via the existing ParallelCoordinator (spawnParallel)
8+
* - a subtask receives ONLY its own prompt plus the compressed results of the
9+
* subtasks it depends on (context isolation — the token win)
10+
*
11+
* The spawn functions are injectable for testing.
12+
*/
13+
14+
const logger = require("../../logger");
15+
16+
// Cap how much of a dependency's result we forward, to preserve the
17+
// context-isolation savings (subagents already return summaries; this bounds
18+
// pathological cases).
19+
const MAX_CONTEXT_CHARS = 2000;
20+
21+
/**
22+
* Group subtasks into dependency levels. Returns array of arrays of ids.
23+
* Throws if the graph is unresolvable (should not happen — planner validated).
24+
*/
25+
function topologicalLevels(subtasks) {
26+
const byId = new Map(subtasks.map((s) => [s.id, s]));
27+
const indegree = new Map(subtasks.map((s) => [s.id, 0]));
28+
const dependents = new Map(subtasks.map((s) => [s.id, []]));
29+
30+
for (const s of subtasks) {
31+
for (const dep of s.dependsOn) {
32+
if (!byId.has(dep)) continue;
33+
indegree.set(s.id, indegree.get(s.id) + 1);
34+
dependents.get(dep).push(s.id);
35+
}
36+
}
37+
38+
const levels = [];
39+
let frontier = subtasks.filter((s) => indegree.get(s.id) === 0).map((s) => s.id);
40+
const resolved = new Set();
41+
42+
while (frontier.length > 0) {
43+
levels.push(frontier);
44+
const next = [];
45+
for (const id of frontier) {
46+
resolved.add(id);
47+
for (const child of dependents.get(id)) {
48+
indegree.set(child, indegree.get(child) - 1);
49+
if (indegree.get(child) === 0) next.push(child);
50+
}
51+
}
52+
frontier = next;
53+
}
54+
55+
if (resolved.size !== subtasks.length) {
56+
throw new Error("Unresolvable subtask graph (cycle or dangling dependency)");
57+
}
58+
return levels;
59+
}
60+
61+
function compressResult(text) {
62+
if (typeof text !== "string") return String(text ?? "");
63+
if (text.length <= MAX_CONTEXT_CHARS) return text;
64+
return text.slice(0, MAX_CONTEXT_CHARS) + "\n…[truncated]";
65+
}
66+
67+
function buildContextForSubtask(subtask, resultsById) {
68+
if (!subtask.dependsOn || subtask.dependsOn.length === 0) return null;
69+
const parts = [];
70+
for (const dep of subtask.dependsOn) {
71+
const r = resultsById.get(dep);
72+
if (r && r.success && r.result) {
73+
parts.push(`Result of subtask ${dep}:\n${compressResult(r.result)}`);
74+
} else if (r) {
75+
parts.push(`Subtask ${dep} did not complete successfully.`);
76+
}
77+
}
78+
return parts.length > 0 ? parts.join("\n\n") : null;
79+
}
80+
81+
/**
82+
* Dispatch a validated plan.
83+
* @param {Object} plan - { subtasks: [...] }
84+
* @param {Object} [options]
85+
* @param {string} [options.sessionId]
86+
* @param {string} [options.cwd]
87+
* @param {Function} [options.spawnParallel] - (agentTypes[], prompts[], opts) => results[]
88+
* @returns {Promise<{results: Array, levels: Array, stats: Object}>}
89+
*/
90+
async function dispatchPlan(plan, options = {}) {
91+
const spawnParallel = options.spawnParallel || require("../index").spawnParallel;
92+
const subtasks = plan.subtasks;
93+
const byId = new Map(subtasks.map((s) => [s.id, s]));
94+
const levels = topologicalLevels(subtasks);
95+
const resultsById = new Map();
96+
97+
let totalInputTokens = 0;
98+
let totalOutputTokens = 0;
99+
let totalSubagents = 0;
100+
101+
for (let li = 0; li < levels.length; li++) {
102+
const levelIds = levels[li];
103+
const levelSubtasks = levelIds.map((id) => byId.get(id));
104+
105+
const agentTypes = levelSubtasks.map((s) => s.agentType);
106+
const prompts = levelSubtasks.map((s) => s.prompt);
107+
const perTaskContext = levelSubtasks.map((s) => buildContextForSubtask(s, resultsById));
108+
109+
logger.info(
110+
{ level: li, count: levelIds.length, ids: levelIds },
111+
"[Decomposition] Dispatching subtask level"
112+
);
113+
114+
// spawnParallel shares one options object; pass per-task context by spawning
115+
// the level as individual parallel calls when contexts differ.
116+
const levelResults = await runLevel(
117+
spawnParallel,
118+
agentTypes,
119+
prompts,
120+
perTaskContext,
121+
options
122+
);
123+
124+
levelResults.forEach((res, idx) => {
125+
const st = levelSubtasks[idx];
126+
totalSubagents += 1;
127+
totalInputTokens += res?.stats?.inputTokens || 0;
128+
totalOutputTokens += res?.stats?.outputTokens || 0;
129+
resultsById.set(st.id, {
130+
id: st.id,
131+
agentType: st.agentType,
132+
success: !!res?.success,
133+
result: res?.success ? res.result : null,
134+
error: res?.success ? null : res?.error || "unknown error",
135+
stats: res?.stats || {},
136+
});
137+
});
138+
}
139+
140+
const results = subtasks.map((s) => resultsById.get(s.id));
141+
return {
142+
results,
143+
levels,
144+
stats: {
145+
subagents: totalSubagents,
146+
inputTokens: totalInputTokens,
147+
outputTokens: totalOutputTokens,
148+
},
149+
};
150+
}
151+
152+
/**
153+
* Run one level. When subtasks in the level have differing injected contexts we
154+
* spawn them as separate parallel calls (each with its own mainContext), then
155+
* await all. When none need context, a single spawnParallel batch is used.
156+
*/
157+
async function runLevel(spawnParallel, agentTypes, prompts, perTaskContext, options) {
158+
const anyContext = perTaskContext.some((c) => c);
159+
160+
if (!anyContext) {
161+
return spawnParallel(agentTypes, prompts, {
162+
sessionId: options.sessionId,
163+
cwd: options.cwd,
164+
});
165+
}
166+
167+
// Mixed/with-context: one spawnParallel call per subtask so each gets its own
168+
// mainContext, executed concurrently.
169+
const calls = agentTypes.map((type, i) =>
170+
spawnParallel([type], [prompts[i]], {
171+
sessionId: options.sessionId,
172+
cwd: options.cwd,
173+
mainContext: perTaskContext[i] ? { relevant_context: perTaskContext[i] } : undefined,
174+
}).then((arr) => arr[0])
175+
);
176+
return Promise.all(calls);
177+
}
178+
179+
module.exports = {
180+
dispatchPlan,
181+
topologicalLevels,
182+
buildContextForSubtask,
183+
compressResult,
184+
MAX_CONTEXT_CHARS,
185+
};

src/agents/decomposition/gate.js

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
/**
2+
* Decomposition gate (Phase 1).
3+
*
4+
* Decides whether breaking a task into isolated-context subtasks is actually
5+
* worth it. This is the make-or-break of the feature: naive "decompose
6+
* everything" loses money, because every subagent carries fixed overhead
7+
* (planning + per-agent handoff/summarisation). Decomposition only pays off
8+
* when the task is (a) genuinely complex, (b) large enough to amortise that
9+
* overhead, and (c) divisible into reasonably independent units.
10+
*
11+
* Pure and synchronous so it can be unit-tested without a model. The caller
12+
* supplies a pre-computed complexity `analysis` (from routing/complexity-analyzer)
13+
* and the raw payload.
14+
*/
15+
16+
const DEFAULTS = {
17+
minComplexity: 60, // 0-100; only decompose genuinely complex work
18+
minTokens: 3000, // estimated monolithic tokens; below this the overhead wins
19+
minIndependentUnits: 2, // need at least 2 separable pieces to bother
20+
maxSubtasks: 6,
21+
};
22+
23+
const ENUMERATION_RE = /^\s*(?:[-*+]|\d+[.)]|step\s+\d+\b)/gim;
24+
const CONJUNCTION_RE = /\b(?:and then|then|also|additionally|as well as|after that|finally|next,)\b/gi;
25+
const IMPERATIVE_RE = /\b(?:add|create|build|implement|write|refactor|update|fix|remove|delete|migrate|test|document|configure|set up|wire|integrate|generate)\b/gi;
26+
const FILE_PATH_RE = /\b[\w./-]+\.(?:js|ts|tsx|jsx|py|go|rs|java|rb|c|cpp|h|json|yaml|yml|md|sql|sh|css|html)\b/gi;
27+
28+
function _uniqueMatches(text, re) {
29+
const set = new Set();
30+
const matches = text.match(re) || [];
31+
for (const m of matches) set.add(m.toLowerCase().trim());
32+
return set;
33+
}
34+
35+
/**
36+
* Heuristically estimate how many independent units a task contains.
37+
* Conservative: takes the strongest of several weak signals rather than summing
38+
* them, so a single rambling sentence doesn't look like five subtasks.
39+
* @param {string} text
40+
* @returns {number}
41+
*/
42+
function estimateIndependentUnits(text) {
43+
if (!text || typeof text !== "string") return 1;
44+
45+
const enumerated = (text.match(ENUMERATION_RE) || []).length;
46+
const conjunctions = (text.match(CONJUNCTION_RE) || []).length;
47+
const imperatives = _uniqueMatches(text, IMPERATIVE_RE).size;
48+
const files = _uniqueMatches(text, FILE_PATH_RE).size;
49+
50+
// Each signal is an independent lower-bound estimate of separable units.
51+
const signals = [
52+
enumerated, // explicit list items
53+
conjunctions + 1, // "do A and then B" → 2 units
54+
imperatives, // distinct action verbs
55+
files, // distinct files usually map to distinct work
56+
];
57+
58+
const estimate = Math.max(...signals, 1);
59+
return estimate;
60+
}
61+
62+
/**
63+
* Decide whether to decompose.
64+
* @param {Object} analysis - result of analyzeComplexity(payload)
65+
* @param {Object} payload - the request payload
66+
* @param {Object} [options]
67+
* @param {Object} [options.config] - threshold overrides (see DEFAULTS)
68+
* @param {string} [options.riskLevel] - 'low'|'medium'|'high'; 'high' disables decomposition
69+
* @param {string} [options.taskText] - explicit task text (else derived from analysis/payload)
70+
* @returns {{ decompose: boolean, reason: string, signals: Object }}
71+
*/
72+
function shouldDecompose(analysis, payload = {}, options = {}) {
73+
const cfg = { ...DEFAULTS, ...(options.config || {}) };
74+
75+
const score = Number(analysis?.score ?? 0);
76+
const estimatedTokens = Number(
77+
analysis?.breakdown?.tokens?.estimated ?? options.estimatedTokens ?? 0
78+
);
79+
80+
const taskText =
81+
options.taskText ||
82+
analysis?.content ||
83+
_firstUserText(payload) ||
84+
"";
85+
86+
const independentUnits = estimateIndependentUnits(taskText);
87+
88+
const signals = {
89+
score,
90+
estimatedTokens,
91+
independentUnits,
92+
riskLevel: options.riskLevel || "low",
93+
thresholds: cfg,
94+
};
95+
96+
// Never decompose high-risk work — keep it in one capable context where the
97+
// exempt-from-laziness concerns (validation/security) stay coherent.
98+
if (options.riskLevel === "high") {
99+
return { decompose: false, reason: "high_risk_skip", signals };
100+
}
101+
102+
if (score < cfg.minComplexity) {
103+
return { decompose: false, reason: "below_complexity_threshold", signals };
104+
}
105+
106+
if (estimatedTokens < cfg.minTokens) {
107+
return { decompose: false, reason: "too_small_to_amortise_overhead", signals };
108+
}
109+
110+
if (independentUnits < cfg.minIndependentUnits) {
111+
return { decompose: false, reason: "not_divisible", signals };
112+
}
113+
114+
return { decompose: true, reason: "decompose_worthwhile", signals };
115+
}
116+
117+
function _firstUserText(payload) {
118+
const messages = payload?.messages;
119+
if (!Array.isArray(messages)) return "";
120+
const user = [...messages].reverse().find((m) => m.role === "user");
121+
if (!user) return "";
122+
if (typeof user.content === "string") return user.content;
123+
if (Array.isArray(user.content)) {
124+
return user.content
125+
.filter((b) => b?.type === "text" || typeof b?.text === "string")
126+
.map((b) => b.text || "")
127+
.join("\n");
128+
}
129+
return "";
130+
}
131+
132+
module.exports = {
133+
shouldDecompose,
134+
estimateIndependentUnits,
135+
DEFAULTS,
136+
};

0 commit comments

Comments
 (0)