Skip to content

Commit 5c111fd

Browse files
fix(llm-tools): deletion guard on tracked rewrite + actionable oneOf validation errors (SD-3287) (#3584)
Closes the two highest-volume LLM-tooling error classes from production traffic. **executor.ts — empty text nodes on tracked rewrite (1001/wk).** The word-diff prefix/suffix trim optimization (#2817/#2832) could reduce a non-empty replacement to an empty delta (e.g. "best endeavours to:" → "endeavours to:" leaves trimmedNew === ""), then the single-change branch called schema.text(""), which ProseMirror rejects with `RangeError: Empty text nodes are not allowed`. Add an `else if (trimmedNew.length === 0)` branch that uses `tr.delete(trimmedFrom, trimmedTo)` instead — same semantics as the existing tracked-review-state guard. Regression tests in tracked-rewrite.integration.test.ts cover both layers (direct executor and end-to-end doc.replace in tracked mode). The remap-length empty-replacement test was asserting the buggy `tr.replaceWith(5, 12, <empty node>)` call — the mocked schema masked the crash — corrected to assert `tr.delete(5, 12)` and `tr.replaceWith` not called. **operation-args.ts — opaque oneOf validation errors (2000+/wk cluster).** `validateValueAgainstTypeSpec`'s oneOf branch returned the unactionable `X must match one of the allowed schema variants.` for every object-union failure. LLM agents looped on the same malformed shape because the error named no specific issue. Make the branch discriminator-aware: * When object variants share a const discriminator (kind/op/type) and the value carries it matching exactly one variant, re-validate against that variant and surface its specific failure (e.g. `at.target is required.`, `target.nodeType is required.`). * When the value carries the discriminator but matches no variant, list the allowed tag values and echo what was sent. * When no discriminator is shared (or the value isn't an object), enumerate the accepted variant shapes via `describeVariant` and report the received value via `describeReceived`. Behavior is message-only: identical pass/fail across every input. Same fix benefits every oneOf-validated parameter (target/at/within/steps[]) across insert/create/lists/styles/format/query/mutations at once. Defensive details: * `return;` after the inner re-validate in the discriminator-match path (unreachable in practice — the matched variant already failed in the outer loop — but pins the invariant against future fast-path refactors). * `details.selectedError` set consistently across all three error paths (matched-variant, unmatched-tag, generic fallback). * `truncateForError` caps serialized values at 64 chars so an accidentally- large payload can't blow up logs or LLM context. Matches the same truncation pattern used by REPAIR_BLOCKED. New tests in validate-type-spec.test.ts cover real contract schemas (doc.create.paragraph:at and doc.insert:target) plus a non-discriminated repeated-error case that pins the `selectRepeatedActionableOneOfError` fallback (which the new discriminator path would otherwise mask). The existing "oneOf with mixed schemas" generic-fallback assertion was updated to the enumerated message.
1 parent 3c23c8b commit 5c111fd

5 files changed

Lines changed: 287 additions & 4 deletions

File tree

apps/cli/src/__tests__/lib/validate-type-spec.test.ts

Lines changed: 115 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,13 +61,88 @@ describe('validateValueAgainstTypeSpec – oneOf with mixed schemas', () => {
6161
oneOf: [{ const: 'block' }, { type: 'object', properties: { kind: { const: 'inline' } }, required: ['kind'] }],
6262
};
6363

64-
test('falls back to generic message when variants are not all const', () => {
64+
test('enumerates the accepted variant shapes when nothing matches', () => {
6565
try {
6666
validateValueAgainstTypeSpec('nope', mixedSchema, 'target');
6767
throw new Error('Expected CliError to be thrown');
6868
} catch (error) {
6969
const cliError = error as CliError;
70-
expect(cliError.message).toBe('target must match one of the allowed schema variants.');
70+
expect(cliError.message).toBe(
71+
'target must match one of: "block" | { kind: "inline" }. Received a string ("nope").',
72+
);
73+
}
74+
});
75+
});
76+
77+
// ---------------------------------------------------------------------------
78+
// Tagged-union (target/at) actionable errors — the dominant LLM failure class.
79+
// These exercise the REAL contract schemas so the assertions track what an
80+
// agent actually sees. See customer error pivot: "insert:target …" and
81+
// "create paragraph:at must match one of the allowed schema variants."
82+
// ---------------------------------------------------------------------------
83+
84+
describe('validateValueAgainstTypeSpec – tagged-union actionable errors (real schemas)', () => {
85+
const atSchema = CLI_OPERATION_METADATA['doc.create.paragraph'].params.find((p) => p.name === 'at')?.schema;
86+
const insertTargetSchema = CLI_OPERATION_METADATA['doc.insert'].params.find((p) => p.name === 'target')?.schema;
87+
88+
if (!atSchema || !insertTargetSchema) {
89+
throw new Error('metadata missing create.paragraph:at or insert:target schema');
90+
}
91+
92+
test('flattened "at" (nodeId hoisted out of target) names the missing field', () => {
93+
try {
94+
validateValueAgainstTypeSpec({ kind: 'after', nodeId: 'ABC123' }, atSchema, 'at');
95+
throw new Error('Expected CliError to be thrown');
96+
} catch (error) {
97+
// Was the misleading "at.nodeId is not allowed by schema."
98+
expect((error as CliError).message).toBe('at.target is required.');
99+
}
100+
});
101+
102+
test('unknown "at" kind enumerates the valid kinds and echoes what was sent', () => {
103+
try {
104+
validateValueAgainstTypeSpec({ kind: 'sidebar' }, atSchema, 'at');
105+
throw new Error('Expected CliError to be thrown');
106+
} catch (error) {
107+
// Was the misleading "at.target is required."
108+
expect((error as CliError).message).toBe(
109+
'at.kind must be one of: "documentStart", "documentEnd", "before", "after" (received "sidebar").',
110+
);
111+
}
112+
});
113+
114+
test('bare string "at" enumerates the accepted shapes instead of the opaque union error', () => {
115+
try {
116+
validateValueAgainstTypeSpec('ABC123', atSchema, 'at');
117+
throw new Error('Expected CliError to be thrown');
118+
} catch (error) {
119+
const message = (error as CliError).message;
120+
expect(message).not.toContain('allowed schema variants');
121+
expect(message).toContain('at must match one of:');
122+
expect(message).toContain('kind: "before"');
123+
expect(message).toContain('Received a string ("ABC123")');
124+
}
125+
});
126+
127+
test('insert target missing nodeType names the missing field', () => {
128+
try {
129+
validateValueAgainstTypeSpec({ kind: 'block', nodeId: 'ABC123' }, insertTargetSchema, 'target');
130+
throw new Error('Expected CliError to be thrown');
131+
} catch (error) {
132+
const message = (error as CliError).message;
133+
expect(message).not.toContain('allowed schema variants');
134+
expect(message).toContain('nodeType');
135+
}
136+
});
137+
138+
test('bare string insert target enumerates the accepted shapes', () => {
139+
try {
140+
validateValueAgainstTypeSpec('ABC123', insertTargetSchema, 'target');
141+
throw new Error('Expected CliError to be thrown');
142+
} catch (error) {
143+
const message = (error as CliError).message;
144+
expect(message).not.toContain('allowed schema variants');
145+
expect(message).toContain('target must match one of:');
71146
}
72147
});
73148
});
@@ -108,6 +183,44 @@ describe('validateValueAgainstTypeSpec – repeated actionable oneOf errors', ()
108183
});
109184
});
110185

186+
// Without a shared const discriminator across object variants, the
187+
// discriminator path cannot intercept — a repeated error must therefore surface
188+
// through `selectRepeatedActionableOneOfError`. Pins that fallback so a future
189+
// change to the discriminator gate can't silently make it unreachable.
190+
describe('validateValueAgainstTypeSpec – repeated actionable oneOf errors (no discriminator)', () => {
191+
const repeatedRequiredSchema: CliTypeSpec = {
192+
oneOf: [
193+
{
194+
type: 'object',
195+
properties: {
196+
id: { type: 'string' },
197+
name: { type: 'string' },
198+
},
199+
required: ['id'],
200+
},
201+
{
202+
type: 'object',
203+
properties: {
204+
id: { type: 'string' },
205+
label: { type: 'string' },
206+
},
207+
required: ['id'],
208+
},
209+
],
210+
};
211+
212+
test('surfaces the repeated required-field error when no discriminator exists', () => {
213+
try {
214+
validateValueAgainstTypeSpec({ name: 'x' }, repeatedRequiredSchema, 'target');
215+
throw new Error('Expected CliError to be thrown');
216+
} catch (error) {
217+
const cliError = error as CliError;
218+
expect(cliError.message).toBe('target.id is required.');
219+
expect((cliError.details as { selectedError?: string }).selectedError).toBe('target.id is required.');
220+
}
221+
});
222+
});
223+
111224
describe('validateValueAgainstTypeSpec – enum branch', () => {
112225
const enumSchema: CliTypeSpec = {
113226
type: 'string',

apps/cli/src/lib/operation-args.ts

Lines changed: 112 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,85 @@ function selectRepeatedActionableOneOfError(path: string, errors: string[]): str
146146
return bestMessage;
147147
}
148148

149+
/**
150+
* Render a variant's shape compactly so a oneOf failure can name the accepted
151+
* forms (e.g. `{ kind: "before", target }`) instead of an opaque "must match
152+
* one of the allowed schema variants". Object properties show their `const`
153+
* discriminator when present, and a trailing `?` when optional.
154+
*/
155+
function describeVariant(variant: CliTypeSpec): string {
156+
if ('const' in variant) return JSON.stringify(variant.const);
157+
if ('oneOf' in variant) return (variant.oneOf as CliTypeSpec[]).map(describeVariant).join(' | ');
158+
if (variant.enum) return variant.enum.map((entry) => JSON.stringify(entry)).join(' | ');
159+
if (variant.type === 'object') {
160+
const properties = variant.properties ?? {};
161+
const required = new Set(variant.required ?? []);
162+
const keys = Object.keys(properties);
163+
if (keys.length === 0) return 'object';
164+
const parts = keys.map((key) => {
165+
const prop = properties[key];
166+
if (prop && 'const' in prop) return `${key}: ${JSON.stringify(prop.const)}`;
167+
return required.has(key) ? key : `${key}?`;
168+
});
169+
return `{ ${parts.join(', ')} }`;
170+
}
171+
if (variant.type === 'array') return 'array';
172+
if (variant.type === 'json') return 'object';
173+
return variant.type ?? 'value';
174+
}
175+
176+
/**
177+
* The property key that carries a `const` in every object variant — the
178+
* discriminator of a tagged union (e.g. `kind` for target/at, `op` for
179+
* mutation steps). Returns null when there is no such shared key. Non-object
180+
* variants (e.g. a bare string ref) are ignored so mixed unions still resolve.
181+
*/
182+
function getOneOfDiscriminator(variants: readonly CliTypeSpec[]): string | null {
183+
const objectVariants = variants.filter(
184+
(variant): variant is Extract<CliTypeSpec, { type: 'object' }> =>
185+
!('const' in variant) && !('oneOf' in variant) && variant.type === 'object' && isRecord(variant.properties),
186+
);
187+
if (objectVariants.length < 2) return null;
188+
for (const key of Object.keys(objectVariants[0].properties)) {
189+
const sharedByAll = objectVariants.every((variant) => {
190+
const prop = variant.properties[key];
191+
return prop != null && 'const' in prop;
192+
});
193+
if (sharedByAll) return key;
194+
}
195+
return null;
196+
}
197+
198+
/** The const value a variant pins for `key`, if any (its discriminator tag). */
199+
function variantConstFor(variant: CliTypeSpec, key: string): unknown {
200+
if ('const' in variant || 'oneOf' in variant || variant.type !== 'object') return undefined;
201+
const prop = variant.properties?.[key];
202+
return prop && 'const' in prop ? prop.const : undefined;
203+
}
204+
205+
/**
206+
* Truncate a serialized value to keep oneOf error messages bounded — a caller
207+
* accidentally passing a multi-MB string as `target`/`at` shouldn't inflate
208+
* logs or the LLM context window. Matches the truncation pattern used by the
209+
* REPAIR_BLOCKED preview.
210+
*/
211+
function truncateForError(serialized: string, max = 64): string {
212+
return serialized.length > max ? `${serialized.slice(0, max)}…` : serialized;
213+
}
214+
215+
/** Human description of a received value, for oneOf error messages. */
216+
function describeReceived(value: unknown): string {
217+
if (value === null) return 'null';
218+
if (Array.isArray(value)) return 'an array';
219+
const valueType = typeof value;
220+
if (valueType === 'object') {
221+
const keys = Object.keys(value as Record<string, unknown>);
222+
return keys.length > 0 ? `an object with keys [${keys.join(', ')}]` : 'an empty object';
223+
}
224+
if (valueType === 'string') return `a string (${truncateForError(JSON.stringify(value))})`;
225+
return `a ${valueType} (${truncateForError(JSON.stringify(value))})`;
226+
}
227+
149228
export function validateValueAgainstTypeSpec(value: unknown, schema: CliTypeSpec, path: string): void {
150229
if ('const' in schema) {
151230
if (value !== schema.const) {
@@ -166,12 +245,44 @@ export function validateValueAgainstTypeSpec(value: unknown, schema: CliTypeSpec
166245
}
167246
}
168247

248+
// Tagged-union path (target/at keyed by `kind`, mutation steps keyed by
249+
// `op`): when the value carries the discriminator, surface the matching
250+
// variant's specific failure ("at.target is required.") or the set of
251+
// valid tags. This lets an LLM self-correct instead of retrying the same
252+
// malformed shape against an opaque union error.
253+
const discriminator = getOneOfDiscriminator(variants);
254+
if (discriminator && isRecord(value) && value[discriminator] !== undefined) {
255+
const received = value[discriminator];
256+
const matched = variants.find((variant) => variantConstFor(variant, discriminator) === received);
257+
if (matched) {
258+
try {
259+
validateValueAgainstTypeSpec(value, matched, path);
260+
// Unreachable in practice: `matched` already failed in the outer
261+
// variant loop above (deterministic same-input revalidation must
262+
// throw again). Explicit return so a future fast-path refactor on
263+
// this function can't silently let control fall through to the
264+
// unmatched-tag throw below with a falsely-matched value.
265+
return;
266+
} catch (error) {
267+
const message = error instanceof Error ? error.message : String(error);
268+
throw new CliError('VALIDATION_ERROR', message, { errors, selectedError: message });
269+
}
270+
}
271+
const allowedTags = variants
272+
.map((variant) => variantConstFor(variant, discriminator))
273+
.filter((tag) => tag !== undefined)
274+
.map((tag) => JSON.stringify(tag));
275+
const unmatchedTagMessage = `${path}.${discriminator} must be one of: ${allowedTags.join(', ')} (received ${truncateForError(JSON.stringify(received))}).`;
276+
throw new CliError('VALIDATION_ERROR', unmatchedTagMessage, { errors, selectedError: unmatchedTagMessage });
277+
}
278+
169279
const allowedValues = extractConstValues(variants);
170280
const selectedError = selectRepeatedActionableOneOfError(path, errors);
171281
const message =
172282
allowedValues.length > 0
173283
? `${path} must be one of: ${allowedValues.join(', ')}.`
174-
: (selectedError ?? `${path} must match one of the allowed schema variants.`);
284+
: (selectedError ??
285+
`${path} must match one of: ${variants.map(describeVariant).join(' | ')}. Received ${describeReceived(value)}.`);
175286
throw new CliError('VALIDATION_ERROR', message, { errors, selectedError });
176287
}
177288

packages/super-editor/src/editors/v1/document-api-adapters/plan-engine/executor.ts

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -998,6 +998,13 @@ export function executeTextRewrite(
998998
tr.replaceWith(remap(change.docFrom), remap(change.docTo), content);
999999
}
10001000
}
1001+
} else if (trimmedNew.length === 0) {
1002+
// Pure deletion after trimming: a non-empty replacement whose new text is
1003+
// fully contained in the old text's common prefix + suffix collapses to an
1004+
// empty delta (e.g. "best endeavours to:" → "endeavours to:" leaves
1005+
// trimmedNew === ""). Delete the removed range rather than building
1006+
// schema.text('') — ProseMirror rejects empty text nodes.
1007+
tr.delete(trimmedFrom, trimmedTo);
10011008
} else {
10021009
// 0 or 1 word change: replace just the trimmed range.
10031010
const content = buildTextWithTabs(editor.state.schema, trimmedNew, asProseMirrorMarks(marks));

packages/super-editor/src/editors/v1/document-api-adapters/plan-engine/remap-length.test.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -227,7 +227,11 @@ describe('remap correctness: mapping.map called with compiled positions', () =>
227227

228228
expect(tr.mapping.map).toHaveBeenCalledWith(3);
229229
expect(tr.mapping.map).toHaveBeenCalledWith(10);
230-
expect(tr.replaceWith).toHaveBeenCalledWith(5, 12, expect.anything());
230+
// An empty replacement collapses to a pure deletion: it must use tr.delete,
231+
// not tr.replaceWith with an empty text node (ProseMirror rejects empty
232+
// text nodes — the mocked schema previously hid that crash).
233+
expect(tr.delete).toHaveBeenCalledWith(5, 12);
234+
expect(tr.replaceWith).not.toHaveBeenCalled();
231235
});
232236

233237
it('equal length replacement: maps positions through mapping', () => {

packages/super-editor/src/editors/v1/document-api-adapters/plan-engine/tracked-rewrite.integration.test.ts

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -434,6 +434,54 @@ describe('doc.replace multi-paragraph integration', () => {
434434
expect(accepted).not.toContain('Smith address');
435435
});
436436

437+
// Customer-reported crash ("Empty text nodes are not allowed"): a non-empty
438+
// replacement whose new text is fully contained in the old text's prefix +
439+
// suffix trims to an EMPTY delta. The single-change branch must delete the
440+
// removed text rather than build schema.text('') (which ProseMirror rejects).
441+
it('rewrites a replacement that trims to an empty delta as a deletion (executor)', () => {
442+
editor = makeEditor(['the Company refers to: the following terms']);
443+
const { step, target } = compileSingleRewrite(editor, 'refers to:', 'to:');
444+
const tr = editor.state.tr;
445+
446+
const outcome = executeTextRewrite(editor, tr, target as any, step as any, tr.mapping as any);
447+
448+
expect(outcome).toEqual({ changed: true });
449+
expect(tr.doc.textContent).toBe('the Company to: the following terms');
450+
});
451+
452+
it('handles a tracked replace that trims to an empty delta (deletion only)', () => {
453+
editor = makeEditor(['We will use our best endeavours to: deliver']);
454+
const receipt = editor.doc.replace(
455+
{
456+
ref: getFirstMatchRef(editor, 'best endeavours to:'),
457+
text: 'endeavours to:',
458+
},
459+
{ changeMode: 'tracked' },
460+
);
461+
462+
expect(receipt.success).toBe(true);
463+
464+
// Accepted view: drop trackDelete marks, keep everything else.
465+
const acceptedParts: string[] = [];
466+
editor.state.doc.descendants((node: any) => {
467+
if (!node.isText || !node.text) return;
468+
const isDeleted = node.marks.some((mark: any) => mark.type.name === TrackDeleteMarkName);
469+
if (!isDeleted) acceptedParts.push(node.text);
470+
});
471+
472+
expect(acceptedParts.join('')).toBe('We will use our endeavours to: deliver');
473+
474+
// The trimmed-away prefix "best " must be represented as a tracked deletion.
475+
const deletedTexts: string[] = [];
476+
editor.state.doc.descendants((node: any) => {
477+
if (!node.isText || !node.text) return;
478+
if (node.marks.some((mark: any) => mark.type.name === TrackDeleteMarkName)) {
479+
deletedTexts.push(node.text);
480+
}
481+
});
482+
expect(deletedTexts.join('')).toContain('best');
483+
});
484+
437485
it('SD-3044: tracked rewrite of long block preserves spacing across multiple equal anchors', () => {
438486
editor = makeEditor([
439487
'[insert] Pty Limited a company incorporated in Australia having its registered office at [insert] (ACN [insert])("Company")',

0 commit comments

Comments
 (0)