Skip to content

Commit 1187589

Browse files
committed
test(v5-g13): tighten V4-3 — strict data_source=parquet_tokenized + tokenizer_used basename
Closes V5-G13 / cppmega-mlx-ay6. V4-3 accepted data_source ∈ {parquet, parquet_tokenized} — let backend silently fall back to V3-2 raw-int path and still pass. Matrix fixtures have an 'original_text' column so V4-2's tokenize path MUST activate; assert strictly. Also assert extras.tokenizer_used contains '.json'. 1/1 e2e green.
1 parent 67ee407 commit 1187589

1 file changed

Lines changed: 9 additions & 4 deletions

File tree

vbgui/e2e/scenarios/29_tokenizer_train_threading.spec.ts

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,12 +46,17 @@ test("V4-3: Tokenizer Playground tokenizer reaches stage_train tokenizer_path",
4646
await modal.waitFor({ timeout: 60_000 });
4747
await page.getByTestId("run-result-expand-train").click();
4848

49-
// The T2 parquet has only input_ids (no 'text' column), so V4-2
50-
// tokenizer path falls through to V3-2 raw-int path. Either way
51-
// data_source must NOT be 'synthetic'.
49+
// G13: tighten V4-3 — matrix parquet fixtures have an
50+
// 'original_text' column (V4-2 helper recognises it as text), so
51+
// the tokenizer path MUST activate. data_source must be exactly
52+
// 'parquet_tokenized', not just non-synthetic.
5253
const dataSource = await page.getByTestId(
5354
"run-result-extras-train-data_source").textContent();
54-
expect(["parquet", "parquet_tokenized"]).toContain(dataSource?.trim());
55+
expect(dataSource?.trim()).toBe("parquet_tokenized");
56+
// tokenizer_used must contain the picked tokenizer's basename.
57+
const tokenizerUsed = await page.getByTestId(
58+
"run-result-extras-train-tokenizer_used").textContent();
59+
expect(tokenizerUsed?.trim()).toContain(".json");
5560

5661
await closeModal(page);
5762
});

0 commit comments

Comments
 (0)