Skip to content

Commit f255792

Browse files
committed
docs(uts-to-kotlin): spec-first fail-fast for UTS spec errors in Step 6
Align the skill's Step 6 with the spec manual (ably/specification#498) so a diagnosed UTS spec error is handled spec-first instead of being quietly adapted to green: - Step 6 intro now names three acceptable end-states (spec-correct pass, SDK deviation stays green, UTS spec error fails fast) — removes the earlier "a red test is never acceptable" contradiction. - Decision tree NO-branch: fix the spec at source + record under deviations.md "UTS Spec Errors" + emit a fail-fast test. - Add the "spec-error fail-fast" Kotlin pattern (mirrors the manual's JS example) and retitle the section to "Test patterns for a diagnosed failure" since fail-fast is not a deviation. - Defer the deviations-file entry format to the manual's "Recording deviations" sections (removes the divergent inline field list).
1 parent dcf733e commit f255792

1 file changed

Lines changed: 32 additions & 11 deletions

File tree

.claude/skills/uts-to-kotlin/SKILL.md

Lines changed: 32 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -420,9 +420,13 @@ Fix any compilation errors and recompile until clean. Common issues:
420420

421421
## Step 6 — Run tests *(evaluate mode only)*
422422

423-
Skip this whole step in "translate only" mode. In "translate and evaluate" mode, run the test and **keep
424-
fixing until it passes** — either the spec-correct assertion passes, or it's deliberately gated/adapted as a
425-
documented deviation (below). A red test is never an acceptable end state here.
423+
Skip this whole step in "translate only" mode. In "translate and evaluate" mode, run the test and resolve
424+
every failure via the decision tree below. Each test must end in exactly one of these states — never an
425+
**unexplained** red:
426+
- the spec-correct assertion **passes**; or
427+
- a documented **SDK deviation** — env-gated or adapted, stays green (SDK ≠ spec; fix belongs in the SDK); or
428+
- a documented **UTS spec error****fails fast** (the spec is wrong; fix belongs in the spec). This is the
429+
one acceptable red.
426430

427431
Use the per-tier task that matches the chosen tier (both are registered in `uts/build.gradle.kts`), and the
428432
resolver's `package` + the spec's `className` for the `--tests` filter:
@@ -442,15 +446,19 @@ Handle test failures using this decision tree (the **Required reading** doc you
442446
```
443447
Test fails
444448
|
445-
+-- Does UTS spec match features spec?
446-
| NO → fix test, record UTS spec error in deviations file
449+
+-- Does UTS spec match features spec? (fetch the features spec — see Required reading)
450+
| NO → UTS SPEC ERROR — fix the spec at source, not the test. Record in deviations.md
451+
| (UTS Spec Errors) + emit a fail-fast test (below) so it's fixed early.
447452
| YES
448453
| +-- Does test accurately translate the UTS spec?
449454
| NO → fix the test (no deviation entry needed)
450455
| YES → SDK deviation — adapt test, record in deviations file
451456
```
452457

453-
### Deviation patterns
458+
### Test patterns for a diagnosed failure
459+
460+
Two patterns are for an **SDK deviation** (both write the spec-correct assertions); the third,
461+
**spec-error fail-fast**, is for a **UTS spec error** and is not a deviation.
454462

455463
**Env-gated skip (preferred)** — test contains spec-correct assertions but is skipped by default:
456464

@@ -474,15 +482,28 @@ fun `RSA4c2 - callback error connecting disconnected`() = runTest {
474482
assertEquals(40160, error.errorInfo.code)
475483
```
476484

485+
**Spec-error fail-fast** *(UTS spec error — NOT an SDK deviation)* — when the decision tree's first branch is **NO** (the UTS spec contradicts the features spec, or is internally inconsistent), the spec is the fixable source of truth, so the test must **fail loudly** pointing at the deviation entry, rather than be quietly adapted to green. This is the opposite of the SDK-deviation patterns above (which stay green / assert actual behaviour) — failing fast is the forcing function that gets the spec fixed early.
486+
487+
```kotlin
488+
/**
489+
* @UTS objects/unit/RTLC7c2/local-source-no-sitetimeserials-0
490+
*/
491+
@Test
492+
fun `RTLC7c2 - LOCAL source does not write siteTimeserials`() = runTest {
493+
// SPEC ERROR RTLC7c2: replayed ACK serial "t:1:0" contradicts the harness ("ack-0:0").
494+
// Fix the UTS spec first — see deviations.md (UTS Spec Errors).
495+
fail("UTS spec error RTLC7c2 — fix the spec first; see deviations.md")
496+
}
497+
```
498+
477499
**Never use the accommodate-both pattern** (accept either spec or SDK behaviour). Every test must assert either spec behaviour or the SDK's actual behaviour — never both at once.
478500

479501
### Deviations file
480502

481-
Append to `uts/src/test/kotlin/io/ably/lib/uts/deviations.md`. Each entry needs:
482-
1. The spec point (e.g. `RSA4c2`)
483-
2. What the spec says
484-
3. What the SDK does
485-
4. Which test is affected and how it was adapted
503+
Append to `uts/src/test/kotlin/io/ably/lib/uts/deviations.md`, using the manual's **Recording deviations**
504+
entry format and sections. The ably-java-specific mapping: a **UTS Spec Error** (test fails fast — fix in
505+
the spec) goes under the manual's *UTS Spec Errors* section; an **SDK deviation** (env-gated/adapted — fix
506+
in the SDK) goes under *Failing Tests* / *Adapted Tests*.
486507

487508
---
488509

0 commit comments

Comments
 (0)