Skip to content

Commit 43a3ab2

Browse files
sacOO7sacOO7
committed
docs(uts): document untranslatable objects specs in private_deviations.md
Record the five objects/unit UTS specs (live_counter, live_map, object_id, objects_pool, parent_references) that cannot be translated into the :uts module. They assert on the internal CRDT graph (ObjectsPool, live nodes, applyOperation, object-id generation, parent references), which is (a) not yet implemented in :liveobjects and (b) Kotlin-`internal`, so unreachable from the :uts test module at compile time. Covers: status of all 15 objects/unit specs (10 translated, 5 blocked), why these target internals, the two blockers, and the realistic visibility options (java-test-fixtures bridge, tests in :liveobjects/src/test, or reflection) with their trade-offs and a recommendation.
1 parent e38b889 commit 43a3ab2

1 file changed

Lines changed: 286 additions & 0 deletions

File tree

‎uts/src/test/kotlin/io/ably/lib/uts/private_deviations.md‎

Lines changed: 286 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,286 @@
1+
# Private Deviations — Objects UTS specs that cannot (yet) be translated
2+
3+
> **Scope.** This file complements [`deviations.md`](./deviations.md). `deviations.md` records
4+
> *per-test* deviations inside tests that **were** translated and compile. This file records the
5+
> opposite: whole UTS spec files from the `objects` module that **could not be translated into the
6+
> `uts` module at all**, why, and what would unblock them. It is written for a human reviewer / the
7+
> LiveObjects implementers — not consumed by any tooling.
8+
9+
---
10+
11+
## 1. Status of all 15 `objects/unit` specs
12+
13+
| # | UTS spec (`objects/unit/…`) | ably-java test class | Status | Layer it targets |
14+
|---|---|---|---|---|
15+
| 1 | `instance.md` | `InstanceTest` | ✅ Translated | Public view (`Instance`) |
16+
| 2 | `live_counter.md` | `LiveCounterTest` | ⛔ **Blocked** | **Internal CRDT node** |
17+
| 3 | `live_counter_api.md` | `LiveCounterApiTest` | ✅ Translated | Public view |
18+
| 4 | `live_map.md` | `LiveMapTest` | ⛔ **Blocked** | **Internal CRDT node** |
19+
| 5 | `live_map_api.md` | `LiveMapApiTest` | ✅ Translated | Public view |
20+
| 6 | `live_object_subscribe.md` | `LiveObjectSubscribeTest` | ✅ Translated | Public view |
21+
| 7 | `object_id.md` | `ObjectIdTest` | ⛔ **Blocked** | **Internal (object-id gen)** |
22+
| 8 | `objects_pool.md` | `ObjectsPoolTest` | ⛔ **Blocked** | **Internal (`ObjectsPool`)** |
23+
| 9 | `parent_references.md` | `ParentReferencesTest` | ⛔ **Blocked** | **Internal (parent graph)** |
24+
| 10 | `path_object.md` | `PathObjectTest` | ✅ Translated | Public view (`PathObject`) |
25+
| 11 | `path_object_mutations.md` | `PathObjectMutationsTest` | ✅ Translated | Public view |
26+
| 12 | `path_object_subscribe.md` | `PathObjectSubscribeTest` | ✅ Translated | Public view |
27+
| 13 | `public_object_message.md` | `PublicObjectMessageTest` | ✅ Translated | Public message layer |
28+
| 14 | `realtime_object.md` | `RealtimeObjectTest` | ✅ Translated (mixed) | Public `get()` + sync events |
29+
| 15 | `value_types.md` | `ValueTypesTest` | ✅ Translated (mixed) | Public `create` surface |
30+
31+
**10 translated, 5 blocked.** The 5 blocked specs are the subject of this document.
32+
33+
> Note: the translated specs that depend on `setupSyncedChannel` (most of the public-view tests)
34+
> compile today but only *run* once the SDK's `OBJECT_SYNC` processing + `RealtimeObject.get()` land.
35+
> That is the same missing engine described below — see [`deviations.md`](./deviations.md) and the
36+
> `helpers.kt` header for the per-test runtime caveat. The blocked specs below are a stronger case:
37+
> they cannot even be *written*.
38+
39+
---
40+
41+
## 2. Why these 5 specs target internals
42+
43+
The objects spec is layered into three tiers (see the skill's `objects-mapping.md`):
44+
45+
1. **Creation value types** — the immutable `LiveMap` / `LiveCounter` blueprints you pass *into* `set`.
46+
2. **Public read/write view** — `PathObject` / `Instance`, what user code navigates and subscribes on.
47+
3. **Internal CRDT graph** — the live conflict-free replicated nodes, the object pool, object-id
48+
generation and the parent-reference graph. This is the convergence engine.
49+
50+
The 10 translated specs live in tiers 1–2. The 5 blocked specs **are** tier 3. They have to assert on
51+
internal state because the behaviour they pin down — last-write-wins arbitration by site-serial,
52+
idempotent re-application, tombstones, create-op merging, garbage collection, object-id derivation —
53+
is **not observable through the public API**. You cannot verify "the second of two concurrent ops
54+
loses by site-serial" with `get()`/`value()`; you have to reach the node's `siteTimeserials` and call
55+
`applyOperation` directly. So the spec is correct to test internals — that is where the hard logic is.
56+
57+
---
58+
59+
## 3. The two blockers (in order of severity)
60+
61+
### Blocker A — the internal implementation does not exist yet *(primary)*
62+
63+
`:liveobjects` currently implements the **view** layer only. A symbol search of
64+
`liveobjects/src/main/kotlin/io/ably/lib/liveobjects` confirms the CRDT engine these specs assert on
65+
is absent:
66+
67+
| Symbol required by the blocked specs | Found in `:liveobjects`? |
68+
|---|---|
69+
| `ObjectsPool` (the live object pool) | ❌ 0 references |
70+
| `generateObjectId` / object-id derivation (`RTO14`) | ❌ 0 references |
71+
| `applyOperation(...)` (apply op to a live node) | ❌ 0 references |
72+
| `replaceData(...)` | ❌ 0 references |
73+
| `createOperationIsMerged` | ❌ 0 references |
74+
| parent-reference graph (`parentRef…`) | ❌ 0 references |
75+
| pool `syncState` | ❌ 0 references |
76+
| `siteTimeserials` | ⚠️ only on the **wire DTO** (`WireObjectState` / `WireObjectsMapEntry`), not on a live CRDT node |
77+
78+
What *does* exist: `DefaultPathObject`, `DefaultInstance`, the typed `Default*PathObject` /
79+
`Default*Instance` views, the `value/` creation types, and the `message/` + `serialization/` wire layer.
80+
There is **no live `InternalLiveMap` / `InternalLiveCounter` node, no `ObjectsPool`, and no
81+
operation-application engine.**
82+
83+
**Consequence:** even with perfect cross-module visibility there is nothing to instantiate or assert
84+
against. These tests cannot be authored until the engine is implemented.
85+
86+
### Blocker B — Kotlin `internal` is not visible across the module boundary *(secondary, applies once A is done)*
87+
88+
When the engine *is* implemented it will (by the codebase's convention, and because `:liveobjects`
89+
uses `explicitApi()`) be declared `internal` — exactly like the existing `Default*` classes
90+
(`internal class DefaultLiveMap`, `internal class DefaultPathObject`, …).
91+
92+
Kotlin's `internal` is scoped to a **module** = one compilation unit = one Gradle source set's compile
93+
task. The `:uts` test source set is a *different* module from `:liveobjects`'s `main`. The Kotlin
94+
compiler enforces `internal` across that boundary **regardless of dependency classpath scope**. So
95+
`:uts` test code cannot name those declarations at compile time.
96+
97+
This is why the existing helper `buildPublicObjectMessage` (in `helpers.kt`) reaches the internal
98+
wire/message classes by **reflection** (`Class.forName(...)`), enabled by the current
99+
`testRuntimeOnly(project(":liveobjects"))` — runtime-only access. Reflection works for a handful of
100+
constructor/field hops but is the wrong tool for whole-CRDT-state assertions (no type safety, brittle,
101+
unreadable).
102+
103+
---
104+
105+
## 4. Per-spec detail
106+
107+
| Spec | What it asserts on | Required internal symbols | Blocked by |
108+
|---|---|---|---|
109+
| **2 `live_counter.md`** | internal counter node state after applying ops | `InternalLiveCounter` (`.data`, `.siteTimeserials`, `.createOperationIsMerged`, `applyOperation`, `replaceData`) | A + B |
110+
| **4 `live_map.md`** | internal map node state after applying ops | `InternalLiveMap` (`.data`, `.siteTimeserials`, `.isTombstone`, `applyOperation`, `replaceData`) | A + B |
111+
| **7 `object_id.md`** | object-id generation & parsing | `generateObjectId` / object-id type (`RTO14`), `*WithObjectId` derivation | A + B |
112+
| **8 `objects_pool.md`** | the object pool and its sync lifecycle | `ObjectsPool`, `.syncState`, pool entry add/get/clear | A + B |
113+
| **9 `parent_references.md`** | the reverse parent-reference graph | parent-reference tracking on the pool/nodes | A + B |
114+
115+
> Specs 2 and 4 have public counterparts (`live_counter_api.md` / `live_map_api.md`, both translated)
116+
> that cover the *outcome* of these operations through the public API. Specs 7–9 have **no** public
117+
> counterpart — they are purely internal and have no representation in tiers 1–2.
118+
119+
---
120+
121+
## 5. Solution options
122+
123+
The ask was to make all of `liveobjects/src/main/kotlin/io/ably/lib/liveobjects` visible to `:uts`,
124+
probably by changing `testRuntimeOnly(project(":liveobjects"))` to `testImplementation(...)`. Here is
125+
the accurate picture, as lead dev.
126+
127+
### 5.1 Why the `testRuntimeOnly` → `testImplementation` swap alone is *not* sufficient
128+
129+
```kotlin
130+
// uts/build.gradle.kts — current
131+
testRuntimeOnly(project(":liveobjects")) // runtime classpath only → reflection-only access
132+
133+
// proposed
134+
testImplementation(project(":liveobjects")) // adds COMPILE classpath too
135+
```
136+
137+
`testImplementation` puts `:liveobjects` on the **compile** classpath, which lets `:uts` reference its
138+
**public** API directly (and lets the reflection helpers drop some `Class.forName`). But it does **not**
139+
grant access to `internal` declarations: Kotlin enforces `internal` at the *module* boundary at
140+
compile time, and a dependency-scope change does not cross that boundary. The CRDT engine will be
141+
`internal`, so the swap by itself does not unblock these tests. It is **necessary but not sufficient.**
142+
143+
### 5.2 The Gradle/Kotlin configs that *can* expose internals
144+
145+
There is exactly **one** primitive that grants one Kotlin compilation access to another's `internal`
146+
declarations: the compiler flag **`-Xfriend-paths`**. Everything below is either that flag directly, or
147+
a higher-level wrapper around it.
148+
149+
**(a) `associateWith()` — the *supported* form, but intra-project only.**
150+
The Kotlin Gradle plugin exposes friend-paths through the `associateWith` API on compilations:
151+
152+
```kotlin
153+
kotlin.target.compilations.getByName("test")
154+
.associateWith(kotlin.target.compilations.getByName("main"))
155+
```
156+
157+
This is how a module's own `test` source set sees its `main` internals (the plugin wires it up
158+
automatically), and how you'd give a *custom* source set (e.g. `integrationTest`) the same access. It is
159+
stable and IDE-aware — **but only between compilations of the same Gradle project.** There is no
160+
supported way to `associateWith` a compilation in a *different* project (`:uts` test ↔ `:liveobjects`
161+
main). Source: KTIJ-7662, KT associated-compilations docs.
162+
163+
**(b) Raw `-Xfriend-paths` across projects — works, but unstable/unsupported.**
164+
You can manually point `:uts`'s test-compile task at `:liveobjects`'s `main` output:
165+
`-Xfriend-paths=…/liveobjects/build/classes/kotlin/main`. Per the Kotlin team this flag has *"no syntax,
166+
no IDE support, and no guarantees of stability — a compiler implementation detail, not a language
167+
feature."* It also hard-couples `:uts` to `:liveobjects`'s internal compile output path. **Not
168+
recommended** for production build config.
169+
170+
**(c) The future fix (not available yet): `shared internal` (KEEP-0451).**
171+
The Kotlin team is *not* stabilizing `-Xfriend-paths`; instead KEEP-0451 proposes a first-class
172+
`shared internal` visibility modifier — declarations visible to designated dependent modules but not
173+
the general public. When it ships this is the clean answer, but it is a proposal today, not usable.
174+
175+
### 5.3 Cleanest technical approach — write the internal-graph tests in `:liveobjects`'s own test source set
176+
177+
> This is the lowest-ceremony option and what the SDK already does for its own internals, but it places
178+
> the tests **outside `uts/unit`**. If keeping them under `uts/unit` is required, prefer §5.4(a) instead.
179+
180+
`:liveobjects` **already has** a unit-test source set and task:
181+
182+
```kotlin
183+
// liveobjects/build.gradle.kts (existing)
184+
tasks.register<Test>("runLiveObjectsUnitTests") {
185+
filter { includeTestsMatching("io.ably.lib.liveobjects.unit.*") }
186+
}
187+
```
188+
189+
Tests placed under `liveobjects/src/test/kotlin/io/ably/lib/liveobjects/unit/…` see **all** of `main`'s
190+
`internal` declarations automatically (the plugin sets the friend-path for a module's own tests). This
191+
is the standard, supported way to test internal Kotlin code.
192+
193+
**Plan once Blocker A is resolved (engine implemented):**
194+
1. Author specs 2, 4, 7, 8, 9 in `:liveobjects`'s own test source set, e.g. package
195+
`io.ably.lib.liveobjects.unit.uts`, mirroring the `uts` conventions (one `@Test` per spec case, a
196+
`/** @UTS objects/unit/… */` KDoc tag, the `deviations.md` discipline).
197+
2. Specs 7–9 (`object_id`, `objects_pool`, `parent_references`) are pure logic with no network/sync —
198+
they will **run immediately** there, no mock-WebSocket harness needed.
199+
3. Specs 2 and 4 need object state applied to a node; reuse / port the relevant `helpers.kt` builders.
200+
4. Keep `:uts`'s `testRuntimeOnly(project(":liveobjects"))` as-is (reflection helpers stay valid), or
201+
optionally promote to `testImplementation` purely for compile-time access to the **public** API.
202+
203+
**Trade-off:** these five tests then live outside the `uts` module the skill normally targets. That is
204+
acceptable and correct — they are internal-implementation tests, and the SDK already groups its own
205+
internal tests under `:liveobjects`. The `@UTS` id convention keeps them traceable to the spec.
206+
207+
### 5.4 Keeping the tests under `uts/unit` — what actually works
208+
209+
This is the stated preference, so it gets its own analysis. To assert on `:liveobjects` internals from
210+
test code that physically lives in `:uts`, the realistic options are:
211+
212+
**(a) `java-test-fixtures` bridge — the recommended way to honour the `uts/unit` preference.**
213+
Apply the `java-test-fixtures` plugin to `:liveobjects` and put a thin **inspection/bridge** layer in
214+
`liveobjects/src/testFixtures/kotlin`. Fixture code *belongs to the module*, so it can touch
215+
`:liveobjects` internals; it then re-exposes them as a small **public** API (e.g.
216+
`fun applyAndSnapshot(...): PublicCounterSnapshot`). `:uts` consumes it with:
217+
218+
```kotlin
219+
// uts/build.gradle.kts
220+
testImplementation(testFixtures(project(":liveobjects")))
221+
```
222+
223+
The **assertions stay in `uts/unit`** (calling the fixture's public API) — your preference is satisfied —
224+
while no raw internal is leaked onto `:uts`'s classpath. Caveat: for the *fixture itself* to see Kotlin
225+
`internal`, its compilation must be associated with `main` (Android exposes
226+
`android.experimental.enableTestFixturesKotlinSupport`; for plain JVM Kotlin verify the testFixtures→main
227+
`associateWith` is wired — it reduces back to §5.2(a), which is supported because it is intra-project).
228+
Cost: you design and maintain the bridge surface.
229+
230+
**(b) Reflection from `:uts` (status quo, no build change).**
231+
Current `testRuntimeOnly(project(":liveobjects"))` already lets `:uts` reach internals by reflection at
232+
runtime — this is what `buildPublicObjectMessage` does. Keeps tests in `uts/unit` with zero build
233+
changes, but: stringly-typed, no compile-time safety, brittle to refactors, and verbose for whole-CRDT
234+
assertions. Fine for a couple of accessors; poor for five spec files of state assertions.
235+
236+
> **On `@VisibleForTesting`:** it does **not** change visibility. It is a documentation/lint hint that
237+
> records what the visibility *would* be if not for tests; the actual access is still governed by the
238+
> `public`/`internal` modifier. So "mark it `@VisibleForTesting`" only helps if you *also* make the
239+
> member `public` (e.g. `@VisibleForTesting(otherwise = PRIVATE) public fun …`). It is not, by itself, a
240+
> cross-module visibility mechanism.
241+
242+
---
243+
244+
## 6. The realistic options
245+
246+
Only **three** approaches are real candidates for our situation. (The mechanisms in §5.2 —
247+
`associateWith`, raw `-Xfriend-paths`, `shared internal` — and the bare dependency-scope swap in §5.1 are
248+
*not* viable on their own; see the note below.)
249+
250+
| Option | Keeps tests in `uts/unit`? | Compile-safe? | Trade-off |
251+
|---|---|---|---|
252+
| **A. `java-test-fixtures` bridge** (§5.4a) | ✅ yes | ✅ yes | Design a small public snapshot surface in `:liveobjects`'s `testFixtures`. **Best fit for the preference.** |
253+
| **B. Tests in `:liveobjects/src/test`** (§5.3) | ❌ no — live in `:liveobjects` | ✅ yes | Least effort; internals visible by design. The SDK already tests its own internals this way. |
254+
| **C. Reflection from `:uts`** (§5.4b) | ✅ yes | ❌ no | No build change, but stringly-typed and brittle — fine for a few hops, poor for 5 spec files. |
255+
256+
**Not viable on their own:** the bare `testRuntimeOnly` → `testImplementation` swap (exposes only the
257+
*public* API, not `internal`); `associateWith` (supported but *intra-project* — cannot bridge `:uts` ↔
258+
`:liveobjects`); raw cross-project `-Xfriend-paths` (unstable/unsupported); `shared internal` /
259+
KEEP-0451 (future proposal, not available yet).
260+
261+
## 7. Recommendation & sequencing
262+
263+
1. **Now:** nothing to translate for specs 2, 4, 7, 8, 9 — the internal engine they test is unbuilt
264+
(Blocker A). Leave them blocked; this document is the record.
265+
2. **When the LiveObjects CRDT engine (`ObjectsPool`, internal live nodes + `applyOperation`,
266+
object-id generation, parent references) is implemented**, pick the visibility approach:
267+
- **To honour the `uts/unit` preference (recommended):** the **`java-test-fixtures` bridge** (§5.4a) —
268+
assertions stay in `uts/unit`, a small fixture in `:liveobjects` exposes the needed internal state
269+
as a public snapshot. Compile-safe and supported.
270+
- **If colocation isn't required:** author them in **`:liveobjects/src/test`** (§5.3) — least
271+
ceremony, internals visible by design (the SDK already tests its own internals this way).
272+
3. **Avoid** the bare `testImplementation` swap *as the internal-access mechanism* (it only exposes the
273+
public API) and the manual cross-project `-Xfriend-paths` hack (unsupported, fragile).
274+
275+
---
276+
277+
## 8. References
278+
279+
- Kotlin associated compilations / `associateWith` (intra-project internal access):
280+
KTIJ-7662, Kotlin Multiplatform "Configure compilations" docs.
281+
- `-Xfriend-paths` is an unstable compiler detail; future `shared internal` modifier: KEEP-0451
282+
("Shared Internals" proposal).
283+
- `@VisibleForTesting` is a lint/documentation hint and does not change visibility.
284+
- Gradle `java-test-fixtures`: test-fixtures code has access to the module's internal API and is
285+
consumed via `testImplementation(testFixtures(project(":…")))`; Kotlin support may require
286+
associating the testFixtures compilation with `main`.

0 commit comments

Comments
 (0)