Skip to content

Commit a3d44ad

Browse files
feat(ingestion): business-logic analyze phase — populate likely_plumbing + candidate_business (#249)
## Summary Makes the merged sieve kernels (#248) end-to-end: `codehub analyze` now writes `likelyPlumbing` + `candidateBusiness` into `nodes.payload` for every Function / Method / Constructor / Class / Interface / Struct in a Python / Java / Go repo. The user gets both concern tags from two commands (`codehub init` + `codehub analyze`) with **no query, no labels, no embeddings**: ``` likely_plumbing — precision-first (~0.94) "this is plumbing" candidate_business — recall-first (~0.93) "look here for domain logic" ``` Queryable via SQLite JSON1: `SELECT name FROM nodes WHERE payload->>'$.candidateBusiness' = 'true'`. ## Components - **core-types** — two optional `CallableShape` fields. Auto-persist through `nodes.payload`; no storage-adapter change (the SQLite store rehydrates payload verbatim). - **`extract/business-logic-features.ts`** — faithful Python→TS port of the feature extractor. Reproduces the marker logic exactly: word-boundary / camelCase-component matching, the precise `nPlumbingSignals` formula (serialization + observability + getter/setter + dto-mapper-ratio≥0.5), and ORM-base class-head detection. **44 unit tests.** - **`pipeline/phases/business-logic.ts`** — the analyze-time phase (after `complexity`). Slices each symbol body, runs `classifyPlumbing` + `classifyBusinessCandidate`, re-adds the node with the tags (richer-entry-wins merge, same contract complexity uses). Python/Java/Go only; other languages skip silently. The class-head slice scans upward over a Javadoc block to reach the real `@Entity` / `@MappedSuperclass` annotation while excluding `@author`-style comment tags. - **default-set + orchestrator test** — registered after complexity; the topological-order assertion updated for the new position. ## The parity contract The whole point of the port is that the shipped numbers survive it. An independent per-symbol harness diffs the TS analyze-pass verdicts against the Python oracle across all four corpus repos: **1368 / 1368 = 100.0% verdict agreement, 0 disagreements.** This was re-verified independently of the porting agent's own report — which is how a JPA-entity divergence surfaced (a Javadoc `@author` tag shadowing the real ORM annotation, dropping `isOrmModel` and flipping 5 entity classes). Caught at 99.63%, fixed to 100%. The 0.936 plumbing precision / 0.925 business recall hold end-to-end. ## Determinism `computePlumbingFeatures` and both kernels are pure; files + definitions iterate in sorted order. Tags are byte-stable across runs (the three `graphHash`-determinism tests pass), safe under the reproducibility contract. ## Verification - core-types / analysis / ingestion typecheck clean. - ingestion **629/629**, core-types **83/83**, analysis **14/14**. - biome + banned-strings + commitlint + pre-push hook pass.
1 parent 383b719 commit a3d44ad

6 files changed

Lines changed: 2138 additions & 0 deletions

File tree

packages/core-types/src/nodes.ts

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,23 @@ interface CallableShape {
138138
* supplied; absent otherwise.
139139
*/
140140
readonly coveredLinesJson?: string;
141+
/**
142+
* Advisory concern tag from the `business-logic` phase. `true` when the
143+
* deterministic sieve is confident the symbol is plumbing (serialization,
144+
* DTO mapping, transport, DI wiring). Absent when the language is outside
145+
* the validated set (python/java/go) or the phase did not run. Measured
146+
* plumbing precision ~0.94. See `@opencodehub/analysis` `classifyPlumbing`.
147+
*/
148+
readonly likelyPlumbing?: boolean;
149+
/**
150+
* Advisory concern tag from the `business-logic` phase — the recall-first
151+
* complement of {@link likelyPlumbing}. `true` when the symbol is a
152+
* business-logic candidate (everything the sieve did NOT classify as
153+
* plumbing). High recall (~0.93), low precision by design: a "look here for
154+
* domain logic" hint, not a confirmed classification. Absent when the phase
155+
* did not run for this symbol's language.
156+
*/
157+
readonly candidateBusiness?: boolean;
141158
}
142159

143160
interface TypeDeclShape {
Lines changed: 371 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,371 @@
1+
/**
2+
* Parity tests for the business-logic feature extractor against the Python
3+
* oracle `och_bizlogic_extract.py`. Each test fixes the expected feature value
4+
* to the value the Python emits for the same snippet (captured by running the
5+
* Python `_extract_one` on each case), then asserts the TS port reproduces it.
6+
*
7+
* The four fields under test are exactly the ones the merged kernel
8+
* (`@opencodehub/analysis` `classifyPlumbing`) consumes, so a passing suite
9+
* means the shipped sieve verdict agrees with the Python substrate.
10+
*/
11+
12+
import assert from "node:assert/strict";
13+
import { test } from "node:test";
14+
import { computePlumbingFeatures } from "./business-logic-features.js";
15+
16+
function py(bodyText: string, symbolName: string, kind = "Function", classHeadText?: string) {
17+
return computePlumbingFeatures({
18+
symbolName,
19+
kind,
20+
bodyText,
21+
lang: "python",
22+
...(classHeadText !== undefined ? { classHeadText } : {}),
23+
});
24+
}
25+
26+
// ── serialization calls ─────────────────────────────────────────────────────
27+
28+
test("serialization: json.dumps(...) counts one serialization call", () => {
29+
const f = py("def to_wire(self):\n return json.dumps(self.payload)\n", "to_wire");
30+
assert.equal(f.nSerializationCalls, 1);
31+
});
32+
33+
test("serialization: self.model_dump() counts one serialization call", () => {
34+
const f = py("def out(self):\n return self.model_dump()\n", "out");
35+
assert.equal(f.nSerializationCalls, 1);
36+
});
37+
38+
test("serialization: nested log.info(json.dumps(x)) — ser=2, observ folds into plumbing", () => {
39+
// Oracle: outer call head contains json/dumps (+1 ser) AND info/log (+1 observ);
40+
// inner json.dumps(...) (+1 ser). So nSerializationCalls = 2.
41+
const f = py("def emit(self):\n log.info(json.dumps(self.x))\n", "emit");
42+
assert.equal(f.nSerializationCalls, 2);
43+
// observ call present → plumbing signal ≥ 1.
44+
assert.ok(f.nPlumbingSignals >= 1);
45+
});
46+
47+
test("serialization: a non-serializer call counts zero", () => {
48+
const f = py("def go(self):\n self.do_work(self.x)\n", "go");
49+
assert.equal(f.nSerializationCalls, 0);
50+
});
51+
52+
// ── domain conditionals vs guards ───────────────────────────────────────────
53+
54+
test("domain conditional: value comparison is a domain signal", () => {
55+
const f = py(
56+
"def check(self, amount):\n if amount > self.limit:\n return True\n",
57+
"check",
58+
);
59+
assert.equal(f.nDomainSignals, 1);
60+
});
61+
62+
test("guard: `if x is None` is NOT a domain conditional", () => {
63+
const f = py("def check(self, x):\n if x is None:\n return 0\n return x\n", "check");
64+
assert.equal(f.nDomainSignals, 0);
65+
});
66+
67+
test("guard: `if isinstance(x, int)` is NOT a domain conditional", () => {
68+
const f = py("def f(self, x):\n if isinstance(x, int):\n pass\n", "f");
69+
assert.equal(f.nDomainSignals, 0);
70+
});
71+
72+
test("guard: `if len(x) > 0` is NOT a domain conditional (len is a guard token)", () => {
73+
const f = py("def f(self, x):\n if len(x) > 0:\n pass\n", "f");
74+
assert.equal(f.nDomainSignals, 0);
75+
});
76+
77+
test("conditional: elif does NOT add a second conditional (elif is a separate node)", () => {
78+
const f = py(
79+
"def f(self, x):\n if x > 5:\n pass\n elif x < 2:\n pass\n",
80+
"f",
81+
);
82+
assert.equal(f.nDomainSignals, 1);
83+
});
84+
85+
test("conditional: two separate if statements count two", () => {
86+
const f = py("def f(self, x):\n if x > 5:\n pass\n if x < 2:\n pass\n", "f");
87+
assert.equal(f.nDomainSignals, 2);
88+
});
89+
90+
// ── arithmetic ───────────────────────────────────────────────────────────────
91+
92+
test("arithmetic: `a + b` is one domain signal", () => {
93+
const f = py("def f(self, a, b):\n return a + b\n", "f");
94+
assert.equal(f.nDomainSignals, 1);
95+
});
96+
97+
test("arithmetic: `a + b * 2` is two domain signals (two binary ops)", () => {
98+
const f = py("def total(self, a, b):\n return a + b * 2\n", "total");
99+
assert.equal(f.nDomainSignals, 2);
100+
});
101+
102+
test("arithmetic: a bare comparison `a > b` is NOT arithmetic", () => {
103+
const f = py("def f(self, a, b):\n return a > b\n", "f");
104+
assert.equal(f.nDomainSignals, 0);
105+
});
106+
107+
test("arithmetic: augmented assignment `-=` is NOT counted as arithmetic", () => {
108+
// Oracle: `self.balance_due -= amount` → n_arithmetic_ops = 0.
109+
const f = py("def f(self):\n self.balance_due -= amount\n", "pay");
110+
assert.equal(f.nDomainSignals, 0);
111+
});
112+
113+
// ── domain exceptions vs stdlib ─────────────────────────────────────────────
114+
115+
test("domain exception: raising InsufficientFundsError is a domain signal", () => {
116+
const f = py(
117+
"def withdraw(self, amount):\n if amount > self.balance:\n raise InsufficientFundsError(amount)\n",
118+
"withdraw",
119+
);
120+
// 1 conditional (amount > balance) + 1 domain exception = 2.
121+
assert.equal(f.nDomainSignals, 2);
122+
});
123+
124+
test("stdlib exception: raising ValueError is NOT a domain signal", () => {
125+
const f = py("def parse(self, x):\n raise ValueError('bad')\n", "do_parse");
126+
assert.equal(f.nDomainSignals, 0);
127+
});
128+
129+
test("domain exceptions: two distinct domain raises count two", () => {
130+
const f = py(
131+
"def f(self):\n raise PaymentDeclinedError('x')\n raise OrderConflict('y')\n",
132+
"f",
133+
);
134+
assert.equal(f.nDomainSignals, 2);
135+
});
136+
137+
// ── state transitions ───────────────────────────────────────────────────────
138+
139+
test("state transition: assigning self.status is a domain signal", () => {
140+
const f = py("def advance(self):\n self.status = self.next_status\n", "advance", "Method");
141+
assert.equal(f.nDomainSignals, 1);
142+
});
143+
144+
test("state transition: assigning self.state (no RHS dot) is a domain signal", () => {
145+
const f = py("def advance(self):\n self.state = 5\n", "advance", "Method");
146+
assert.equal(f.nDomainSignals, 1);
147+
});
148+
149+
// ── qualified persistence / raw-SQL / bootstrap are NOT in nPlumbingSignals ──
150+
//
151+
// The shipped kernel reads ONLY nSerializationCalls / nDomainSignals /
152+
// nPlumbingSignals / isOrmModel, and the Python `n_plumbing_signals` (lines
153+
// 767-769) is composed EXACTLY as
154+
// n_serialization_calls + n_observ_calls + (is_getter_setter?1:0)
155+
// + (dto_mapper_ratio>=0.5?1:0).
156+
// Qualified-persistence / raw-SQL / bootstrap-name feed the Python's
157+
// `touches_persistence` / `is_framework_bootstrap` fields, which the kernel does
158+
// NOT consume — so they do NOT enter nPlumbingSignals. The expected values below
159+
// were captured by running the Python oracle `_extract_one` on each snippet.
160+
161+
test("persistence: session.execute(...) is NOT in nPlumbingSignals (feeds touches_persistence)", () => {
162+
// Oracle: ser=0 obs=0 gs=False dto=0 → n_plumbing_signals = 0.
163+
const f = py("def save(self):\n self.session.execute('foo')\n db.commit()\n", "save");
164+
assert.equal(f.nPlumbingSignals, 0);
165+
assert.equal(f.nDomainSignals, 0);
166+
});
167+
168+
test("persistence: a BARE verb `update(self.x)` is NOT a plumbing signal", () => {
169+
const f = py("def thing(self):\n update(self.x)\n", "thing");
170+
assert.equal(f.nPlumbingSignals, 0);
171+
});
172+
173+
test("persistence: `self.repo.get(ref)` does NOT enter nPlumbingSignals (qualified persistence is excluded)", () => {
174+
// Oracle: ser=0 obs=0 gs=False (has a call) dto=0 → n_plumbing_signals = 0.
175+
const f = py("def fetch(self, ref):\n return self.repo.get(ref)\n", "fetch");
176+
assert.equal(f.nPlumbingSignals, 0);
177+
});
178+
179+
test("persistence: Flask web `session.get('_flashes')` is dict access, NOT persistence", () => {
180+
const f = py(
181+
"def flash(message):\n flashes = session.get('_flashes', [])\n return flashes\n",
182+
"do_flash",
183+
);
184+
assert.equal(f.nPlumbingSignals, 0);
185+
});
186+
187+
test("persistence: `context.update(...)` is NOT persistence (ctx is not a DB receiver)", () => {
188+
const f = py(
189+
"def update_template_context(self, context):\n context.update(self.dispatch(name))\n return context\n",
190+
"update_template_context",
191+
);
192+
assert.equal(f.nPlumbingSignals, 0);
193+
});
194+
195+
// ── raw SQL is NOT in nPlumbingSignals (feeds touches_persistence) ───────────
196+
197+
test("raw SQL: SELECT ... FROM does NOT enter nPlumbingSignals", () => {
198+
// Oracle: raw-SQL feeds touches_persistence, not n_plumbing_signals → 0.
199+
const f = py("def q(self):\n cur.execute('SELECT id FROM users WHERE x = 1')\n", "q");
200+
assert.equal(f.nPlumbingSignals, 0);
201+
});
202+
203+
test("raw SQL: INSERT INTO does NOT enter nPlumbingSignals", () => {
204+
// Oracle: raw-SQL feeds touches_persistence, not n_plumbing_signals → 0.
205+
const f = py('def do_add(self):\n run("INSERT INTO user (name) VALUES (?)")\n', "do_add");
206+
assert.equal(f.nPlumbingSignals, 0);
207+
});
208+
209+
// ── observability ────────────────────────────────────────────────────────────
210+
211+
test("observability: logger.info(...) is a plumbing signal", () => {
212+
const f = py("def run(self):\n logger.info('hi')\n", "run");
213+
assert.ok(f.nPlumbingSignals >= 1);
214+
assert.equal(f.nDomainSignals, 0);
215+
});
216+
217+
// ── bootstrap name is NOT in nPlumbingSignals (feeds is_framework_bootstrap) ──
218+
219+
test("bootstrap: create_app does NOT enter nPlumbingSignals (bootstrap is excluded)", () => {
220+
// Oracle: ser=0 obs=0 gs=False (loc 4, has calls) dto=0 → n_plumbing_signals = 0.
221+
const f = py(
222+
"def create_app(config=None):\n app = Flask(__name__)\n app.config.update(config or {})\n return app\n",
223+
"create_app",
224+
);
225+
assert.equal(f.nPlumbingSignals, 0);
226+
});
227+
228+
test("getter/setter: `to_wire` is a plumbing signal because it is a getter/setter", () => {
229+
// Oracle: is_getter_setter == True (loc 2, no conditionals, <=1 return, 0 calls)
230+
// → n_plumbing_signals = 1. (The old `wire` bootstrap path is NOT why it fires.)
231+
const f = py("def to_wire(self):\n return self.x\n", "to_wire");
232+
assert.equal(f.nPlumbingSignals, 1);
233+
});
234+
235+
test("getter/setter: a tiny pass-through `allocate` IS a getter/setter (loc<=3, no cond, <=1 return, 0 calls)", () => {
236+
// Oracle: gs=True → n_plumbing_signals = 1. A plain bootstrap NAME never enters
237+
// the formula, but this tiny pass-through trips the getter/setter tell.
238+
const f = py("def allocate(self):\n return self.x\n", "allocate");
239+
assert.equal(f.nPlumbingSignals, 1);
240+
});
241+
242+
test("bootstrap WITH domain residue: register_payment carries domain residue and no plumbing tell", () => {
243+
// Oracle: n_domain_signals = 2 (the `amount > balance_due` conditional + the
244+
// raised OverpaymentError). bootstrap-name does NOT enter n_plumbing_signals,
245+
// and there is no serializer / observ / getter-setter / dto tell → plumb = 0.
246+
const f = py(
247+
"def register_payment(self, amount):\n if amount > self.balance_due:\n raise OverpaymentError(amount)\n self.balance_due -= amount\n",
248+
"register_payment",
249+
"Method",
250+
);
251+
assert.equal(f.nDomainSignals, 2);
252+
assert.equal(f.nPlumbingSignals, 0);
253+
});
254+
255+
// ── ORM base-class match vs flask false positive ────────────────────────────
256+
257+
test("ORM: class User(Base) is an ORM model (exact base superclass)", () => {
258+
const f = py("pass\n", "User", "Class", "class User(Base):");
259+
assert.equal(f.isOrmModel, true);
260+
});
261+
262+
test("ORM: class Order(Model) is an ORM model", () => {
263+
const f = py("pass\n", "Order", "Class", "class Order(Model):");
264+
assert.equal(f.isOrmModel, true);
265+
});
266+
267+
test("ORM false positive guard: class Request(RequestBase) is NOT an ORM model", () => {
268+
// The precision fix: `Base` matches ONLY as an exact superclass identifier,
269+
// never as a component of `RequestBase`.
270+
const f = py("pass\n", "Request", "Class", "class Request(RequestBase):");
271+
assert.equal(f.isOrmModel, false);
272+
});
273+
274+
test("ORM: UserEntity (component role in the name) is an ORM model", () => {
275+
const f = py("pass\n", "UserEntity", "Class", "class UserEntity:");
276+
assert.equal(f.isOrmModel, true);
277+
});
278+
279+
test("ORM: pydantic BaseModel base is dropped — class UserDTO(BaseModel) is NOT an ORM model", () => {
280+
const f = py("pass\n", "UserDTO", "Class", "class UserDTO(BaseModel):");
281+
assert.equal(f.isOrmModel, false);
282+
});
283+
284+
test("ORM: AbstractRepository(abc.ABC) is infra plumbing, NOT an ORM model", () => {
285+
// Repository is an infra ROLE component → is_orm_model is False (it is
286+
// plumbing, but not a mapped entity).
287+
const f = py(
288+
"def add(self, p):\n self._add(p)\n",
289+
"AbstractRepository",
290+
"Class",
291+
"class AbstractRepository(abc.ABC):",
292+
);
293+
assert.equal(f.isOrmModel, false);
294+
});
295+
296+
// ── Java / Go class-head + persistence parity ───────────────────────────────
297+
298+
test("java ORM: @Entity class Owner extends BaseEntity is an ORM model", () => {
299+
const f = computePlumbingFeatures({
300+
symbolName: "Owner",
301+
kind: "Class",
302+
bodyText: "",
303+
classHeadText: "@Entity\nclass Owner extends BaseEntity",
304+
lang: "java",
305+
});
306+
assert.equal(f.isOrmModel, true);
307+
});
308+
309+
test("java infra: interface OwnerRepository extends JpaRepository is NOT an ORM model", () => {
310+
const f = computePlumbingFeatures({
311+
symbolName: "OwnerRepository",
312+
kind: "Class",
313+
bodyText: "",
314+
classHeadText: "interface OwnerRepository extends JpaRepository<Owner, Integer>",
315+
lang: "java",
316+
});
317+
assert.equal(f.isOrmModel, false);
318+
});
319+
320+
test("java persistence: em.persist(entity) does NOT enter nPlumbingSignals", () => {
321+
// Oracle: qualified persistence feeds touches_persistence, not the kernel's
322+
// n_plumbing_signals (ser=0 obs=0 gs=False dto=0) → 0.
323+
const f = computePlumbingFeatures({
324+
symbolName: "save",
325+
kind: "Method",
326+
bodyText: "void save() {\n em.persist(entity);\n}",
327+
lang: "java",
328+
});
329+
assert.equal(f.nPlumbingSignals, 0);
330+
});
331+
332+
test("go persistence: uc.repo.Store(ctx, &task) does NOT enter nPlumbingSignals", () => {
333+
// Oracle: qualified persistence feeds touches_persistence, not the kernel's
334+
// n_plumbing_signals (ser=0 obs=0 gs=False dto=0) → 0.
335+
const f = computePlumbingFeatures({
336+
symbolName: "Create",
337+
kind: "Method",
338+
bodyText:
339+
"func (uc *UseCase) Create(ctx context.Context) error {\n err := uc.repo.Store(ctx, &task)\n return err\n}",
340+
lang: "go",
341+
});
342+
assert.equal(f.nPlumbingSignals, 0);
343+
});
344+
345+
test("go: no raise/throw, so a domain-exception scan is zero", () => {
346+
const f = computePlumbingFeatures({
347+
symbolName: "Run",
348+
kind: "Function",
349+
bodyText: "func Run() {\n panic(MyDomainError{})\n}",
350+
lang: "go",
351+
});
352+
// Go has no raise node — domain exceptions are 0 regardless of the name.
353+
assert.equal(f.nDomainSignals, 0);
354+
});
355+
356+
// ── end-to-end kernel agreement spot-check ──────────────────────────────────
357+
358+
test("kernel agreement: a pure serializer is swept (ser>0, domain=0)", () => {
359+
const f = py("def to_wire(self):\n return json.dumps(self.payload)\n", "marshal_out");
360+
assert.equal(f.nSerializationCalls, 1);
361+
assert.equal(f.nDomainSignals, 0);
362+
});
363+
364+
test("kernel agreement: a domain method is NOT swept (domain>0)", () => {
365+
const f = py(
366+
"def allocate(self, line):\n if self.can_allocate(line):\n return line.qty * self.unit_price\n",
367+
"allocate",
368+
"Method",
369+
);
370+
assert.ok(f.nDomainSignals > 0);
371+
});

0 commit comments

Comments
 (0)