Skip to content

Commit 383b719

Browse files
feat(analysis): plumbing sieve + candidate_business tag (deterministic, advisory) (#248)
## Summary Two deterministic, label-free per-symbol tags for `@opencodehub/analysis`, the two halves of one idea: - **`classifyPlumbing`** (precision-first) — flags high-confidence plumbing (serialization, DTO mapping, transport, DI wiring), abstains otherwise. Never asserts "business logic". **Plumbing precision 0.936 aggregate, ≥0.85 on every repo** (flask 1.00, java 0.94, go 0.92, cosmic 0.89) under per-repo eval. - **`classifyBusinessCandidate`** (recall-first) — the exact complement: a symbol is a `candidate_business` unless the sieve is confident it's plumbing. **Business recall 0.925** (misses 6 of 80), per-repo 0.80–1.00. This is the "look here for domain logic" tag the user gets at analyze time with no query. ## Why this shape Asserting "this IS business logic" needs a trained classifier and didn't generalize across repos (held-out F1 ~0.3). **Subtracting** confident plumbing does generalize, because the sieve does. So: ``` likely_plumbing — sieve, precision-first (0.94) "this is plumbing" candidate_business — !likely_plumbing, recall-first (0.925) "look here for domain logic" ``` Recall-first **by construction**: a symbol only loses the candidate tag when we're confident it's plumbing, so real domain logic can't be silently dropped. The 0.385 candidate precision (tags ~67%) is the intended trade — the tag is the safety net; an optional embedding-derived `business_rank` (follow-up) orders candidates so the most domain-like surface first. Embeddings become a *rank*, not a yes/no judge. ## Provenance Distilled from a teacher/student loop — a 3-model LLM panel labeled ~300 symbols across Python/Java/Go, a shallow tree was fit, the cleanest high-precision plumbing leaves were lifted out. Both kernels are pure functions of a small `PlumbingFeatures` struct (no I/O, no model, no randomness — mirrors `page-rank.ts`), so verdicts are safe to persist into `nodes.payload` and survive the `graphHash` byte-identity contract. The complement invariant (every symbol is either confident-plumbing or a candidate, never both/neither) is pinned by test. ## End-user flow (unchanged) `codehub init` + `codehub analyze [--embeddings]`. The tags land in `nodes.payload`, queryable as `payload->>'$.candidate_business'`. No labeling, ever — labeling was only the offline method-validation, it does not ship. ## Scope - **This PR:** both kernels + types + 14 unit tests (incl. pinned regressions: `AbstractRepository` → plumbing, `Batch.allocate` → candidate), exported from the package index. - **Follow-up:** wire the analyze-time pass that computes `PlumbingFeatures` from the AST and writes `likely_plumbing` + `candidate_business` (+ optional embedding `business_rank`) into `nodes.payload`. Also refreshes the stale `sql` MCP tool description in `CLAUDE.md` to ADR 0019. ## Verification - `packages/analysis` typecheck clean; full suite **154 pass / 0 fail**. - banned-strings + commitlint + pre-push test hook all pass.
1 parent 90f40a2 commit 383b719

4 files changed

Lines changed: 343 additions & 1 deletion

File tree

CLAUDE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ tiers.
1010
- `context` — inbound/outbound refs and participating flows for one symbol.
1111
- `impact` — dependents of a target up to a configurable depth, with a risk tier.
1212
- `detect_changes` — map an uncommitted or committed diff to affected symbols.
13-
- `sql` — read-only SQL against the local temporal store (the `cochanges` and `symbol_summaries` tables), 5 s timeout. The node/edge graph lives in `graph.lbug` (ADR 0016) and is reached via the typed tools (`query`/`context`/`impact`) or Cypher via the MCP `sql` tool's `cypher` arg — NOT via this SQL path.
13+
- `sql` — read-only SQL against the single-file `store.sqlite` index (ADR 0019). `nodes`, `edges`, `embeddings`, `cochanges`, `symbol_summaries`, and `store_meta` are all directly SQL-queryable (e.g. `SELECT id, name FROM nodes WHERE kind = 'Function'`; reach kind-specific fields via SQLite JSON1, `payload->>'$.field'`). 5 s timeout. The typed tools (`query`/`context`/`impact`) remain the high-level path; the `cypher` arg is reserved for community-fork graph adapters and is not supported by the default backend.
1414

1515
Run `codehub analyze` after pulling new commits so the index stays aligned
1616
with the working tree. `codehub status` reports staleness.
Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
import { strict as assert } from "node:assert";
2+
import { test } from "node:test";
3+
import {
4+
classifyBusinessCandidate,
5+
classifyPlumbing,
6+
type PlumbingFeatures,
7+
SIEVE_VALIDATED_LANGUAGES,
8+
} from "./business-logic.js";
9+
10+
/** Build a feature vector with all-zero defaults, overriding what the case needs. */
11+
function feat(over: Partial<PlumbingFeatures>): PlumbingFeatures {
12+
return {
13+
nSerializationCalls: 0,
14+
nDomainSignals: 0,
15+
nPlumbingSignals: 0,
16+
isOrmModel: false,
17+
...over,
18+
};
19+
}
20+
21+
// ── The domain-signal veto: any real decision forces abstain ────────────────
22+
23+
test("a symbol with any domain signal abstains, even amid plumbing", () => {
24+
// The dangerous error is calling a domain rule "plumbing". A serializer call
25+
// sitting next to a domain conditional must NOT be swept into plumbing.
26+
const v = classifyPlumbing(
27+
feat({ nDomainSignals: 1, nSerializationCalls: 2, nPlumbingSignals: 3 }),
28+
);
29+
assert.equal(v.likelyPlumbing, false);
30+
assert.equal(v.tier, "none");
31+
assert.equal(v.plumbingConfidence, 0);
32+
});
33+
34+
// ── Tier 1: serialization-pure (precision ~1.0) ─────────────────────────────
35+
36+
test("a pure serializer (no domain signal) is tier-1 plumbing at 0.95", () => {
37+
// e.g. cosmic-DDD `to_dict` / a Marshal helper.
38+
const v = classifyPlumbing(feat({ nSerializationCalls: 1 }));
39+
assert.equal(v.likelyPlumbing, true);
40+
assert.equal(v.tier, "serialization-pure");
41+
assert.equal(v.plumbingConfidence, 0.95);
42+
});
43+
44+
// ── Tier 2: plumbing-no-domain (precision ~0.94) ────────────────────────────
45+
46+
test("plumbing signals with no domain signal, not ORM, is tier-2 plumbing at 0.90", () => {
47+
// e.g. a DI-wiring constructor or a logging wrapper.
48+
const v = classifyPlumbing(feat({ nPlumbingSignals: 2 }));
49+
assert.equal(v.likelyPlumbing, true);
50+
assert.equal(v.tier, "plumbing-no-domain");
51+
assert.equal(v.plumbingConfidence, 0.9);
52+
});
53+
54+
test("an ORM entity with plumbing signals is excluded from tier-2 (abstains)", () => {
55+
// ORM entities carry domain methods; the rule must not sweep them up.
56+
const v = classifyPlumbing(feat({ nPlumbingSignals: 2, isOrmModel: true }));
57+
assert.equal(v.likelyPlumbing, false);
58+
assert.equal(v.tier, "none");
59+
});
60+
61+
// ── Abstention: no signal at all ────────────────────────────────────────────
62+
63+
test("a symbol with no serialization, no plumbing, no domain signal abstains", () => {
64+
const v = classifyPlumbing(feat({}));
65+
assert.equal(v.likelyPlumbing, false);
66+
assert.equal(v.tier, "none");
67+
});
68+
69+
// ── Regression fixtures (the iter-0 cases Laith pinned) ─────────────────────
70+
71+
test("regression: AbstractRepository (infra base, no domain rule) reads plumbing", () => {
72+
// An abstract repository base: plumbing signals (persistence wiring), zero
73+
// domain decision, and it is NOT itself an ORM-mapped entity row. Must flag
74+
// plumbing — this inverted in an earlier iteration and is pinned here.
75+
const v = classifyPlumbing(feat({ nPlumbingSignals: 1, nDomainSignals: 0, isOrmModel: false }));
76+
assert.equal(v.likelyPlumbing, true);
77+
});
78+
79+
test("regression: Batch.allocate (domain rule) is never called plumbing", () => {
80+
// The canonical domain method: a conditional on available quantity + a raised
81+
// domain exception => nDomainSignals > 0 => abstain. The sieve must never hide it.
82+
const v = classifyPlumbing(feat({ nDomainSignals: 2, nPlumbingSignals: 0 }));
83+
assert.equal(v.likelyPlumbing, false);
84+
assert.equal(v.tier, "none");
85+
});
86+
87+
// ── Determinism ─────────────────────────────────────────────────────────────
88+
89+
test("classifyPlumbing is a pure function — identical inputs, identical verdict", () => {
90+
const f = feat({ nSerializationCalls: 1, nPlumbingSignals: 1 });
91+
const a = classifyPlumbing(f);
92+
const b = classifyPlumbing(f);
93+
assert.deepEqual(a, b);
94+
});
95+
96+
test("validated-language set is exactly python/java/go", () => {
97+
assert.equal(SIEVE_VALIDATED_LANGUAGES.has("python"), true);
98+
assert.equal(SIEVE_VALIDATED_LANGUAGES.has("java"), true);
99+
assert.equal(SIEVE_VALIDATED_LANGUAGES.has("go"), true);
100+
assert.equal(SIEVE_VALIDATED_LANGUAGES.has("ruby"), false);
101+
});
102+
103+
// ── candidate_business: the recall-first complement of the sieve ────────────
104+
105+
test("candidate_business is the exact complement of likely_plumbing", () => {
106+
// The core invariant: every symbol is either confident-plumbing or a
107+
// candidate, never both, never neither. Spot-check across the rule surface.
108+
for (const f of [
109+
feat({}),
110+
feat({ nSerializationCalls: 1 }),
111+
feat({ nPlumbingSignals: 2 }),
112+
feat({ nPlumbingSignals: 2, isOrmModel: true }),
113+
feat({ nDomainSignals: 3, nPlumbingSignals: 1 }),
114+
]) {
115+
const sieve = classifyPlumbing(f);
116+
const cand = classifyBusinessCandidate(f);
117+
assert.equal(cand.candidateBusiness, !sieve.likelyPlumbing);
118+
assert.deepEqual(cand.plumbing, sieve); // carries the verdict through verbatim
119+
}
120+
});
121+
122+
test("recall-first: a bare domain conditional stays a candidate", () => {
123+
// A symbol with a domain decision and no plumbing must be a candidate — this
124+
// is the recall the tag is built to protect (don't drop domain logic).
125+
const v = classifyBusinessCandidate(feat({ nDomainSignals: 1 }));
126+
assert.equal(v.candidateBusiness, true);
127+
});
128+
129+
test("recall-first: a symbol with NO signals at all is still a candidate", () => {
130+
// The sieve only removes CONFIDENT plumbing. An empty/unknown symbol is kept
131+
// as a candidate rather than silently excluded — high recall by construction.
132+
const v = classifyBusinessCandidate(feat({}));
133+
assert.equal(v.candidateBusiness, true);
134+
assert.equal(v.plumbing.tier, "none");
135+
});
136+
137+
test("confident plumbing is NOT a candidate", () => {
138+
// A pure serializer is removed from the candidate set.
139+
const v = classifyBusinessCandidate(feat({ nSerializationCalls: 1 }));
140+
assert.equal(v.candidateBusiness, false);
141+
});
142+
143+
test("regression: Batch.allocate is a business candidate, AbstractRepository is not", () => {
144+
// The two pinned cases, viewed through the candidate tag.
145+
assert.equal(classifyBusinessCandidate(feat({ nDomainSignals: 2 })).candidateBusiness, true);
146+
assert.equal(classifyBusinessCandidate(feat({ nPlumbingSignals: 1 })).candidateBusiness, false);
147+
});
Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
/**
2+
* Deterministic business-logic / plumbing classifier for
3+
* `@opencodehub/analysis`.
4+
*
5+
* This is the SIEVE half of business-logic detection: a high-precision,
6+
* conservative rule that flags symbols which are almost certainly plumbing
7+
* (serialization, DTO mapping, transport, DI wiring) and ABSTAINS everywhere
8+
* else. It does NOT assert "this is business logic" — calling a real domain
9+
* rule "plumbing" and hiding it is the costly error, so the rule is tuned for
10+
* plumbing PRECISION and stays silent when unsure.
11+
*
12+
* ## Provenance
13+
*
14+
* The rule was distilled from a teacher/student loop: a 3-model LLM panel
15+
* labeled ~300 symbols across 4 repos (Python / Java / Go), a shallow decision
16+
* tree was fit, and the two cleanest, highest-precision plumbing leaves were
17+
* lifted out as the shippable rule. Measured plumbing precision on the labeled
18+
* corpus: 0.936 aggregate, and >= 0.85 on EVERY repo under per-repo evaluation
19+
* (py-flask 1.00, java-petclinic 0.94, go-clean 0.92, py-cosmic-ddd 0.89). The
20+
* full classifier (asserting business too) did not generalize cross-repo and is
21+
* intentionally NOT shipped here — only the plumbing direction is.
22+
*
23+
* ## Determinism
24+
*
25+
* Pure function of the per-symbol feature vector — no I/O, no model, no
26+
* randomness. The same inputs always yield the same verdict, so the result is
27+
* safe to persist into `nodes.payload` and survives the `graphHash` byte-
28+
* identity contract. Mirrors the `page-rank.ts` "request-time deterministic
29+
* kernel" idiom.
30+
*
31+
* ## Feature binding
32+
*
33+
* The kernel consumes a small {@link PlumbingFeatures} struct. OCH's ingestion
34+
* computes these from the AST at parse time (the same place
35+
* `cyclomaticComplexity` is produced); see the companion extractor spec. The
36+
* kernel is deliberately decoupled from HOW the features are computed so the
37+
* rule can be unit-tested in isolation and re-tuned without touching ingestion.
38+
*/
39+
40+
/**
41+
* The minimal per-symbol feature vector the plumbing sieve needs. Every field
42+
* is a non-negative integer count or a boolean, computable deterministically
43+
* from the symbol's AST + its place in the file.
44+
*/
45+
export interface PlumbingFeatures {
46+
/**
47+
* Count of serialization / wire-format calls in the body: `json.dumps`,
48+
* `model_dump`, `to_dict`, `Marshal`/`Unmarshal`, `writeValue`, `JSON.parse`,
49+
* etc. A serializer with no domain decision is plumbing.
50+
*/
51+
readonly nSerializationCalls: number;
52+
/**
53+
* Count of POSITIVE domain-logic signals: conditionals comparing domain
54+
* values (not None/nil/type guards), arithmetic/aggregation on domain
55+
* quantities, raised domain exceptions, state-machine transitions. When this
56+
* is > 0 the symbol carries a real decision and the sieve MUST abstain — the
57+
* recall-first half (business detection) owns those.
58+
*/
59+
readonly nDomainSignals: number;
60+
/**
61+
* Count of NEGATIVE plumbing signals: raw-SQL execution, DI wiring,
62+
* framework callbacks/registration, logging/metrics/tracing calls,
63+
* pass-through attribute assignments.
64+
*/
65+
readonly nPlumbingSignals: number;
66+
/**
67+
* True when the symbol is (or is a method on) an ORM-mapped persistence
68+
* entity. ORM entities frequently carry domain methods, so a symbol on one
69+
* is NOT swept into plumbing by the sieve — the rule excludes it.
70+
*/
71+
readonly isOrmModel: boolean;
72+
}
73+
74+
/** Advisory verdict written into `nodes.payload`. */
75+
export interface PlumbingVerdict {
76+
/**
77+
* `true` only when the rule is confident the symbol is plumbing. `false`
78+
* means ABSTAIN — NOT an assertion that the symbol is business logic. A
79+
* consumer should treat `false` as "no signal", never as "this is business".
80+
*/
81+
readonly likelyPlumbing: boolean;
82+
/**
83+
* Confidence in [0, 1] attached to a `likelyPlumbing: true` verdict, keyed to
84+
* the tier that fired. `0` when abstaining. Tier confidences are the measured
85+
* per-tier precisions, rounded: 0.95 (serialization-pure) / 0.90 (standard).
86+
*/
87+
readonly plumbingConfidence: number;
88+
/**
89+
* Which rule tier fired, for auditability. `"none"` when abstaining.
90+
* - `"serialization-pure"`: a serializer with zero domain signal (precision ~1.0).
91+
* - `"plumbing-no-domain"`: plumbing signals present, zero domain signal,
92+
* not an ORM entity (precision ~0.94).
93+
*/
94+
readonly tier: "serialization-pure" | "plumbing-no-domain" | "none";
95+
}
96+
97+
const ABSTAIN: PlumbingVerdict = {
98+
likelyPlumbing: false,
99+
plumbingConfidence: 0,
100+
tier: "none",
101+
};
102+
103+
/**
104+
* Classify one symbol. Two tiers, evaluated high-confidence first; both require
105+
* ZERO domain signal so a symbol that carries any real decision always abstains.
106+
*
107+
* Tier 1 (conf 0.95): serialization calls present AND no domain signal.
108+
* Tier 2 (conf 0.90): plumbing signals present AND no domain signal AND not an ORM entity.
109+
*
110+
* Anything else abstains. The order matters only for the reported `tier`;
111+
* the two tiers never disagree on `likelyPlumbing`.
112+
*/
113+
export function classifyPlumbing(f: PlumbingFeatures): PlumbingVerdict {
114+
// A real domain decision anywhere in the symbol vetoes the sieve outright.
115+
if (f.nDomainSignals > 0) return ABSTAIN;
116+
117+
if (f.nSerializationCalls > 0) {
118+
return { likelyPlumbing: true, plumbingConfidence: 0.95, tier: "serialization-pure" };
119+
}
120+
121+
if (f.nPlumbingSignals > 0 && !f.isOrmModel) {
122+
return { likelyPlumbing: true, plumbingConfidence: 0.9, tier: "plumbing-no-domain" };
123+
}
124+
125+
return ABSTAIN;
126+
}
127+
128+
/**
129+
* The recall-first complement of the sieve: a symbol is a `candidate_business`
130+
* unless the sieve is confident it is plumbing. This is the "look here for
131+
* domain logic" tag the user gets at analyze time without a query, labels, or
132+
* embeddings.
133+
*
134+
* ## Why subtraction, not assertion
135+
*
136+
* Asserting "this IS business logic" needs a trained classifier and did not
137+
* generalize across repos (held-out F1 ~0.3). SUBTRACTING confident plumbing
138+
* does generalize, because the plumbing sieve does. So the candidate set is
139+
* "everything the sieve did not remove" — recall-first BY CONSTRUCTION: a
140+
* symbol only loses the tag when we are confident it is plumbing, so real
141+
* domain logic cannot be silently dropped.
142+
*
143+
* ## Measured (286 labeled symbols, Python / Java / Go)
144+
*
145+
* Business RECALL 0.925 (misses 6 of 80 business symbols); per-repo recall
146+
* 0.80–1.00 (flask 1.00, java 0.96, go 0.88, cosmic 0.80). Precision 0.385 —
147+
* the tag fires on ~67% of symbols, which is the intended recall-first trade:
148+
* the tag is the safety net (nothing important falls out), and an optional
149+
* embedding-derived rank orders the candidates so the most domain-like surface
150+
* first. The tag NEVER tries to be precise on its own.
151+
*/
152+
export interface BusinessCandidateVerdict {
153+
/**
154+
* `true` when the symbol is a candidate for business logic — i.e. the sieve
155+
* did NOT classify it as plumbing. High recall, low precision by design. A
156+
* consumer should treat this as "worth a look", not "confirmed business".
157+
*/
158+
readonly candidateBusiness: boolean;
159+
/**
160+
* The complementary plumbing verdict that produced this tag, carried through
161+
* for auditability so a consumer can see WHY a symbol was (or was not) a
162+
* candidate without re-running the sieve.
163+
*/
164+
readonly plumbing: PlumbingVerdict;
165+
}
166+
167+
/**
168+
* Tag a symbol as a business-logic candidate. Pure complement of
169+
* {@link classifyPlumbing}: `candidateBusiness === !likelyPlumbing`. Shares the
170+
* exact same feature inputs so the two tags can never disagree about a symbol
171+
* (every symbol is either confident-plumbing or a candidate, never both,
172+
* never neither).
173+
*/
174+
export function classifyBusinessCandidate(f: PlumbingFeatures): BusinessCandidateVerdict {
175+
const plumbing = classifyPlumbing(f);
176+
return { candidateBusiness: !plumbing.likelyPlumbing, plumbing };
177+
}
178+
179+
/**
180+
* Languages the sieve is validated on. The rule's precision floor was measured
181+
* on Python, Java, and Go corpora; calling it on other languages is allowed but
182+
* unvalidated, so the analyze pass should gate on this set and skip the rest
183+
* rather than emit an unbacked verdict.
184+
*/
185+
export const SIEVE_VALIDATED_LANGUAGES: ReadonlySet<string> = new Set(["python", "java", "go"]);

packages/analysis/src/index.ts

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,15 @@
11
export type { ApiImpactFilter, ApiImpactRow } from "./api-impact.js";
22
export { listApiImpact, scoreRisk, worseRisk } from "./api-impact.js";
3+
export type {
4+
BusinessCandidateVerdict,
5+
PlumbingFeatures,
6+
PlumbingVerdict,
7+
} from "./business-logic.js";
8+
export {
9+
classifyBusinessCandidate,
10+
classifyPlumbing,
11+
SIEVE_VALIDATED_LANGUAGES,
12+
} from "./business-logic.js";
313
export {
414
type ChangePackInternal,
515
COST_TOKENIZER_MODEL,

0 commit comments

Comments
 (0)