Skip to content

test: property-check forecast recommendation and summary contracts#599

Open
ndycode wants to merge 2 commits into
mainfrom
claude/audit-80-forecast-property
Open

test: property-check forecast recommendation and summary contracts#599
ndycode wants to merge 2 commits into
mainfrom
claude/audit-80-forecast-property

Conversation

@ndycode

@ndycode ndycode commented Jun 11, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds a fast-check property suite for the forecast engine's pure decision layer — recommendForecastAccount, summarizeForecast, and buildForecastExplanation in lib/forecast.ts. These drive the codex-multi-auth forecast recommendation and its --explain output; the suite pins the selection policy over the full availability/risk/wait/flag space instead of hand-picked pools.

What Changed

New test/property/forecast.property.test.ts (7 properties over generated ForecastAccountResult pools, 0–8 accounts):

  1. Candidate soundness — a non-null recommendation always points at a recommendable result (not disabled, not hard-failed, not exhausted, not unavailable); null exactly when no such candidate exists.
  2. Ready dominance — if any ready candidate exists, the pick is ready with the minimal risk score among ready candidates, and the reason says so.
  3. Delayed fallback — with only delayed candidates, the shortest wait wins.
  4. Blocker-class guidance — an empty candidate pool yields "blocked or exhausted" guidance iff some account is blocked/exhausted rather than disabled/hard-failed. This is the regression guard for the exhausted-flag fix the code comments document (an all-exhausted pool must not produce a bogus "shortest wait" pick).
  5. Order invariance — the recommendation doesn't depend on input order (the comparator's tie-breaks are total over unique indexes).
  6. Summary partitionready + delayed + unavailable === total exactly, with high-risk counted by risk level.
  7. Explanation fidelityconsidered mirrors the inputs in order, and selected is true on exactly the recommended index.

No SUT bugs found.

Validation

  • npm test -- test/property/forecast.property.test.ts test/forecast.test.ts — 42/42 (new 7 + existing 35 untouched)
  • npm run typecheck (also via pre-commit hook)
  • npx eslint test/property/forecast.property.test.ts --max-warnings=0

Docs and Governance Checklist

  • Test-only; no behavior or docs surface changed

Risk and Rollback

  • Risk level: minimal — additive test file; conventions match the existing test/property/ suites.
  • Rollback plan: revert the single commit.

https://claude.ai/code/session_01XNtnkLbBiXZxfQQYLMpucB


Generated by Claude Code

note: greptile review for oc-chatgpt-multi-auth. cite files like lib/foo.ts:123. confirm regression tests + windows concurrency/token redaction coverage.

Greptile Summary

adds a 7-property fast-check suite for the forecast engine's pure decision layer (recommendForecastAccount, summarizeForecast, buildForecastExplanation), covering soundness, ready-dominance, delayed-fallback, blocker guidance, order-invariance, summary partition, and explanation fidelity. addresses both items from the previous review cycle: the exhausted → availability = "delayed" domain constraint is now enforced in arbResults, and order-invariance now uses fc.shuffledSubarray for arbitrary permutations rather than reversal alone.

  • the arbResults generator derives availability from exhausted to restrict the test space to the reachable domain, preventing false positives if a future fast-path relies on exhausted ⟹ availability !== "ready".
  • the summary partition test asserts ready + delayed + unavailable === total and pins summary.ready, but does not individually assert summary.delayed or summary.unavailable, leaving a transposition of those two counts undetected.

Confidence Score: 5/5

additive test-only file with no behavior or api surface changes; safe to merge.

the change is a single new test file exercising pure, side-effect-free functions. no production code is modified, no new dependencies are introduced, and the suite's 7 properties are internally consistent with the sut. the one gap (missing individual delayed/unavailable assertions in the summary test) is a coverage nit that doesn't risk a regression.

test/property/forecast.property.test.ts — the summary partition test could be tightened with individual delayed and unavailable count assertions.

Important Files Changed

Filename Overview
test/property/forecast.property.test.ts new property suite (7 properties) covering recommendation soundness, ready-dominance, delayed-fallback, blocker guidance, order-invariance, summary partition, and explanation fidelity; addresses previous feedback by deriving availability from exhausted and using fc.shuffledSubarray for full-permutation invariance testing

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[results pool] --> B{filter candidates}
    B -->|not disabled, not hardFailure, not exhausted, not unavailable| C[candidates]
    B -->|empty| D{hasBlockedOrExhausted?}
    D -->|yes| E[null + blocked-or-exhausted reason]
    D -->|no| F[null + no-healthy-accounts reason]
    C --> G[sort via compareForecastResults]
    G --> H{best availability}
    H -->|ready| I[pick best index - Lowest risk ready account]
    H -->|delayed| J[pick best index - pick shortest wait]
Loading

Fix All in Codex

Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
test/property/forecast.property.test.ts:156-172
**summary partition misses individual `delayed`/`unavailable` pins** — the `ready + delayed + unavailable === total` check combined with the single `summary.ready` assertion leaves `delayed` and `unavailable` unconstrained against each other. a bug that transposes those two counts satisfies every current assertion. add `expect(summary.delayed).toBe(results.filter(r => r.availability === "delayed").length)` and the equivalent for `unavailable` to close the gap.

Reviews (2): Last reviewed commit: "test: keep forecast generator in the rea..." | Re-trigger Greptile

Seven fast-check properties over recommendForecastAccount,
summarizeForecast, and buildForecastExplanation with generated result
pools (0-8 accounts, full availability/risk/wait/flag space):

- a recommendation always points at a recommendable result (not
  disabled, hard-failed, exhausted, or unavailable); null only when
  no such candidate exists
- a ready candidate always wins with the minimal risk score among
  ready candidates, and the reason says so
- with no ready candidate, the shortest delayed wait wins
- an empty candidate pool names the actual blocker class: 'blocked or
  exhausted' guidance iff some account is blocked/exhausted rather
  than disabled/hard-failed (the #exhausted-flag regression guard)
- the recommendation is invariant under input order
- the summary partitions availability exactly (ready + delayed +
  unavailable === total) and counts high-risk rows
- the explanation mirrors inputs in order and marks selected on
  exactly the recommended index

Companion to the property suites in #574/#575/#592-#598.

https://claude.ai/code/session_01XNtnkLbBiXZxfQQYLMpucB
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@coderabbitai

coderabbitai Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@ndycode, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 6 minutes and 59 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more credits in the billing tab to continue.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 596b5bfa-39ad-48be-8b23-0be69150504c

📥 Commits

Reviewing files that changed from the base of the PR and between dae10cb and c4b39bf.

📒 Files selected for processing (1)
  • test/property/forecast.property.test.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/audit-80-forecast-property
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch claude/audit-80-forecast-property

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Comment thread test/property/forecast.property.test.ts Outdated
Comment thread test/property/forecast.property.test.ts
…ations

Greptile flagged that availability and exhausted were drawn
independently (evaluateForecastAccount only emits exhausted accounts
as delayed) and that order-invariance only probed reversal. The
generator now derives the pairing, and the invariance property checks
an arbitrary generated permutation plus the reversal. Validated at
FAST_CHECK_NUM_RUNS=1000.

https://claude.ai/code/session_01XNtnkLbBiXZxfQQYLMpucB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants