feat: configurable interview panel size with stratified sampling - #28
Open
jcartagenamartinez-legaltech wants to merge 8 commits into
Open
Conversation
Two issues prevented the claude-cli LLM provider from running on
Windows:
- cwd="/tmp" is a POSIX-only path; on Windows this raises
WinError 267 (invalid directory name). Replaced with
tempfile.gettempdir() for both the claude and codex subprocess
calls, which resolves correctly on all platforms.
- The prompt was passed as a trailing argv element
(["claude", "-p", "--output-format", "json", prompt]). Large
prompts (tens of KB, common once a few input files are attached)
exceed Windows' command-line length limit and raise
WinError 206 ("filename or extension is too long"). Passing the
prompt via stdin instead removes this ceiling. Bumped the
subprocess timeout from 300s to 600s to give slower/CLI-based
providers headroom, and set encoding="utf-8", errors="replace"
explicitly since Windows' default locale encoding (cp1252) was
raising UnicodeDecodeError on non-ASCII input/output.
Tested on Windows 11 running full simulations (100+ agents,
15 rounds) end to end with LLM_PROVIDER=claude-cli.
rapidfuzz's latest release at time of writing (3.14.4) has no win_amd64 wheel published, which makes `uv sync` fail outright on Windows (no sdist build toolchain assumed). Overriding to 3.13.0, the latest version with a Windows wheel, unblocks installation. This can be relaxed/removed once upstream publishes Windows wheels for a newer version again.
interview_agents() (used by the report generator to interview the top N most-active agents post-simulation) sends a batch IPC command with a hardcoded 180s timeout covering up to N agents x 2 platforms of individual LLM interview calls. With CLI-based LLM providers this was consistently timing out with 0/N successful interviews on runs with 100+ agent profiles, even though the simulation itself completed normally — the report would silently fall back to "(no interview data)" since the failure is caught as non-fatal by the caller. 180s doesn't leave enough headroom when each individual call can legitimately take up to the LLM client's own per-call ceiling (600s in llm_client.py, e.g. for claude-cli subprocess spawns). Raised to match that ceiling.
Addresses self-review findings on the original three commits: - Single CLI_CALL_TIMEOUT_SECONDS constant governs both providers and every timeout error message (the claude handler still said "300s" after the ceiling moved to 600; codex was left at 180 with no encoding handling — same cp1252 hazards the claude path already fixed). - Interview batch budget now scales with the batch instead of a flat 600s: one full per-call ceiling per interviewed agent. A fixed budget equals the worst case of a single call and still starves multi-agent batches. - rapidfuzz override scoped with sys_platform == 'win32' marker so macOS/Linux keep the resolver's pick; pyproject now documents why the pin exists and when to drop it (the missing wheels are specifically Windows cp312, not all of win_amd64). - Provider binaries resolved via shutil.which for a clear early error when the CLI is missing (and to tolerate npm-style .cmd shims on Windows). - Warn when undecodable bytes were replaced with U+FFFD instead of corrupting output silently. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
The post-simulation interview panel was hardcoded to the 5 most active
agents, which (a) caps the qualitative evidence the final report can
draw on and (b) over-represents the vocal minority — silent agents,
who watched the whole simulation and engaged with none of it, often
hold the most informative objections.
- MIROFISH_INTERVIEW_MAX_AGENTS env var (default 5, preserving current
behavior) controls panel size. Cost scales linearly: one LLM call per
agent per platform; the batch timeout already scales with panel size.
- With activity data and a panel of >=6, selection becomes stratified:
~40% most vocal, ~30% LLM-picked for topical relevance from the
remainder, ~30% silent non-participants sampled across the roster.
Below the threshold the existing LLM selection path is unchanged.
- MIROFISH_INTERVIEW_EXTRA_QUESTIONS (';'-separated) appends fixed
questions to the generated questionnaire — closed questions (price
bands, adoption timing) whose answers can be coded into
distributions for downstream quantitative use.
Tested: stratified selector unit-smoke (quotas, no duplicates, range,
classic path below threshold, no-LLM fallback) plus py_compile.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
- LLMTimeoutError (RuntimeError subclass) raised on TimeoutExpired and excluded from the x3 retry loop: a prompt that exhausts the 600s ceiling will almost certainly exhaust it again, so retrying turned one slow call into ~30 minutes of blocked wall clock. - _resolve_cli cached with lru_cache: the binary does not move during the process lifetime, so the per-request PATH scan was waste. (Failures are not cached: lru_cache does not memoize exceptions.) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…atified-interviews
…dedup - _get_int_env helper: malformed MIROFISH_INTERVIEW_MAX_AGENTS values degrade to the default with a warning instead of crashing at import from the Config class body; floor of 1 guards zero/negative panels. - Interview batches now run in chunks of 5 with a per-chunk timeout (ceiling x chunk size): the previous single all-or-nothing IPC transaction scaled to hours with large panels and lost every completed interview if one hung. A failed chunk now loses only itself; results accumulate across chunks. - LLM-selected interview indices are deduped (order-preserving), coerced from strings and range-checked before use. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The post-simulation interview panel is hardcoded to the 5 most active agents.
This caps the qualitative evidence the report can draw on and biases it toward
the vocal minority — the silent agents, who watched the whole simulation and
engaged with none of it, often hold the most informative objections ("why
didn't this interest you?" is half of any real market research).
MIROFISH_INTERVIEW_MAX_AGENTS(default5, current behaviorpreserved) controls panel size. Cost scales linearly — one LLM call per
agent per platform.
~40% most vocal, ~30% LLM-picked for topical relevance from the remainder,
~30% silent non-participants sampled across the roster. Below the threshold
the existing LLM-selection path is unchanged.
MIROFISH_INTERVIEW_EXTRA_QUESTIONS(;-separated) appends fixedquestions to the generated questionnaire — useful for closed questions
(price bands, adoption timing) whose answers can be coded into
distributions for quantitative downstream use.
Stacked on #26
Branched from #26 because a larger panel is only viable with its scaled batch
timeout (a flat 180s/600s budget starves any panel beyond a handful of
agents). Merge #26 first and this rebases clean; the interview-related diff
of this PR is the last commit only.
Testing
10-agent panel over 20 profiles), no duplicate indices, all in range,
most-active agents always included, classic path below threshold, and the
no-LLM fallback still deterministic.
py_compileclean on the three touched modules.🤖 Generated with Claude Code