Decide the Modal worker concurrency model: async handler with concurrent inputs, or sync handler with max_inputs=1

[anth-volk]

Follow-up to #1609 / #1610. The worker currently runs a synchronous handle_household_request under @modal.concurrent (interim config after #1610: max_inputs=3, target_inputs=2, cpu=2.0). That combination carries a structural hazard confirmed both in the 2026-07-06 incident logs and in Modal's documentation: for synchronous functions with input concurrency, "a single input cancellation will terminate the entire container" — one cancelled or timed-out request destroys the warm container and reschedules its sibling inputs, and the replacement container can pay a ~70–90s snapshot rebuild. Modal's guidance also calls input concurrency for CPU-bound workloads "likely not as effective (or even counterproductive)."

We should commit to one of two clean end states rather than keep the interim shape indefinitely:

Option A — async handler, keep concurrent inputs per container

Convert the method to async def and offload the blocking WSGI dispatch to a bounded thread pool (await asyncio.to_thread(dispatch_to_flask_app, self.flask_app, payload)).

• Cancellation becomes per-input: Modal raises asyncio.CancelledError in the affected task only; the container survives and sibling inputs are untouched.
• Keeps warm-pool economics: several cheap requests share one container.
• Caveat: the underlying calculation thread cannot be interrupted, so a cancelled request leaves an orphaned calculation burning CPU until it completes (bounded by max_inputs). The thread pool must be sized to max_inputs.
• Thread-safety is not a new requirement — the sync+concurrent setup already runs the dispatch on concurrent threads today.
• Should ship with a cancellation test against a throwaway Modal environment verifying the container outlives a cancelled input.

Option B — stay sync, drop input concurrency (max_inputs=1)

• Simplest possible semantics: one request per container; a cancellation costs only that container with zero sibling collateral; no GIL/BLAS contention between requests, so per-request latency is the solo latency.
• Matches Modal's guidance for CPU-bound work.
• Cost: every concurrent request needs its own container — more cold starts (mitigated by memory snapshots) and a larger container fleet under load; autoscaler responsiveness becomes the latency bound for bursts.

Decision inputs

• Observe how max_inputs=3 / target_inputs=2 / cpu=2.0 (Cap Modal worker input concurrency at 3 and raise execution budgets #1610) behaves over a release cycle or two: per-request latency spread on heavy calculates, container fleet size, and cost.
• Measure the warm-pool economics that concurrency actually buys for the real traffic mix (routine calculates are <1 CPU-second; the heavy customer-regression households are 12–50 CPU-seconds).
• Open question flagged in Heavy calculates exceed the 180s execution budget under parallel staging tests, and timeout cancellation destroys warm worker capacity #1609: whether a worker-side timeout on a sync concurrent container also recycles the container (docs are explicit only about caller cancellation). If it does, Option A is the only shape that fully removes the capacity-destruction failure class.

Related: #1609 (incident diagnosis), #1610 (interim mitigation).