2026-04-23 Engineering Team

[joecastiglione]

Agenda

• Poisson Sampling plan for code
• Discussion on value of testing EET against more than SANDAG
• Other topics as requested

[jpn--]

ActivitySim Meeting Notes

Meeting Overview

Agenda

1. Poisson sampling plan — review Jan Zill's written proposal and discuss path forward
2. Discussion on the value of testing the explicit error terms (EET) approach on models beyond SANDAG

Participants

• Jeff Newman (@jpn--) — Driftless
• Joe Castiglione (@joecastiglione) — Zephyr Foundation
• Jan Zill (@janzill) — Outer Loop
• David Hensle (@dhensle) — RSG
• Sijia Wang (@i-am-sijia) — WSP

Duration

Approximately 76 minutes

---

Executive Summary

The meeting focused on the Poisson sampling pull request contributed by VLC, reviewing Jan Zill's effort estimate and integration plan. The group had an extended theoretical discussion on the relative merits of Poisson sampling, explicit error term (EET) sampling, and a newly conceived approach: sampling without replacement using fixed explicit error terms carried forward to a maximum-utility final choice. While the new idea was considered theoretically attractive for stability and runtime, the group acknowledged it requires further mathematical vetting. The meeting concluded with rough consensus to move forward with Poisson sampling as a near-term production win, with Jan taking the lead, while the new sampling-without-replacement idea is explored in parallel for possible inclusion in a future work phase.

---

Meeting Notes

Poisson sampling: status of the pull request

Jan summarized the state of the Poisson sampling pull request originally opened by Matt from VLC. The branch also contains two other changes — sampling consistency of error terms when the sample set changes, and a shadow pricing random number reset — both of which Jan believes are already covered more generally in Outer Loop's explicit error terms branch. Jan noted a merge conflict is causing the branch to fail tests, likely due to the shadow pricing changes rather than the Poisson sampling itself.

Jan proposed isolating the Poisson sampling change from the broader branch and integrating it on top of the existing explicit error term branch. He outlined the following effort estimate and requirements:

• Review the method theoretically and compile references (~5 hours minimum)
• Integrate with existing sampling methods (currently the PR drops all other methods)
• Fix two noted issues in the PR text
• Add unit tests and integration tests (considered mandatory)
• Write documentation to guide users on when to choose each sampling method
• Run a large-scale full-model validation on at least one two-zone and one one-zone model to confirm aggregate statistics are stable across sampling methods

Jan estimated approximately 25 hours for production-ready integration, plus additional time if large-scale validation is included. He noted a Jupyter notebook in the ActivitySim or MTC model repository could serve as the basis for comparison.

A separate, longer-term task was also identified: developing empirical guidelines for choosing sample sizes, which would require investigating shadow pricing interactions and multiple model segments. This was flagged as out of scope for the current effort.

Theoretical discussion: Poisson vs. EET sampling

The group discussed the behavioral and statistical differences between Poisson (probabilistic) sampling and the existing EET (explicit error term / importance) sampling approach.

Jeff explained that in the current fixed-sample-size EET approach, whether a zone is included in the sample of 30 is a function of all other zones' utilities (via the multinomial denominator), creating marginal artifacts. Jan confirmed Poisson sampling has the same property — inclusion probabilities are still derived from a multinomial logit, so a zone's inclusion probability changes when any other zone's utility changes.

Sijia asked for clarification on the mechanics of Poisson sampling. Jan explained that for each alternative, an inclusion probability for a set of size 30 is derived from the sampling utility via the multinomial logit, and then a Monte Carlo draw is made for each alternative independently. This results in a variable-size choice set and uses one fixed uniform random per alternative, giving the method more stability than a single cumulative-distribution draw across all alternatives.

David raised the question of whether the explicit error terms in the current method are seeded by chooser ID and alternative ID. Jan clarified that the sampling step currently seeds by chooser ID and draws sequentially (not by zone ID), but the application step does seed by chooser and alternative, which is effectively equivalent. He noted that modernizing random number generation could make the application step faster by allowing draws directly by alternative ID.

The group agreed that from a behavioral standpoint, Poisson and EET sampling are very close substitutes in practice, and the primary current advantage of Poisson is runtime performance.

New idea: sampling without replacement with fixed error terms

During the discussion, Jeff, David, and Jan developed a new conceptual approach:

• Draw explicit error terms only one time per chooser-alternative pair for the sampling step
• Sample without replacement by taking the top-N (rather than the current process of taking the top-1 and then regenerating random error terms, which is "sampling with replacement")
• Carry the fixed error terms forward to the final choice
• Compute log sums and add to utility, then select the maximum utility alternative — eliminating the need for random draws in the final choice step

The group identified several potential advantages: stronger consistency between the sampling and application steps, elimination of the sample correction factor (McFadden's correction), and significant runtime improvements. Jeff noted this approach would also cleanly handle mode choice if one wished to take the maximum-utility mode-destination combination, though David pointed out that not all mode choice terms are available at the destination choice stage.

Jan flagged that the correction factor becomes complex for sampling without replacement, because removing an alternative from the set requires recalculating probabilities. Jeff and David argued that if the final choice is simply the maximum utility (not a probabilistic draw), no correction factor is needed at all.

Sijia raised a concern about potential double-counting, since the error terms used in sampling are drawn without mode choice log sums, but log sums are added before the final pick. Jan responded that because sampling utility parameters are not independently estimated — they are kept the same as final utility parameters with distance substituted for log sums — the error term variance is approximately the same, and the concern may not apply in practice. However, Jan acknowledged the point needs to be worked through mathematically.

Jeff and the group also noted that this approach does not apply to estimation mode, where probabilities (not maximum utilities) are needed and sample correction factors are essential. The approach is therefore specific to model application.

Path forward

Joe C. expressed a preference for getting Poisson sampling into production as a near-term win, given that VLC has already done substantial work and testing, the lift to finish is modest, and runtime benefits are demonstrated. He noted that some consortium members are not deeply engaged with the technical work and that demonstrating runtime improvements will be better received than opening a new, uncertain line of investigation.

David raised the concern that having three sampling methods (Monte Carlo, Poisson, and a potential new EET-without-replacement approach) would add complexity to configuration, maintenance, and documentation. He noted that if the new approach proves sound, it could be the single recommended method, which would be simpler to explain to users.

Jeff acknowledged the new idea could be significant but agreed it warrants more theoretical development before any commitment. The group reached the following informal consensus:

• Jan will review the mathematics of the new sampling-without-replacement idea and report back next week on whether it appears theoretically valid and practically feasible
• If promising, the group will bring the idea to the broader consortium for discussion before committing to a direction
• In parallel, moving Poisson sampling over the finish line remains the default near-term plan
• Joe will raise the funding and contracting question with the broader group to identify a vehicle for Jan to complete the work, with a fallback of directing funds to Outer Loop

Jeff also noted that random number generation improvements David is working on (expected around May 5) could reduce the runtime gap between methods, potentially making the choice between Poisson and EET sampling less consequential from a performance standpoint.

---

Action Items

• Jan Zill: Review the mathematics of the proposed sampling-without-replacement / fixed-error-term / maximum-utility approach; assess theoretical validity and whether a correction factor is needed or can be eliminated
• Jan Zill: Report findings to the group next week so the team can decide whether to pursue this as an alternative or complementary path to Poisson sampling
• Jeff Newman: Add Jan's Poisson sampling effort-estimate document to the meeting notes
• Joe Castiglione: Put the Poisson sampling decision and funding question on the agenda for the next executive/product meeting (Tuesday)
• Joe Castiglione: Re-explore contracting mechanisms to engage VLC (Tim) and/or direct funds to Outer Loop for Jan to complete the Poisson sampling integration work
• David Hensle: Continue work on random number generation improvements; update expected around May 5
• All: Consider the tradeoffs between (a) getting Poisson sampling into production now, (b) pursuing the new EET-without-replacement idea, or (c) doing both, ahead of next week's discussion

[jpn--]

@janzill's proposed Poisson sampling steps: Poisson sampling validation steps.docx