Agenda
- Admin
- Phase 12 / FY27 agreements
- 5/12 meeting topic
- Model Calibration – Partial Automation: Task 1.1: Design (RSG, Driftless, WSP)
- Integration of EET-related development
- Poisson sampling validation tasks and resource requirements
- "Optionality" design concerns
- Other Phase 12 tasks
- RNG Testing (5/5 meeting)
- Telecommute Frequency Model Update and Tests (5/19 meeting)
- EET output analysis: OuterLoop (5/26 meeting)
- ActivitySim Application & Analysis Guide Discussions (feedback reminder)
Notes
Admin Items
Phase 12 Agreements: Joe sent agreements to all agencies last week. They are identical to last year's with updated dates. Agencies are asked to process them promptly; questions should go to Joe.
Upcoming Schedule: Next week's meeting features an RSG report-out on random number generation. The meeting two weeks out has no agenda item yet. Consultant teams were asked to identify work-in-progress tasks that could fill that slot.
Model Calibration – Partial Automation: Task 1.1: Design
Presentation
Motivation
Current ActivitySim calibration relies on ad hoc, agency-specific scripts with no built-in method. This leads to redundant code, repeated data reads/writes across iterations, and inconsistent documentation. The goal is a built-in, automated calibration mode that is efficient, well-structured, and well-documented.
Scope
The work has two phases:
- Phase 1 (Design): Produce a design document and gather consortium input through regular check-ins.
- Phase 2 (Implementation & Testing): Implement and test calibration in three SANDAG model components — Auto Ownership, Work Location Choice, and Work Tour Mode Choice.
Design Considerations
The tool must support calibration of any model coefficient (not just alternative-specific constants), offer multiple adjustment methods, and give users control over convergence criteria, iteration limits, and coefficient bounds. The framework needs access to all ActivitySim data tables and skims. It should also support calibrating multiple sub-models in a bundled loop, write updated coefficient files after each iteration, automatically adjust damping factors when oscillations are detected, and surface convergence warnings proactively.
User expectations: Users are responsible for preparing calibration targets, specifying convergence settings, and ensuring the model is not over-specified. The design document will include guidance on reasonableness checks for calibrated coefficients. As Jeff Newman emphasized, this tool is for users who already understand calibration — it accelerates the process but does not replace domain expertise.
Calibration Methods
Three approaches for updating coefficients:
- Log-ratio (most common): Constant adjustment = ln(observed share / estimated share), with an optional damping factor.
- Odds-ratio: A more complex formula that can reduce oscillations; less commonly used but will be supported.
- Regression, stepwise, distance bin (for distance terms): A number of methods exist for adjusts destination/location choice distance-decay terms. Distance bin and stepwise coefficients will be considered first, regression will be addressed if there is sufficient budget.
Software Implementation
A new calibration run mode will be added to ActivitySim, controlled by a calibration.yaml file. Key settings include a global iteration count, an ordered list of sub-models to calibrate, output directories, and per-component settings (max iterations, tolerance, update rule, and reporting options).
Each sub-model has a CSV-based calibration spec defining the coefficient name, model value expression, target value, constraints, min/max bounds, and damping factor. Users can also supply a Python helper file with custom summary functions for flexible data aggregation and survey weighting.
After each iteration, the tool automatically produces an output table (target vs. modeled values, coefficient changes, convergence flags, clip flags) and default plots (target vs. model bar chart, convergence by iteration, coefficient changes over time).
Discussion Highlights
Coefficient bounds: Users can specify min/max values per coefficient. If a limit is hit for one coefficient, the process continues for others. Joel noted an inherent trade-off between flexibility and the user's ability to manage a large number of options.
Sub-model prioritization: Joel suggested that per-target convergence thresholds can act as a proxy for data confidence — tighter tolerances for high-confidence targets, looser for uncertain ones.
Convergence criteria: Both relative (% difference) and absolute tolerances should be supported, and users should be able to specify them per coefficient. One approach can be to combine a percent-difference setting with an absolute count tolerance for small-share alternatives (e.g., TNC shared trips).
Non-convergence behavior: When a sub-model fails to converge within its iteration limit, the tool will issue a warning and continue to the next sub-model rather than halting. This will be a user-configurable option.
Prior calibration notebooks: RSG agreed to share existing calibration notebooks and outputs in the consortium directory to help members understand what results will look like and inform feedback on report design.
Action Items
| # |
Action |
Owner |
| 1 |
Share today's presentation slides |
RSG |
| 2 |
Share existing calibration notebooks in consortium directory |
RSG |
| 3 |
Continue design document; circulate draft for consortium review |
RSG |
| 4 |
RSG returns next Tuesday for RNG report-out |
RSG |
| 5 |
Agencies review slides/doc and provide feedback on report design |
All agencies |
| 6 |
Consultant teams identify agenda item for meeting in two weeks |
All consultants |
| 7 |
Agencies process Phase 12 agreements |
All agencies |
Agenda
Notes
Admin Items
Phase 12 Agreements: Joe sent agreements to all agencies last week. They are identical to last year's with updated dates. Agencies are asked to process them promptly; questions should go to Joe.
Upcoming Schedule: Next week's meeting features an RSG report-out on random number generation. The meeting two weeks out has no agenda item yet. Consultant teams were asked to identify work-in-progress tasks that could fill that slot.
Model Calibration – Partial Automation: Task 1.1: Design
Presentation
Motivation
Current ActivitySim calibration relies on ad hoc, agency-specific scripts with no built-in method. This leads to redundant code, repeated data reads/writes across iterations, and inconsistent documentation. The goal is a built-in, automated calibration mode that is efficient, well-structured, and well-documented.
Scope
The work has two phases:
Design Considerations
The tool must support calibration of any model coefficient (not just alternative-specific constants), offer multiple adjustment methods, and give users control over convergence criteria, iteration limits, and coefficient bounds. The framework needs access to all ActivitySim data tables and skims. It should also support calibrating multiple sub-models in a bundled loop, write updated coefficient files after each iteration, automatically adjust damping factors when oscillations are detected, and surface convergence warnings proactively.
User expectations: Users are responsible for preparing calibration targets, specifying convergence settings, and ensuring the model is not over-specified. The design document will include guidance on reasonableness checks for calibrated coefficients. As Jeff Newman emphasized, this tool is for users who already understand calibration — it accelerates the process but does not replace domain expertise.
Calibration Methods
Three approaches for updating coefficients:
Software Implementation
A new calibration run mode will be added to ActivitySim, controlled by a
calibration.yamlfile. Key settings include a global iteration count, an ordered list of sub-models to calibrate, output directories, and per-component settings (max iterations, tolerance, update rule, and reporting options).Each sub-model has a CSV-based calibration spec defining the coefficient name, model value expression, target value, constraints, min/max bounds, and damping factor. Users can also supply a Python helper file with custom summary functions for flexible data aggregation and survey weighting.
After each iteration, the tool automatically produces an output table (target vs. modeled values, coefficient changes, convergence flags, clip flags) and default plots (target vs. model bar chart, convergence by iteration, coefficient changes over time).
Discussion Highlights
Coefficient bounds: Users can specify min/max values per coefficient. If a limit is hit for one coefficient, the process continues for others. Joel noted an inherent trade-off between flexibility and the user's ability to manage a large number of options.
Sub-model prioritization: Joel suggested that per-target convergence thresholds can act as a proxy for data confidence — tighter tolerances for high-confidence targets, looser for uncertain ones.
Convergence criteria: Both relative (% difference) and absolute tolerances should be supported, and users should be able to specify them per coefficient. One approach can be to combine a percent-difference setting with an absolute count tolerance for small-share alternatives (e.g., TNC shared trips).
Non-convergence behavior: When a sub-model fails to converge within its iteration limit, the tool will issue a warning and continue to the next sub-model rather than halting. This will be a user-configurable option.
Prior calibration notebooks: RSG agreed to share existing calibration notebooks and outputs in the consortium directory to help members understand what results will look like and inform feedback on report design.
Action Items