add feasible_xhat_creator convention for per-scenario-feasible candidates#677
Open
DLWoodruff wants to merge 15 commits into
Open
add feasible_xhat_creator convention for per-scenario-feasible candidates#677DLWoodruff wants to merge 15 commits into
DLWoodruff wants to merge 15 commits into
Conversation
…ates
Downstream consumers (e.g. findW's pin-dual algorithm) need a candidate
first-stage point that is feasible to fix in every real scenario's
per-scenario subproblem. Jensen's xhat path tolerates infeasibility by
silently skipping; the pin-dual path cannot. This is a separate
contract.
Convention: example modules expose feasible_xhat_creator(*, solver_name,
solver_options=None, **kwargs) -> {nodename: np.ndarray}. The rounding
rule that turns a deterministic-proxy solution into a feasible
candidate is model-specific (depends on monotonicity of recourse
feasibility in each first-stage variable) and lives in each module,
not in mpi-sppy.
mpisppy/utils/xhat_helpers.py provides two engines for the common
cases: average_xhat_nonants (solves the average scenario, optionally
LP-relaxed) and lp_xbar_nonants (solves each real scenario's LP
relaxation, returns probability-weighted xbar). The second is what
"feasible everywhere" callers actually want, since LP-relaxing the
average scenario can underestimate which first-stage variables need
to be active across real scenarios.
netdes_auxiliary.feasible_xhat_creator = lp_xbar_nonants + np.ceil,
relying on the y[e] - u[e]*x[e] <= 0 monotonicity (more arcs open
never tightens recourse). sslp_auxiliary rolls the same pattern with
np.round; sslp ships no average_scenario_creator, so the convention
explicitly does not require one.
Both files live in examples/<model>/<model>_auxiliary.py rather than
in <model>.py to keep introductory examples uncluttered for first-time
users; see Pyomo#676 for the broader cleanup proposal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #677 +/- ##
==========================================
+ Coverage 71.43% 71.55% +0.11%
==========================================
Files 154 155 +1
Lines 19463 19566 +103
==========================================
+ Hits 13903 14000 +97
- Misses 5560 5566 +6 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
The convention's call site goes through feasible_xhat_creator, not through average_xhat_nonants directly, so swapping in a model with integer first-stage doesn't require changes at the consumer. To make that point load-bearing, farmer needs a feasible_xhat_creator too -- the trivial continuous-first-stage no-rounding case. farmer_auxiliary delegates to average_xhat_nonants and returns its result unrounded. doc/src/feasible_xhat.rst documents the contract, the two helpers in xhat_helpers.py, the choose-an-engine and choose-a-rounding-rule decisions that belong to the caller, and three worked examples (farmer continuous, netdes lp_xbar+ceil, sslp lp_xbar+round). Listed under Advanced Topics in index.rst, with a see-also note added to the top-of-file warning admonition in jensens.rst pointing readers who need a per-scenario-feasible candidate (vs. a silently-skip-on- infeasibility candidate) to the new doc. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add test_feasible_xhat.py to both coverage entry points so codecov sees
the lines that test_feasible_xhat already exercises locally:
- run_coverage.bash: new test_feasible_xhat (serial) phase, mirroring
the existing test_jensens phase.
- .github/workflows/test_pr_and_main.yml:
- "Basic regression tests" job (has cplex/xpress) gains a Test
feasible_xhat step alongside Test xhat from file, so the solver-
needing tests run and produce coverage data for xhat_helpers.py.
- "unit tests (no solver required)" pytest list gains
test_feasible_xhat.py alongside test_jensens.py, so the contract
tests count in the no-solver job.
Without these the test runs fine locally and in any plain `pytest`
invocation but is invisible to codecov, so codecov/patch reports 0%
patch coverage on the new lines in mpisppy/utils/xhat_helpers.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds --<xhatter>-try-feasible-xhat-first per xhat spoke (xhatshuffle, xhatxbar, xhatlooper, xhatspecific), parallel to the existing --*-try-jensens-first surface. When set, the spoke calls the module's feasible_xhat_creator once before its main loop, fixes the returned ROOT cache as the candidate first-stage, evaluates the expected objective across all real scenarios, and -- if feasible -- sends that as its first inner bound. Implementation lives in _JensensMixin._try_feasible_xhat alongside _try_average_scenario_xhat. Per spoke, the two pre-loop candidates are mutually exclusive: cfg_vanilla._maybe_attach_feasible_xhat raises with a message naming both CLI options if a user enables both --<xhatter>-try-jensens-first and --<xhatter>-try-feasible-xhat-first on the same spoke. The docstring spells out why -- jensens often gives a tighter incumbent when its candidate is feasible, feasible_xhat is guaranteed feasible but can be looser, and per-spoke the right pick depends on model structure. Mixing across spokes is fine. Discovery is flag-gated and lives in cfg_vanilla._find_feasible_xhat_creator: tries the main scenario module first, falls back to importlib-importing <module>_auxiliary, and raises if neither defines feasible_xhat_creator. Auxiliary import is skipped entirely when no flag is set. Tests: extends test_feasible_xhat.py from 10 to 22 tests covering the cfg_vanilla wiring (no-op, raise on missing creator, mutual-exclusion raise, install path), the discovery helper (flag-off no-op, main- module hit, auxiliary fallback, both-missing raise), and the mixin method (flag off, feasible-update, infeasible-skip, bad return-shape). Docs: feasible_xhat.rst gains a Discovery admonition and a "In- cylinder use" section with the four flags and a mutual-exclusion warning admonition. jensens.rst's see-also note is refined to mention the new flags and the per-spoke mutual exclusion. examples/run_all.py gains a netdes smoke entry exercising --xhatshuffle-try-feasible- xhat-first end-to-end via the auxiliary-discovery path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When average_scenario_creator is not directly usable, the underscore helpers (_scenario_data, _average_scenario_data, _build_model) can still be worth shipping: feasible_xhat_creator may want to build an averaged-data model internally -- including with relaxed Var domains -- to derive a starting first-stage point, since its output is projected and verified feasible per scenario rather than handed off as a Jensen's bound. The data-only-averaging principle binds average_scenario_creator, not feasible_xhat_creator. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
"Pin" isn't standard solver vocabulary; SP textbooks and Pyomo's
var.fix() API both use "fix". Search-and-replace across the three
paragraphs that used it:
- "Pin-dual algorithms" → "Fix-dual algorithms"
- "pin the candidate" → "fix the candidate"
- "at the pin" → "at the fixed values"
- "If the pin is infeasible" → "If the fixed candidate is infeasible"
- "feasible to pin in every real scenario" → "feasible to fix in
every real scenario"
- "any pin is technically feasible" → "any fixed candidate is
technically feasible"
- "pin-dual machinery" → "fix-dual machinery"
No semantic change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the (made-up) noun phrase "fix-dual algorithms" with a descriptive sentence: algorithms that fix the candidate first stage in every scenario, solve the per-scenario MIP/LP, and read duals from that solve to drive the outer-bound algorithm. Also standardizes "outer algorithm" → "outer-bound algorithm" so the two passes that referred to this class of algorithm match the "outer-bound path" wording already used in the See-also section. No semantic change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "Why this convention exists" section described a class of outer- bound algorithm that fixes the candidate first stage in every scenario and reads duals from the per-scenario subproblem to drive the outer-bound algorithm. No such consumer exists in mpi-sppy: all four consumers of ``feasible_xhat_creator`` are inner-bound xhat spokes (XhatXbarInnerBound, XhatShuffleInnerBound, XhatLooperInnerBound, XhatSpecificInnerBound, all calling ``_jensens_mixin._try_feasible_xhat``). The Lagrangian/Lagranger/ Subgradient/FWPH outer-bound spokes do not fix nonants. Recast the contrast as inner-bound-vs-inner-bound: Jensen's xhat is opportunistic (silently skips infeasible scenarios via ``_jensens_evaluate_xhat`` returning ``None``); ``feasible_xhat_creator`` tries to guarantee by construction that the candidate is feasible in every real scenario, so the inner-bound spoke that consumes it produces a usable second-stage objective rather than potentially never updating the inner bound. Also softened "provides" → "aims to provide" since the user's repair logic may not always succeed at producing a feasible candidate. Fixed the sslp paragraph's tail clause similarly: the low-slack rationale refers to the inner-bound spoke consumer, not the made-up outer-bound dual-reading consumer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… Methods" Adds a new section "Average-data-based Methods" between "Why this convention exists" and the existing helper/rounding/file-layout/ in-cylinder/worked-example/see-also content, and demotes those eight section headings to subsections (^^^) of the new section. Pure restructuring: no prose changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Move "File layout" out of "Average-data-based Methods" and place it between "The contract" and "Why this convention exists" as a top-level section. File layout is a cross-cutting convention that applies to every feasible_xhat_creator regardless of which path is used to compute the candidate, so it shouldn't be nested under the average-data-based engines. - Replace the ambiguous "By convention these helpers live in..." with "By convention the model author's feasible_xhat_creator (and any average_scenario_creator) lives in...". The previous wording appeared to point back at average_xhat_nonants/lp_xbar_nonants, but those live in mpisppy/utils/xhat_helpers.py, not in examples/<model>/<model>_auxiliary.py. - Promote "See also" back to a top-level section. The earlier demote-everything-below-Why pass pulled it under "Average-data-based Methods"; restoring it to top level matches the conventional placement for See-also sections. - Fix "Hueristics" → "Heuristics" in the new bottom section heading. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CLI-flag wiring and mutually-exclusive-with-jensens-first rules apply to any feasible_xhat_creator regardless of which path is used to compute the candidate (data-averaging engines, custom heuristic, or anything else), so they belong at the top level rather than nested under Average-data-based Methods. Placed between "Why this convention exists" and "Average-data-based Methods" so the reading order is: contract → where to put it → motivation → how the framework calls it → specific methods. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Conflicts: # .github/workflows/test_pr_and_main.yml # mpisppy/utils/cfg_vanilla.py
Follows the doc-level rename in earlier commits. Three sites in
mpisppy/tests/test_feasible_xhat.py:
- test_pins_are_feasible_in_every_real_scenario
→ test_fixes_are_feasible_in_every_real_scenario
- class docstring "feasible to pin" → "feasible to fix"
- assertion message "netdes pin infeasible" → "netdes fix infeasible"
No behavior change; 22 tests in the file still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a NOTE comment on _try_feasible_xhat flagging that it is not Jensen's-specific despite living on _JensensMixin. The containing class will be renamed in a subsequent PR to reflect that it covers the generic pre-loop xhat candidate machinery, not just Jensen's xhat. Follow-up tracked separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
feasible_xhat_creator(*, solver_name, ..., **kwargs) -> {nodename: np.ndarray}convention for example modules, returning a candidate first-stage point feasible to fix in every real scenario's per-scenario subproblem. Distinct contract from Jensen's xhat (which silently skips infeasibility); needed by downstream consumers (e.g. findW) where the candidate cannot fall back.Adds
mpisppy/utils/xhat_helpers.pywith two stateless primitives:average_xhat_nonantsbuilds and solves the average scenario; returns ROOT nonants.lp_xbar_nonantsbuilds each scenario, LP-relaxes, solves, and returns the probability-weighted average of ROOT nonants across scenarios.Each caller composes one of these with its own rounding/repair rule -- which rule is appropriate (e.g.
np.ceil,np.round, identity, something else) is a model-specific decision that depends on monotonicity of recourse feasibility in each first-stage variable. The choice between the two helpers is also the caller's. The two are not interchangeable for a model like netdes: averaging data and averaging solutions don't commute when first-stage is binary, soaverage_xhat_nonantscan produce a candidate that's infeasible to fix in real scenarios whose individually needed arcs the averaged-data problem missed. (Specifically whatnetdes_auxiliaryuses below:lp_xbar_nonantsfollowed bynp.ceil, where the ceil is justified because in netdes opening more arcs only loosens recourse. A different model could pair either helper with a different rounding rule.)Prototypes the convention in
examples/netdes/netdes_auxiliary.pyandexamples/sslp/sslp_auxiliary.py. Auxiliary files keep the intro examples uncluttered; see Move example auxiliary functions into <model>_auxiliary.py to declutter intro examples #676 for the broader cleanup proposal that would also moveaverage_scenario_creatorout offarmer.pyandnetdes.py.Not added to farmer (continuous first stage, no rounding needed).
Documentation: new
doc/src/feasible_xhat.rstcovers the contract, file-layout convention, motivation (the contrast with Jensen's xhat's opportunistic skip-on-infeasibility behavior), the in-cylinder use flags (--<xhatter>-try-feasible-xhat-first, mutually exclusive with--*-try-jensens-firstper spoke), and the two engines (average_xhat_nonantsandlp_xbar_nonants) with worked examples for farmer (continuous first-stage), netdes (binary, arc-open monotonicity), and sslp (binary, set-covering monotonicity).Follow-up tracked in rename _JensensMixin to reflect its generic pre-loop xhat candidate role #713:
_JensensMixinshould be renamed because_try_feasible_xhatis not Jensen's-specific despite living there; it reuses_jensens_evaluate_xhatfrom the same mixin (and a# NOTEon the method points to the follow-up).Test plan
mpisppy/tests/test_feasible_xhat.py(22 tests, all pass) -- including a per-scenario fix-feasibility check on netdes that exercises the actual contract: build each real scenario, fix nonants to the candidate, solve, expect optimal/feasible.test_jensens.pyregression -- the helper refactor doesn't touch the Jensen's mixin.test_config,test_incumbent_writing,test_feasible_xhat(120 pass) after merging upstream/main (resolved conflicts in.github/workflows/test_pr_and_main.ymltest list andmpisppy/utils/cfg_vanilla.py:shared_optionssignature change).ruff checkclean on changed Python files.mpisppy/tests/test_feasible_xhat.pywired into.github/workflows/test_pr_and_main.ymlunit-tests job so codecov picks it up.🤖 Generated with Claude Code