add feasible_xhat_creator convention for per-scenario-feasible candidates by DLWoodruff · Pull Request #677 · Pyomo/mpi-sppy

DLWoodruff · 2026-05-04T17:18:28Z

Summary

Adds a feasible_xhat_creator(*, solver_name, ..., **kwargs) -> {nodename: np.ndarray} convention for example modules, returning a candidate first-stage point feasible to fix in every real scenario's per-scenario subproblem. Distinct contract from Jensen's xhat (which silently skips infeasibility); needed by downstream consumers (e.g. findW) where the candidate cannot fall back.
Adds mpisppy/utils/xhat_helpers.py with two stateless primitives:
- average_xhat_nonants builds and solves the average scenario; returns ROOT nonants.
- lp_xbar_nonants builds each scenario, LP-relaxes, solves, and returns the probability-weighted average of ROOT nonants across scenarios.
Each caller composes one of these with its own rounding/repair rule -- which rule is appropriate (e.g. np.ceil, np.round, identity, something else) is a model-specific decision that depends on monotonicity of recourse feasibility in each first-stage variable. The choice between the two helpers is also the caller's. The two are not interchangeable for a model like netdes: averaging data and averaging solutions don't commute when first-stage is binary, so average_xhat_nonants can produce a candidate that's infeasible to fix in real scenarios whose individually needed arcs the averaged-data problem missed. (Specifically what netdes_auxiliary uses below: lp_xbar_nonants followed by np.ceil, where the ceil is justified because in netdes opening more arcs only loosens recourse. A different model could pair either helper with a different rounding rule.)
Prototypes the convention in examples/netdes/netdes_auxiliary.py and examples/sslp/sslp_auxiliary.py. Auxiliary files keep the intro examples uncluttered; see Move example auxiliary functions into <model>_auxiliary.py to declutter intro examples #676 for the broader cleanup proposal that would also move average_scenario_creator out of farmer.py and netdes.py.
Not added to farmer (continuous first stage, no rounding needed).
Documentation: new doc/src/feasible_xhat.rst covers the contract, file-layout convention, motivation (the contrast with Jensen's xhat's opportunistic skip-on-infeasibility behavior), the in-cylinder use flags (--<xhatter>-try-feasible-xhat-first, mutually exclusive with --*-try-jensens-first per spoke), and the two engines (average_xhat_nonants and lp_xbar_nonants) with worked examples for farmer (continuous first-stage), netdes (binary, arc-open monotonicity), and sslp (binary, set-covering monotonicity).
Follow-up tracked in rename _JensensMixin to reflect its generic pre-loop xhat candidate role #713: _JensensMixin should be renamed because _try_feasible_xhat is not Jensen's-specific despite living there; it reuses _jensens_evaluate_xhat from the same mixin (and a # NOTE on the method points to the follow-up).

Test plan

mpisppy/tests/test_feasible_xhat.py (22 tests, all pass) -- including a per-scenario fix-feasibility check on netdes that exercises the actual contract: build each real scenario, fix nonants to the candidate, solve, expect optimal/feasible.
test_jensens.py regression -- the helper refactor doesn't touch the Jensen's mixin.
Post-merge sanity slice: test_config, test_incumbent_writing, test_feasible_xhat (120 pass) after merging upstream/main (resolved conflicts in .github/workflows/test_pr_and_main.yml test list and mpisppy/utils/cfg_vanilla.py:shared_options signature change).
ruff check clean on changed Python files.
Test file mpisppy/tests/test_feasible_xhat.py wired into .github/workflows/test_pr_and_main.yml unit-tests job so codecov picks it up.
CI on PR.

🤖 Generated with Claude Code

…ates Downstream consumers (e.g. findW's pin-dual algorithm) need a candidate first-stage point that is feasible to fix in every real scenario's per-scenario subproblem. Jensen's xhat path tolerates infeasibility by silently skipping; the pin-dual path cannot. This is a separate contract. Convention: example modules expose feasible_xhat_creator(*, solver_name, solver_options=None, **kwargs) -> {nodename: np.ndarray}. The rounding rule that turns a deterministic-proxy solution into a feasible candidate is model-specific (depends on monotonicity of recourse feasibility in each first-stage variable) and lives in each module, not in mpi-sppy. mpisppy/utils/xhat_helpers.py provides two engines for the common cases: average_xhat_nonants (solves the average scenario, optionally LP-relaxed) and lp_xbar_nonants (solves each real scenario's LP relaxation, returns probability-weighted xbar). The second is what "feasible everywhere" callers actually want, since LP-relaxing the average scenario can underestimate which first-stage variables need to be active across real scenarios. netdes_auxiliary.feasible_xhat_creator = lp_xbar_nonants + np.ceil, relying on the y[e] - u[e]*x[e] <= 0 monotonicity (more arcs open never tightens recourse). sslp_auxiliary rolls the same pattern with np.round; sslp ships no average_scenario_creator, so the convention explicitly does not require one. Both files live in examples/<model>/<model>_auxiliary.py rather than in <model>.py to keep introductory examples uncluttered for first-time users; see Pyomo#676 for the broader cleanup proposal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

codecov · 2026-05-04T17:47:17Z

Codecov Report

❌ Patch coverage is 96.36364% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.49%. Comparing base (9f02623) to head (f82573a).

Files with missing lines	Patch %	Lines
mpisppy/utils/cfg_vanilla.py	94.28%	2 Missing ⚠️
mpisppy/cylinders/xhatspecific_bounder.py	0.00%	1 Missing ⚠️
mpisppy/utils/xhat_helpers.py	98.03%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #677      +/-   ##
==========================================
+ Coverage   72.36%   72.49%   +0.13%     
==========================================
  Files         161      162       +1     
  Lines       20495    20605     +110     
==========================================
+ Hits        14831    14938     +107     
- Misses       5664     5667       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The convention's call site goes through feasible_xhat_creator, not through average_xhat_nonants directly, so swapping in a model with integer first-stage doesn't require changes at the consumer. To make that point load-bearing, farmer needs a feasible_xhat_creator too -- the trivial continuous-first-stage no-rounding case. farmer_auxiliary delegates to average_xhat_nonants and returns its result unrounded. doc/src/feasible_xhat.rst documents the contract, the two helpers in xhat_helpers.py, the choose-an-engine and choose-a-rounding-rule decisions that belong to the caller, and three worked examples (farmer continuous, netdes lp_xbar+ceil, sslp lp_xbar+round). Listed under Advanced Topics in index.rst, with a see-also note added to the top-of-file warning admonition in jensens.rst pointing readers who need a per-scenario-feasible candidate (vs. a silently-skip-on- infeasibility candidate) to the new doc. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Add test_feasible_xhat.py to both coverage entry points so codecov sees the lines that test_feasible_xhat already exercises locally: - run_coverage.bash: new test_feasible_xhat (serial) phase, mirroring the existing test_jensens phase. - .github/workflows/test_pr_and_main.yml: - "Basic regression tests" job (has cplex/xpress) gains a Test feasible_xhat step alongside Test xhat from file, so the solver- needing tests run and produce coverage data for xhat_helpers.py. - "unit tests (no solver required)" pytest list gains test_feasible_xhat.py alongside test_jensens.py, so the contract tests count in the no-solver job. Without these the test runs fine locally and in any plain `pytest` invocation but is invisible to codecov, so codecov/patch reports 0% patch coverage on the new lines in mpisppy/utils/xhat_helpers.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds --<xhatter>-try-feasible-xhat-first per xhat spoke (xhatshuffle, xhatxbar, xhatlooper, xhatspecific), parallel to the existing --*-try-jensens-first surface. When set, the spoke calls the module's feasible_xhat_creator once before its main loop, fixes the returned ROOT cache as the candidate first-stage, evaluates the expected objective across all real scenarios, and -- if feasible -- sends that as its first inner bound. Implementation lives in _JensensMixin._try_feasible_xhat alongside _try_average_scenario_xhat. Per spoke, the two pre-loop candidates are mutually exclusive: cfg_vanilla._maybe_attach_feasible_xhat raises with a message naming both CLI options if a user enables both --<xhatter>-try-jensens-first and --<xhatter>-try-feasible-xhat-first on the same spoke. The docstring spells out why -- jensens often gives a tighter incumbent when its candidate is feasible, feasible_xhat is guaranteed feasible but can be looser, and per-spoke the right pick depends on model structure. Mixing across spokes is fine. Discovery is flag-gated and lives in cfg_vanilla._find_feasible_xhat_creator: tries the main scenario module first, falls back to importlib-importing <module>_auxiliary, and raises if neither defines feasible_xhat_creator. Auxiliary import is skipped entirely when no flag is set. Tests: extends test_feasible_xhat.py from 10 to 22 tests covering the cfg_vanilla wiring (no-op, raise on missing creator, mutual-exclusion raise, install path), the discovery helper (flag-off no-op, main- module hit, auxiliary fallback, both-missing raise), and the mixin method (flag off, feasible-update, infeasible-skip, bad return-shape). Docs: feasible_xhat.rst gains a Discovery admonition and a "In- cylinder use" section with the four flags and a mutual-exclusion warning admonition. jensens.rst's see-also note is refined to mention the new flags and the per-spoke mutual exclusion. examples/run_all.py gains a netdes smoke entry exercising --xhatshuffle-try-feasible- xhat-first end-to-end via the auxiliary-discovery path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

When average_scenario_creator is not directly usable, the underscore helpers (_scenario_data, _average_scenario_data, _build_model) can still be worth shipping: feasible_xhat_creator may want to build an averaged-data model internally -- including with relaxed Var domains -- to derive a starting first-stage point, since its output is projected and verified feasible per scenario rather than handed off as a Jensen's bound. The data-only-averaging principle binds average_scenario_creator, not feasible_xhat_creator. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

"Pin" isn't standard solver vocabulary; SP textbooks and Pyomo's var.fix() API both use "fix". Search-and-replace across the three paragraphs that used it: - "Pin-dual algorithms" → "Fix-dual algorithms" - "pin the candidate" → "fix the candidate" - "at the pin" → "at the fixed values" - "If the pin is infeasible" → "If the fixed candidate is infeasible" - "feasible to pin in every real scenario" → "feasible to fix in every real scenario" - "any pin is technically feasible" → "any fixed candidate is technically feasible" - "pin-dual machinery" → "fix-dual machinery" No semantic change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Replaces the (made-up) noun phrase "fix-dual algorithms" with a descriptive sentence: algorithms that fix the candidate first stage in every scenario, solve the per-scenario MIP/LP, and read duals from that solve to drive the outer-bound algorithm. Also standardizes "outer algorithm" → "outer-bound algorithm" so the two passes that referred to this class of algorithm match the "outer-bound path" wording already used in the See-also section. No semantic change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The "Why this convention exists" section described a class of outer- bound algorithm that fixes the candidate first stage in every scenario and reads duals from the per-scenario subproblem to drive the outer-bound algorithm. No such consumer exists in mpi-sppy: all four consumers of ``feasible_xhat_creator`` are inner-bound xhat spokes (XhatXbarInnerBound, XhatShuffleInnerBound, XhatLooperInnerBound, XhatSpecificInnerBound, all calling ``_jensens_mixin._try_feasible_xhat``). The Lagrangian/Lagranger/ Subgradient/FWPH outer-bound spokes do not fix nonants. Recast the contrast as inner-bound-vs-inner-bound: Jensen's xhat is opportunistic (silently skips infeasible scenarios via ``_jensens_evaluate_xhat`` returning ``None``); ``feasible_xhat_creator`` tries to guarantee by construction that the candidate is feasible in every real scenario, so the inner-bound spoke that consumes it produces a usable second-stage objective rather than potentially never updating the inner bound. Also softened "provides" → "aims to provide" since the user's repair logic may not always succeed at producing a feasible candidate. Fixed the sslp paragraph's tail clause similarly: the low-slack rationale refers to the inner-bound spoke consumer, not the made-up outer-bound dual-reading consumer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… Methods" Adds a new section "Average-data-based Methods" between "Why this convention exists" and the existing helper/rounding/file-layout/ in-cylinder/worked-example/see-also content, and demotes those eight section headings to subsections (^^^) of the new section. Pure restructuring: no prose changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- Move "File layout" out of "Average-data-based Methods" and place it between "The contract" and "Why this convention exists" as a top-level section. File layout is a cross-cutting convention that applies to every feasible_xhat_creator regardless of which path is used to compute the candidate, so it shouldn't be nested under the average-data-based engines. - Replace the ambiguous "By convention these helpers live in..." with "By convention the model author's feasible_xhat_creator (and any average_scenario_creator) lives in...". The previous wording appeared to point back at average_xhat_nonants/lp_xbar_nonants, but those live in mpisppy/utils/xhat_helpers.py, not in examples/<model>/<model>_auxiliary.py. - Promote "See also" back to a top-level section. The earlier demote-everything-below-Why pass pulled it under "Average-data-based Methods"; restoring it to top level matches the conventional placement for See-also sections. - Fix "Hueristics" → "Heuristics" in the new bottom section heading. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The CLI-flag wiring and mutually-exclusive-with-jensens-first rules apply to any feasible_xhat_creator regardless of which path is used to compute the candidate (data-averaging engines, custom heuristic, or anything else), so they belong at the top level rather than nested under Average-data-based Methods. Placed between "Why this convention exists" and "Average-data-based Methods" so the reading order is: contract → where to put it → motivation → how the framework calls it → specific methods. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

# Conflicts: # .github/workflows/test_pr_and_main.yml # mpisppy/utils/cfg_vanilla.py

Follows the doc-level rename in earlier commits. Three sites in mpisppy/tests/test_feasible_xhat.py: - test_pins_are_feasible_in_every_real_scenario → test_fixes_are_feasible_in_every_real_scenario - class docstring "feasible to pin" → "feasible to fix" - assertion message "netdes pin infeasible" → "netdes fix infeasible" No behavior change; 22 tests in the file still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds a NOTE comment on _try_feasible_xhat flagging that it is not Jensen's-specific despite living on _JensensMixin. The containing class will be renamed in a subsequent PR to reflect that it covers the generic pre-loop xhat candidate machinery, not just Jensen's xhat. Follow-up tracked separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Introduces a new feasible_xhat_creator convention (and wiring/flags) to supply a per-scenario-fix-feasible first-stage candidate for xhat spokes and downstream tools that cannot fall back when a candidate is infeasible in any scenario.

Changes:

Adds mpisppy/utils/xhat_helpers.py with reusable deterministic-proxy engines (average_xhat_nonants, lp_xbar_nonants) for building candidate nonants.
Wires --*-try-feasible-xhat-first into config, spoke construction, and xhat spoke main loops via _JensensMixin._try_feasible_xhat.
Prototypes the convention in example auxiliary modules, adds docs, and adds a dedicated test suite + CI/coverage hooks.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
run_coverage.bash	Adds coverage run for the new feasible_xhat test module.
mpisppy/utils/xhat_helpers.py	New helper utilities to compute candidate ROOT nonants from deterministic proxies.
mpisppy/utils/config.py	Adds CLI/config flags `--<xhatter>-try-feasible-xhat-first` for xhat spokes.
mpisppy/utils/cfg_vanilla.py	Adds feasible-xhat discovery + spoke option wiring and updates spoke factory signatures.
mpisppy/tests/test_feasible_xhat.py	New unit/integration tests for helper contracts, discovery/wiring, and netdes/sslp prototypes.
mpisppy/generic/spokes.py	Threads `feasible_xhat_creator` into xhat spoke construction.
mpisppy/generic/decomp.py	Discovers `feasible_xhat_creator` (including `_auxiliary` fallback) and passes it into spoke building.
mpisppy/cylinders/xhatxbar_bounder.py	Calls `_try_feasible_xhat()` pre-loop when enabled.
mpisppy/cylinders/xhatspecific_bounder.py	Calls `_try_feasible_xhat()` pre-loop when enabled.
mpisppy/cylinders/xhatshufflelooper_bounder.py	Calls `_try_feasible_xhat()` pre-loop when enabled.
mpisppy/cylinders/xhatlooper_bounder.py	Calls `_try_feasible_xhat()` pre-loop when enabled.
mpisppy/cylinders/_jensens_mixin.py	Implements `_try_feasible_xhat` + enable check in the existing mixin.
examples/sslp/sslp_auxiliary.py	Adds sslp `feasible_xhat_creator` prototype using LP-xbar + rounding.
examples/run_all.py	Adds a smoke-test run exercising `--xhatshuffle-try-feasible-xhat-first` on netdes.
examples/netdes/netdes_auxiliary.py	Adds netdes `feasible_xhat_creator` prototype using LP-xbar + ceil.
examples/farmer/farmer_auxiliary.py	Adds farmer `feasible_xhat_creator` prototype using average-scenario solve.
doc/src/jensens.rst	Documents the contrast between Jensen’s opportunistic xhat and the stronger feasible-xhat contract.
doc/src/index.rst	Adds `feasible_xhat.rst` to the docs index.
doc/src/feasible_xhat.rst	New documentation page defining the contract, discovery, flags, and helper engines.
.github/workflows/test_pr_and_main.yml	Adds feasible_xhat tests to CI and to the unit-test list.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- lp_xbar_nonants: materialize scenario_names, raise ValueError on empty input and on zero-sum probabilities (was ZeroDivisionError / silent NaN). - _find_feasible_xhat_creator: narrow ImportError catch to ModuleNotFoundError for the aux module itself; unrelated import failures inside the aux module now bubble up unmasked. Chain with `from e` when reframing. - Tests: cover the two new validation paths and the narrowed catch (solver-free via monkey-patched _solve_and_extract_root / importlib.import_module). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

# Conflicts: # run_coverage.bash

Only some of the mixin's methods are Jensen's-specific; the rest are generic pre-loop xhat candidate machinery shared with the feasible_xhat_creator path added in #677. Rename the class and file to reflect the broader role, drop the misleading _jensens_ prefix from the two generic helpers, and keep the prefix on the genuinely Jensen's-specific methods. - _jensens_mixin.py -> _preloop_xhat_mixin.py - _JensensMixin -> _PreLoopXhatMixin - _jensens_pack_nonant_cache -> _pack_nonant_cache - _jensens_evaluate_xhat -> _evaluate_xhat - remove the #677 follow-up breadcrumb; rewrite module docstring to cover both candidate paths Updates all 7 spokes, cfg_vanilla docstrings, the two test modules, and doc/src/feasible_xhat.rst. No behavior change. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

DLWoodruff marked this pull request as draft May 4, 2026 18:05

DLWoodruff and others added 12 commits May 4, 2026 11:05

Merge remote-tracking branch 'upstream/main' into feasible-xhat-creator

06cca5e

Merge remote-tracking branch 'upstream/main' into feasible-xhat-creator

0a032a2

# Conflicts: # .github/workflows/test_pr_and_main.yml # mpisppy/utils/cfg_vanilla.py

DLWoodruff mentioned this pull request May 18, 2026

rename _JensensMixin to reflect its generic pre-loop xhat candidate role #713

Closed

DLWoodruff marked this pull request as ready for review May 18, 2026 19:40

bknueven requested a review from Copilot May 22, 2026 18:23

Copilot started reviewing on behalf of bknueven May 22, 2026 18:24 View session

Copilot AI reviewed May 22, 2026

View reviewed changes

Comment thread mpisppy/utils/xhat_helpers.py

Comment thread mpisppy/utils/xhat_helpers.py Outdated

Comment thread mpisppy/utils/cfg_vanilla.py Outdated

DLWoodruff and others added 2 commits May 22, 2026 17:56

Merge remote-tracking branch 'upstream/main' into feasible-xhat-creator

f82573a

# Conflicts: # run_coverage.bash

DLWoodruff merged commit d938347 into Pyomo:main May 23, 2026
32 checks passed

DLWoodruff deleted the feasible-xhat-creator branch May 23, 2026 01:17

DLWoodruff mentioned this pull request May 29, 2026

rename _JensensMixin to _PreLoopXhatMixin (#713) #738

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add feasible_xhat_creator convention for per-scenario-feasible candidates#677

add feasible_xhat_creator convention for per-scenario-feasible candidates#677
DLWoodruff merged 17 commits into
Pyomo:mainfrom
DLWoodruff:feasible-xhat-creator

DLWoodruff commented May 4, 2026 •

edited

Loading

Uh oh!

codecov Bot commented May 4, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

DLWoodruff commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

codecov Bot commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DLWoodruff commented May 4, 2026 •

edited

Loading

codecov Bot commented May 4, 2026 •

edited

Loading