Skip to content

add feasible_xhat_creator convention for per-scenario-feasible candidates#677

Open
DLWoodruff wants to merge 15 commits into
Pyomo:mainfrom
DLWoodruff:feasible-xhat-creator
Open

add feasible_xhat_creator convention for per-scenario-feasible candidates#677
DLWoodruff wants to merge 15 commits into
Pyomo:mainfrom
DLWoodruff:feasible-xhat-creator

Conversation

@DLWoodruff
Copy link
Copy Markdown
Collaborator

@DLWoodruff DLWoodruff commented May 4, 2026

Summary

  • Adds a feasible_xhat_creator(*, solver_name, ..., **kwargs) -> {nodename: np.ndarray} convention for example modules, returning a candidate first-stage point feasible to fix in every real scenario's per-scenario subproblem. Distinct contract from Jensen's xhat (which silently skips infeasibility); needed by downstream consumers (e.g. findW) where the candidate cannot fall back.

  • Adds mpisppy/utils/xhat_helpers.py with two stateless primitives:

    • average_xhat_nonants builds and solves the average scenario; returns ROOT nonants.
    • lp_xbar_nonants builds each scenario, LP-relaxes, solves, and returns the probability-weighted average of ROOT nonants across scenarios.

    Each caller composes one of these with its own rounding/repair rule -- which rule is appropriate (e.g. np.ceil, np.round, identity, something else) is a model-specific decision that depends on monotonicity of recourse feasibility in each first-stage variable. The choice between the two helpers is also the caller's. The two are not interchangeable for a model like netdes: averaging data and averaging solutions don't commute when first-stage is binary, so average_xhat_nonants can produce a candidate that's infeasible to fix in real scenarios whose individually needed arcs the averaged-data problem missed. (Specifically what netdes_auxiliary uses below: lp_xbar_nonants followed by np.ceil, where the ceil is justified because in netdes opening more arcs only loosens recourse. A different model could pair either helper with a different rounding rule.)

  • Prototypes the convention in examples/netdes/netdes_auxiliary.py and examples/sslp/sslp_auxiliary.py. Auxiliary files keep the intro examples uncluttered; see Move example auxiliary functions into <model>_auxiliary.py to declutter intro examples #676 for the broader cleanup proposal that would also move average_scenario_creator out of farmer.py and netdes.py.

  • Not added to farmer (continuous first stage, no rounding needed).

  • Documentation: new doc/src/feasible_xhat.rst covers the contract, file-layout convention, motivation (the contrast with Jensen's xhat's opportunistic skip-on-infeasibility behavior), the in-cylinder use flags (--<xhatter>-try-feasible-xhat-first, mutually exclusive with --*-try-jensens-first per spoke), and the two engines (average_xhat_nonants and lp_xbar_nonants) with worked examples for farmer (continuous first-stage), netdes (binary, arc-open monotonicity), and sslp (binary, set-covering monotonicity).

  • Follow-up tracked in rename _JensensMixin to reflect its generic pre-loop xhat candidate role #713: _JensensMixin should be renamed because _try_feasible_xhat is not Jensen's-specific despite living there; it reuses _jensens_evaluate_xhat from the same mixin (and a # NOTE on the method points to the follow-up).

Test plan

  • mpisppy/tests/test_feasible_xhat.py (22 tests, all pass) -- including a per-scenario fix-feasibility check on netdes that exercises the actual contract: build each real scenario, fix nonants to the candidate, solve, expect optimal/feasible.
  • test_jensens.py regression -- the helper refactor doesn't touch the Jensen's mixin.
  • Post-merge sanity slice: test_config, test_incumbent_writing, test_feasible_xhat (120 pass) after merging upstream/main (resolved conflicts in .github/workflows/test_pr_and_main.yml test list and mpisppy/utils/cfg_vanilla.py:shared_options signature change).
  • ruff check clean on changed Python files.
  • Test file mpisppy/tests/test_feasible_xhat.py wired into .github/workflows/test_pr_and_main.yml unit-tests job so codecov picks it up.
  • CI on PR.

🤖 Generated with Claude Code

…ates

Downstream consumers (e.g. findW's pin-dual algorithm) need a candidate
first-stage point that is feasible to fix in every real scenario's
per-scenario subproblem. Jensen's xhat path tolerates infeasibility by
silently skipping; the pin-dual path cannot. This is a separate
contract.

Convention: example modules expose feasible_xhat_creator(*, solver_name,
solver_options=None, **kwargs) -> {nodename: np.ndarray}. The rounding
rule that turns a deterministic-proxy solution into a feasible
candidate is model-specific (depends on monotonicity of recourse
feasibility in each first-stage variable) and lives in each module,
not in mpi-sppy.

mpisppy/utils/xhat_helpers.py provides two engines for the common
cases: average_xhat_nonants (solves the average scenario, optionally
LP-relaxed) and lp_xbar_nonants (solves each real scenario's LP
relaxation, returns probability-weighted xbar). The second is what
"feasible everywhere" callers actually want, since LP-relaxing the
average scenario can underestimate which first-stage variables need
to be active across real scenarios.

netdes_auxiliary.feasible_xhat_creator = lp_xbar_nonants + np.ceil,
relying on the y[e] - u[e]*x[e] <= 0 monotonicity (more arcs open
never tightens recourse). sslp_auxiliary rolls the same pattern with
np.round; sslp ships no average_scenario_creator, so the convention
explicitly does not require one.

Both files live in examples/<model>/<model>_auxiliary.py rather than
in <model>.py to keep introductory examples uncluttered for first-time
users; see Pyomo#676 for the broader cleanup proposal.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 4, 2026

Codecov Report

❌ Patch coverage is 96.11650% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.55%. Comparing base (265345b) to head (e6126de).

Files with missing lines Patch % Lines
mpisppy/utils/cfg_vanilla.py 93.93% 2 Missing ⚠️
mpisppy/cylinders/xhatspecific_bounder.py 0.00% 1 Missing ⚠️
mpisppy/utils/xhat_helpers.py 97.82% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #677      +/-   ##
==========================================
+ Coverage   71.43%   71.55%   +0.11%     
==========================================
  Files         154      155       +1     
  Lines       19463    19566     +103     
==========================================
+ Hits        13903    14000      +97     
- Misses       5560     5566       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The convention's call site goes through feasible_xhat_creator, not
through average_xhat_nonants directly, so swapping in a model with
integer first-stage doesn't require changes at the consumer. To make
that point load-bearing, farmer needs a feasible_xhat_creator too --
the trivial continuous-first-stage no-rounding case. farmer_auxiliary
delegates to average_xhat_nonants and returns its result unrounded.

doc/src/feasible_xhat.rst documents the contract, the two helpers in
xhat_helpers.py, the choose-an-engine and choose-a-rounding-rule
decisions that belong to the caller, and three worked examples
(farmer continuous, netdes lp_xbar+ceil, sslp lp_xbar+round). Listed
under Advanced Topics in index.rst, with a see-also note added to the
top-of-file warning admonition in jensens.rst pointing readers who
need a per-scenario-feasible candidate (vs. a silently-skip-on-
infeasibility candidate) to the new doc.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@DLWoodruff DLWoodruff marked this pull request as draft May 4, 2026 18:05
DLWoodruff and others added 12 commits May 4, 2026 11:05
Add test_feasible_xhat.py to both coverage entry points so codecov sees
the lines that test_feasible_xhat already exercises locally:

- run_coverage.bash: new test_feasible_xhat (serial) phase, mirroring
  the existing test_jensens phase.
- .github/workflows/test_pr_and_main.yml:
  - "Basic regression tests" job (has cplex/xpress) gains a Test
    feasible_xhat step alongside Test xhat from file, so the solver-
    needing tests run and produce coverage data for xhat_helpers.py.
  - "unit tests (no solver required)" pytest list gains
    test_feasible_xhat.py alongside test_jensens.py, so the contract
    tests count in the no-solver job.

Without these the test runs fine locally and in any plain `pytest`
invocation but is invisible to codecov, so codecov/patch reports 0%
patch coverage on the new lines in mpisppy/utils/xhat_helpers.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds --<xhatter>-try-feasible-xhat-first per xhat spoke
(xhatshuffle, xhatxbar, xhatlooper, xhatspecific), parallel to the
existing --*-try-jensens-first surface. When set, the spoke calls the
module's feasible_xhat_creator once before its main loop, fixes the
returned ROOT cache as the candidate first-stage, evaluates the
expected objective across all real scenarios, and -- if feasible --
sends that as its first inner bound. Implementation lives in
_JensensMixin._try_feasible_xhat alongside _try_average_scenario_xhat.

Per spoke, the two pre-loop candidates are mutually exclusive:
cfg_vanilla._maybe_attach_feasible_xhat raises with a message naming
both CLI options if a user enables both --<xhatter>-try-jensens-first
and --<xhatter>-try-feasible-xhat-first on the same spoke. The
docstring spells out why -- jensens often gives a tighter incumbent
when its candidate is feasible, feasible_xhat is guaranteed feasible
but can be looser, and per-spoke the right pick depends on model
structure. Mixing across spokes is fine.

Discovery is flag-gated and lives in
cfg_vanilla._find_feasible_xhat_creator: tries the main scenario
module first, falls back to importlib-importing <module>_auxiliary,
and raises if neither defines feasible_xhat_creator. Auxiliary import
is skipped entirely when no flag is set.

Tests: extends test_feasible_xhat.py from 10 to 22 tests covering the
cfg_vanilla wiring (no-op, raise on missing creator, mutual-exclusion
raise, install path), the discovery helper (flag-off no-op, main-
module hit, auxiliary fallback, both-missing raise), and the mixin
method (flag off, feasible-update, infeasible-skip, bad return-shape).

Docs: feasible_xhat.rst gains a Discovery admonition and a "In-
cylinder use" section with the four flags and a mutual-exclusion
warning admonition. jensens.rst's see-also note is refined to mention
the new flags and the per-spoke mutual exclusion. examples/run_all.py
gains a netdes smoke entry exercising --xhatshuffle-try-feasible-
xhat-first end-to-end via the auxiliary-discovery path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When average_scenario_creator is not directly usable, the underscore
helpers (_scenario_data, _average_scenario_data, _build_model) can
still be worth shipping: feasible_xhat_creator may want to build an
averaged-data model internally -- including with relaxed Var domains
-- to derive a starting first-stage point, since its output is
projected and verified feasible per scenario rather than handed off
as a Jensen's bound. The data-only-averaging principle binds
average_scenario_creator, not feasible_xhat_creator.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
"Pin" isn't standard solver vocabulary; SP textbooks and Pyomo's
var.fix() API both use "fix". Search-and-replace across the three
paragraphs that used it:

  - "Pin-dual algorithms" → "Fix-dual algorithms"
  - "pin the candidate" → "fix the candidate"
  - "at the pin" → "at the fixed values"
  - "If the pin is infeasible" → "If the fixed candidate is infeasible"
  - "feasible to pin in every real scenario" → "feasible to fix in
    every real scenario"
  - "any pin is technically feasible" → "any fixed candidate is
    technically feasible"
  - "pin-dual machinery" → "fix-dual machinery"

No semantic change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the (made-up) noun phrase "fix-dual algorithms" with a
descriptive sentence: algorithms that fix the candidate first stage
in every scenario, solve the per-scenario MIP/LP, and read duals
from that solve to drive the outer-bound algorithm.

Also standardizes "outer algorithm" → "outer-bound algorithm" so the
two passes that referred to this class of algorithm match the
"outer-bound path" wording already used in the See-also section.

No semantic change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "Why this convention exists" section described a class of outer-
bound algorithm that fixes the candidate first stage in every
scenario and reads duals from the per-scenario subproblem to drive
the outer-bound algorithm. No such consumer exists in mpi-sppy: all
four consumers of ``feasible_xhat_creator`` are inner-bound xhat
spokes (XhatXbarInnerBound, XhatShuffleInnerBound,
XhatLooperInnerBound, XhatSpecificInnerBound, all calling
``_jensens_mixin._try_feasible_xhat``). The Lagrangian/Lagranger/
Subgradient/FWPH outer-bound spokes do not fix nonants.

Recast the contrast as inner-bound-vs-inner-bound: Jensen's xhat is
opportunistic (silently skips infeasible scenarios via
``_jensens_evaluate_xhat`` returning ``None``); ``feasible_xhat_creator``
tries to guarantee by construction that the candidate is feasible
in every real scenario, so the inner-bound spoke that consumes it
produces a usable second-stage objective rather than potentially
never updating the inner bound.

Also softened "provides" → "aims to provide" since the user's
repair logic may not always succeed at producing a feasible
candidate.

Fixed the sslp paragraph's tail clause similarly: the low-slack
rationale refers to the inner-bound spoke consumer, not the made-up
outer-bound dual-reading consumer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… Methods"

Adds a new section "Average-data-based Methods" between "Why this
convention exists" and the existing helper/rounding/file-layout/
in-cylinder/worked-example/see-also content, and demotes those
eight section headings to subsections (^^^) of the new section.

Pure restructuring: no prose changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Move "File layout" out of "Average-data-based Methods" and place it
  between "The contract" and "Why this convention exists" as a
  top-level section. File layout is a cross-cutting convention that
  applies to every feasible_xhat_creator regardless of which path is
  used to compute the candidate, so it shouldn't be nested under the
  average-data-based engines.
- Replace the ambiguous "By convention these helpers live in..." with
  "By convention the model author's feasible_xhat_creator (and any
  average_scenario_creator) lives in...". The previous wording
  appeared to point back at average_xhat_nonants/lp_xbar_nonants,
  but those live in mpisppy/utils/xhat_helpers.py, not in
  examples/<model>/<model>_auxiliary.py.
- Promote "See also" back to a top-level section. The earlier
  demote-everything-below-Why pass pulled it under
  "Average-data-based Methods"; restoring it to top level matches
  the conventional placement for See-also sections.
- Fix "Hueristics" → "Heuristics" in the new bottom section heading.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CLI-flag wiring and mutually-exclusive-with-jensens-first rules
apply to any feasible_xhat_creator regardless of which path is used
to compute the candidate (data-averaging engines, custom heuristic,
or anything else), so they belong at the top level rather than
nested under Average-data-based Methods.

Placed between "Why this convention exists" and "Average-data-based
Methods" so the reading order is: contract → where to put it →
motivation → how the framework calls it → specific methods.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Conflicts:
#	.github/workflows/test_pr_and_main.yml
#	mpisppy/utils/cfg_vanilla.py
Follows the doc-level rename in earlier commits. Three sites in
mpisppy/tests/test_feasible_xhat.py:

  - test_pins_are_feasible_in_every_real_scenario
      → test_fixes_are_feasible_in_every_real_scenario
  - class docstring "feasible to pin" → "feasible to fix"
  - assertion message "netdes pin infeasible" → "netdes fix infeasible"

No behavior change; 22 tests in the file still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a NOTE comment on _try_feasible_xhat flagging that it is not
Jensen's-specific despite living on _JensensMixin. The containing
class will be renamed in a subsequent PR to reflect that it covers
the generic pre-loop xhat candidate machinery, not just Jensen's
xhat. Follow-up tracked separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@DLWoodruff DLWoodruff marked this pull request as ready for review May 18, 2026 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant