Pluggable HPO backends (flaml/hyperopt/optuna) via hiertunehub#342
Open
ZmeiGorynych wants to merge 7 commits into
Open
Pluggable HPO backends (flaml/hyperopt/optuna) via hiertunehub#342ZmeiGorynych wants to merge 7 commits into
ZmeiGorynych wants to merge 7 commits into
Conversation
Dependencies for pluggable hyperparameter-optimization backends via hiertunehub: hiertunehub and optuna as hard deps (optuna is the new default backend, numpy-2 clean from 4.0), hyperopt as an optional [hyperopt] extra. hyperopt 0.2.7 imports the deprecated pkg_resources, removed in setuptools 81, so the setuptools<81 ceiling ships with that extra. poetry.lock regenerated. Ports part of PR #337 by Yikun Jiang. Co-authored-by: Yikun Jiang <jiangyikun9663@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
Route CausalTune's estimator-level HPO through hiertunehub.create_tuner so fit() supports flaml, hyperopt and optuna via new framework=/algo= arguments; optuna is the default. search_space() now returns a hiertunehub.SearchSpace and parse_tuner_params() maps the tuner settings to each backend (hyperopt defaults algo to tpe.suggest; optuna/hyperopt raise on an unbounded search). Preserve full FLAML-option parity on framework="flaml": cost_attr, low_cost_partial_config, try_init_configs warm-start (points_to_evaluate) and resume (evaluated_rewards rebuilt from the previous tuner's results). On hyperopt/optuna, resume raises NotImplementedError and try_init_configs warns rather than silently dropping. Also fix two bugs surfaced by the port: - best_score_by_estimator iterated a dict and referenced a removed loop variable in its error message (NameError); it now takes a list. - A failed trial returned -inf regardless of metric direction, which a minimizing backend (the optuna default) treats as best and so poisons best_estimator selection; the failure sentinel is now the worst value for the metric direction (+inf for minimized metrics). The model property now resolves the fitted estimator from scores, since the objective pops it out of the result dict in the non-store_all path. Ports PR #337 by Yikun Jiang. Co-authored-by: Yikun Jiang <jiangyikun9663@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
Cover parse_tuner_params mapping for all three frameworks (incl. the num_samples==-1 rule, hyperopt tpe.suggest default, exact key sets and the unbounded-budget guard), search_space returning a hiertunehub SearchSpace, the list-based best_score_by_estimator (and its ValueError on a malformed entry), end-to-end fits per backend, the optuna default and custom sampler, FLAML parity (cost_attr/warm-start/resume params actually reaching flaml.tune.run, resume rebuilt from the previous tuner), the failed-trial-not-selected-as-best regression, and the best-effort NotImplementedError/warn on non-flaml init/resume. Co-authored-by: Yikun Jiang <jiangyikun9663@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
- test_packaging_dependencies: importorskip tomllib (stdlib only from 3.11) so the Python 3.10 CI job doesn't ModuleNotFoundError. - Rename ct.results.trials -> ct.tuner.trials and trial.last_result -> trial.result in the experiment runner and example notebooks, following the self.results -> self.tuner move. HierTuneHub's Trial exposes .result. Co-authored-by: Yikun Jiang <jiangyikun9663@gmail.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
Describe the optuna (default) / hyperopt / flaml backend choice via the framework and algo arguments to fit(), note that try_init_configs and resume are flaml-only, and mention the optional [hyperopt] extra. Also refresh the stale supported-Python line (3.10-3.12) and add optuna to the dependency lists. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
hiertunehub 0.2.1's to_hyperopt() raises a cryptic NameError when the search space contains only parameterless choices (e.g. estimator_list=["Dummy"] with outcome_model!="auto"), because it imports its hyperopt global only while converting a sampler leaf. Raise a clear ValueError up front instead, pointing at the optuna/flaml alternatives. Also fix the Mac install guide's stale python=3.9.x. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
test_random_from_biased compares an ERUPT estimate against an empirical mean with atol=5e-2, using unseeded global np.random. Each CI process draws different randomness, so it failed intermittently (1 of 6 matrix jobs). Seed the RNG in make_dataset for deterministic data; seed 0 leaves ~16x margin on the tightest assertion. Unrelated to the HPO-backend change but surfaced by this PR's CI run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports #337 (pluggable HPO backends via hiertunehub) onto current
main(Poetry, econml 0.16, dowhy 0.14), fixes the issues found in that PR, and keeps full parity with today's FLAML options. Original work by @ItsYikunJiang.CausalTune.fit()now takesframework=("optuna"default,"hyperopt","flaml") andalgo=, routing estimator-level HPO throughhiertunehub.create_tunerinstead offlaml.tune.run.Key decisions
hiertunehub+optunaare hard deps;hyperoptis an optional[hyperopt]extra (it needssetuptools<81for the removedpkg_resources, so that pin ships only with the extra).framework="flaml":cost_attr,low_cost_partial_config,try_init_configswarm-start (points_to_evaluate) andresume(evaluated_rewards, rebuilt from the previous tuner) all preserved. On hyperopt/optuna,resumeraisesNotImplementedErrorandtry_init_configswarns rather than silently dropping.Fixes beyond the original PR
best_score_by_estimatornow takes a list and no longerNameErrors on a malformed entry (referenced a removed loop var).-inf, which a minimizing backend treats as best, poisoningbest_estimator. The sentinel is now the worst value for the metric direction (+inffor minimized metrics).modelresolves the fitted estimator fromscores(the objective pops it out of the result dict in the non-store_allpath).pyproject.toml/poetry.lock(the PR edited the now-removedsetup.py).Known upstream (hiertunehub 0.2.1) limitations
to_hyperopt()NameErrors when every estimator in the search is parameterless — only affects the optional hyperopt backend with a paramless-only estimator list; optuna/flaml and real multi-estimator searches are unaffected.UserWarnings for the dict-valued estimator categorical; results are correct.Tests
New
tests/causaltune/test_tuner_backends.py(34 tests): per-frameworkparse_tuner_params,SearchSpaceround-trip, list-based scoring, e2e fits per backend, optuna default + custom sampler, FLAML parity (params reachingflaml.tune.run, resume rebuild), the failed-trial regression, and the best-effort guards. Full non-slow suite: 102 passed, 2 skipped, 0 failed.DCO
All three commits are signed off by the author (re-ported, not cherry-picked, so no unsigned commits).
🤖 Generated with Claude Code