Fix selection direction, scorer handling, and fit kwargs; resolve sktime doctest (#182)

Problems
  
- Grid/Random search picked the wrong best `params` (argmin on signed
scores) and didn’t consistently set best_* attributes.
- Sklearn fit `kwargs` were silently dropped by the verify_fit
decorator.
- Doctest failed in sktime classification by accessing private
`scorer._score_func` (not present for _PassthroughScorer).
  - “mixed” experiment objectives had undefined score sign behavior.
- _score_params returned `experiment(**params)`, which calls `score().
score()` applies a sign flip based on the experiment tag, so Grid/Random
search were selecting on “signed” scores and then also making
assumptions about
  direction (previously hardcoded argmin).

  Solutions
  
- Grid/Random search: select min/max based on experiment tag using raw
`evaluate()` values; set `best_params_`, `best_index_`, and compute
signed `best_score_` via `experiment.score(...)`.
- Centralized scorer handling: `_coerce_to_scorer` now attaches a `safe
._metric_func` fallback (e.g., accuracy/r2) and robust sign inference.
- Sklearn decorator: `verify_fit` now preserves *args, **kwargs and
marks fit success.
- BaseExperiment: `score()` raises on "mixed" to avoid undefined
behavior (users should define a concrete direction or override).
- `_score_params` now returns the raw `evaluate()` value (float), not
the signed `score()`. Selecting the best config should use raw objective
values and then choose min or max based on the tag (higher/lower). This
removes ambiguity, avoids double sign logic, and makes selection correct
and explicit. We still compute the public `best_score_` via
`experiment.score(best_params)` so external consumers see the
standardized “higher-is-better” `score_`.

---------

Co-authored-by: Franz Király <fkiraly@gcos.ai>

Author: SimonBlanke

examples/hyperactive_intro.ipynb

@@ -51,17 +51,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "id": "8c428229",
    "metadata": {},
    "outputs": [],
-   "source": [
-    "def sphere(opt):\n",
-    "    x = opt[\"x\"]\n",
-    "    y = opt[\"y\"]\n",
-    "\n",
-    "    return -x**2 - y**2"
-   ]
+   "source": "\"\"\"Hyperactive optimization library introduction notebook.\n\nThis notebook demonstrates unified interfaces for optimizers and experiments\nusing the Hyperactive optimization library.\n\"\"\"\n\n\ndef sphere(opt):\n    \"\"\"Evaluate sphere function for optimization.\n\n    Parameters\n    ----------\n    opt : dict\n        Dictionary with 'x' and 'y' keys containing numeric values.\n\n    Returns\n    -------\n    float\n        Negative sum of squares (for maximization).\n    \"\"\"\n    x = opt[\"x\"]\n    y = opt[\"y\"]\n\n    return -(x**2) - y**2"
   },
   {
    "cell_type": "markdown",
@@ -139,7 +133,7 @@
    "source": [
     "from hyperactive.experiment.bench import Parabola\n",
     "\n",
-    "?Parabola"
+    "Parabola?"
    ]
   },
   {
@@ -319,27 +313,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": null,
    "id": "57110e86",
    "metadata": {},
    "outputs": [],
-   "source": [
-    "from hyperactive.experiment.integrations import SklearnCvExperiment\n",
-    "from sklearn.datasets import load_iris\n",
-    "from sklearn.svm import SVC\n",
-    "from sklearn.metrics import accuracy_score\n",
-    "from sklearn.model_selection import KFold\n",
-    "\n",
-    "X, y = load_iris(return_X_y=True)\n",
-    "\n",
-    "sklearn_exp = SklearnCvExperiment(\n",
-    "    estimator=SVC(),\n",
-    "    scoring=accuracy_score,\n",
-    "    cv=KFold(n_splits=3, shuffle=True),\n",
-    "    X=X,\n",
-    "    y=y,\n",
-    ")"
-   ]
+   "source": "from sklearn.datasets import load_iris\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import KFold\nfrom sklearn.svm import SVC\n\nfrom hyperactive.experiment.integrations import SklearnCvExperiment\n\nX, y = load_iris(return_X_y=True)\n\nsklearn_exp = SklearnCvExperiment(\n    estimator=SVC(),\n    scoring=accuracy_score,\n    cv=KFold(n_splits=3, shuffle=True),\n    X=X,\n    y=y,\n)"
   },
   {
    "cell_type": "markdown",
@@ -500,57 +478,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": null,
    "id": "ab78b796",
    "metadata": {},
    "outputs": [],
-   "source": [
-    "def sphere(opt):\n",
-    "    x = opt[\"x\"]\n",
-    "    y = opt[\"y\"]\n",
-    "\n",
-    "    return -x**2 - y**2"
-   ]
+   "source": "def sphere(opt):\n    \"\"\"Evaluate sphere function for optimization.\n\n    Parameters\n    ----------\n    opt : dict\n        Dictionary with 'x' and 'y' keys containing numeric values.\n\n    Returns\n    -------\n    float\n        Negative sum of squares (for maximization).\n    \"\"\"\n    x = opt[\"x\"]\n    y = opt[\"y\"]\n\n    return -(x**2) - y**2"
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": null,
    "id": "7104e5ec",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "                                                                                                                       \r"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "{'x': np.float64(0.10101010101010033), 'y': np.float64(0.10101010101010033)}"
-      ]
-     },
-     "execution_count": 16,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "import numpy as np\n",
-    "from hyperactive.opt import HillClimbing\n",
-    "\n",
-    "hillclimbing_config = {\n",
-    "    \"search_space\": {\n",
-    "        \"x\": np.linspace(-10, 10, 100),\n",
-    "        \"y\": np.linspace(-10, 10, 100),\n",
-    "    },\n",
-    "    \"n_iter\": 1000,\n",
-    "}\n",
-    "hill_climbing = HillClimbing(**hillclimbing_config, experiment=sphere)\n",
-    "\n",
-    "hill_climbing.solve()"
-   ]
+   "outputs": [],
+   "source": "import numpy as np\n\nfrom hyperactive.opt import HillClimbing\n\nhillclimbing_config = {\n    \"search_space\": {\n        \"x\": np.linspace(-10, 10, 100),\n        \"y\": np.linspace(-10, 10, 100),\n    },\n    \"n_iter\": 1000,\n}\nhill_climbing = HillClimbing(**hillclimbing_config, experiment=sphere)\n\nhill_climbing.solve()"
   },
   {
    "cell_type": "markdown",
@@ -562,56 +502,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": null,
    "id": "5e2328c9",
    "metadata": {},
    "outputs": [],
-   "source": [
-    "from hyperactive.experiment.integrations import SklearnCvExperiment\n",
-    "from sklearn.datasets import load_iris\n",
-    "from sklearn.svm import SVC\n",
-    "from sklearn.metrics import accuracy_score\n",
-    "from sklearn.model_selection import KFold\n",
-    "\n",
-    "X, y = load_iris(return_X_y=True)\n",
-    "\n",
-    "sklearn_exp = SklearnCvExperiment(\n",
-    "    estimator=SVC(),\n",
-    "    scoring=accuracy_score,\n",
-    "    cv=KFold(n_splits=3, shuffle=True),\n",
-    "    X=X,\n",
-    "    y=y,\n",
-    ")"
-   ]
+   "source": "from sklearn.datasets import load_iris\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import KFold\nfrom sklearn.svm import SVC\n\nfrom hyperactive.experiment.integrations import SklearnCvExperiment\n\nX, y = load_iris(return_X_y=True)\n\nsklearn_exp = SklearnCvExperiment(\n    estimator=SVC(),\n    scoring=accuracy_score,\n    cv=KFold(n_splits=3, shuffle=True),\n    X=X,\n    y=y,\n)"
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": null,
    "id": "e9a07a73",
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "{'C': 0.01, 'gamma': 1}"
-      ]
-     },
-     "execution_count": 18,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "from hyperactive.opt import GridSearchSk as GridSearch\n",
-    "\n",
-    "param_grid = {\n",
-    "    \"C\": [0.01, 0.1, 1, 10],\n",
-    "    \"gamma\": [0.0001, 0.01, 0.1, 1, 10],\n",
-    "}\n",
-    "grid_search = GridSearch(param_grid=param_grid, experiment=sklearn_exp)\n",
-    "\n",
-    "grid_search.solve()"
-   ]
+   "outputs": [],
+   "source": "from hyperactive.opt import GridSearchSk as GridSearch\n\nparam_grid = {\n    \"C\": [0.01, 0.1, 1, 10],\n    \"gamma\": [0.0001, 0.01, 0.1, 1, 10],\n}\ngrid_search = GridSearch(param_grid=param_grid, experiment=sklearn_exp)\n\ngrid_search.solve()"
   },
   {
    "cell_type": "markdown",
@@ -623,67 +526,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": null,
    "id": "f9a4d922",
    "metadata": {},
    "outputs": [],
-   "source": [
-    "from hyperactive.experiment.integrations import SklearnCvExperiment\n",
-    "from sklearn.datasets import load_iris\n",
-    "from sklearn.svm import SVC\n",
-    "from sklearn.metrics import accuracy_score\n",
-    "from sklearn.model_selection import KFold\n",
-    "\n",
-    "X, y = load_iris(return_X_y=True)\n",
-    "\n",
-    "sklearn_exp = SklearnCvExperiment(\n",
-    "    estimator=SVC(),\n",
-    "    scoring=accuracy_score,\n",
-    "    cv=KFold(n_splits=3, shuffle=True),\n",
-    "    X=X,\n",
-    "    y=y,\n",
-    ")"
-   ]
+   "source": "from sklearn.datasets import load_iris\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import KFold\nfrom sklearn.svm import SVC\n\nfrom hyperactive.experiment.integrations import SklearnCvExperiment\n\nX, y = load_iris(return_X_y=True)\n\nsklearn_exp = SklearnCvExperiment(\n    estimator=SVC(),\n    scoring=accuracy_score,\n    cv=KFold(n_splits=3, shuffle=True),\n    X=X,\n    y=y,\n)"
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": null,
    "id": "9a13b4f3",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "                                                                                                           \r"
-     ]
-    },
-    {
-     "data": {
-      "text/plain": [
-       "{'C': np.float64(10.0), 'gamma': np.float64(0.1)}"
-      ]
-     },
-     "execution_count": 20,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "import numpy as np\n",
-    "from hyperactive.opt import HillClimbing\n",
-    "\n",
-    "hillclimbing_config = {\n",
-    "    \"search_space\": {\n",
-    "    \"C\": np.array([0.01, 0.1, 1, 10]),\n",
-    "    \"gamma\": np.array([0.0001, 0.01, 0.1, 1, 10]),\n",
-    "    },\n",
-    "    \"n_iter\": 100,\n",
-    "}\n",
-    "hill_climbing = HillClimbing(**hillclimbing_config, experiment=sklearn_exp)\n",
-    "\n",
-    "hill_climbing.solve()"
-   ]
+   "outputs": [],
+   "source": "import numpy as np\n\nfrom hyperactive.opt import HillClimbing\n\nhillclimbing_config = {\n    \"search_space\": {\n        \"C\": np.array([0.01, 0.1, 1, 10]),\n        \"gamma\": np.array([0.0001, 0.01, 0.1, 1, 10]),\n    },\n    \"n_iter\": 100,\n}\nhill_climbing = HillClimbing(**hillclimbing_config, experiment=sklearn_exp)\n\nhill_climbing.solve()"
   },
   {
    "cell_type": "markdown",
@@ -716,31 +571,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": null,
    "id": "4bdf2d49",
    "metadata": {},
    "outputs": [],
-   "source": [
-    "# 1. defining the tuned estimator\n",
-    "from sklearn.svm import SVC\n",
-    "from hyperactive.integrations.sklearn import OptCV\n",
-    "from hyperactive.opt import GridSearchSk as GridSearch\n",
-    "\n",
-    "param_grid = {\"kernel\": [\"linear\", \"rbf\"], \"C\": [1, 10]}\n",
-    "tuned_svc = OptCV(SVC(), optimizer=GridSearch(param_grid))\n",
-    "\n",
-    "# 2. fitting the tuned estimator = tuning the hyperparameters\n",
-    "from sklearn.datasets import load_iris\n",
-    "from sklearn.model_selection import train_test_split\n",
-    "\n",
-    "X, y = load_iris(return_X_y=True)\n",
-    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
-    "\n",
-    "tuned_svc.fit(X_train, y_train)\n",
-    "\n",
-    "# 3. making predictions with the tuned estimator\n",
-    "y_pred = tuned_svc.predict(X_test)"
-   ]
+   "source": "# 1. defining the tuned estimator\nfrom sklearn.datasets import load_iris\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.svm import SVC\n\nfrom hyperactive.integrations.sklearn import OptCV\nfrom hyperactive.opt import GridSearchSk as GridSearch\n\nparam_grid = {\"kernel\": [\"linear\", \"rbf\"], \"C\": [1, 10]}\ntuned_svc = OptCV(SVC(), optimizer=GridSearch(param_grid))\n\n# 2. fitting the tuned estimator = tuning the hyperparameters\nX, y = load_iris(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n\ntuned_svc.fit(X_train, y_train)\n\n# 3. making predictions with the tuned estimator\ny_pred = tuned_svc.predict(X_test)"
   },
   {
    "cell_type": "markdown",
@@ -1198,48 +1033,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": null,
    "id": "f606284b",
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "                                                                                                              \r"
-     ]
-    }
-   ],
-   "source": [
-    "# 1. defining the tuned estimator\n",
-    "from sklearn.svm import SVC\n",
-    "from hyperactive.integrations.sklearn import OptCV\n",
-    "from hyperactive.opt import HillClimbing\n",
-    "\n",
-    "# picking the optimizer is the only part that changes!\n",
-    "hill_climbing_config = {\n",
-    "    \"search_space\": {\n",
-    "    \"C\": np.array([0.01, 0.1, 1, 10]),\n",
-    "    \"gamma\": np.array([0.0001, 0.01, 0.1, 1, 10]),\n",
-    "    },\n",
-    "    \"n_iter\": 100,\n",
-    "}\n",
-    "hill_climbing = HillClimbing(**hill_climbing_config)\n",
-    "\n",
-    "tuned_svc = OptCV(SVC(), optimizer=hill_climbing)\n",
-    "\n",
-    "# 2. fitting the tuned estimator = tuning the hyperparameters\n",
-    "from sklearn.datasets import load_iris\n",
-    "from sklearn.model_selection import train_test_split\n",
-    "\n",
-    "X, y = load_iris(return_X_y=True)\n",
-    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
-    "\n",
-    "tuned_svc.fit(X_train, y_train)\n",
-    "\n",
-    "# 3. making predictions with the tuned estimator\n",
-    "y_pred = tuned_svc.predict(X_test)"
-   ]
+   "outputs": [],
+   "source": "# 1. defining the tuned estimator\nfrom sklearn.datasets import load_iris\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.svm import SVC\n\nfrom hyperactive.integrations.sklearn import OptCV\nfrom hyperactive.opt import HillClimbing\n\n# picking the optimizer is the only part that changes!\nhill_climbing_config = {\n    \"search_space\": {\n        \"C\": np.array([0.01, 0.1, 1, 10]),\n        \"gamma\": np.array([0.0001, 0.01, 0.1, 1, 10]),\n    },\n    \"n_iter\": 100,\n}\nhill_climbing = HillClimbing(**hill_climbing_config)\n\ntuned_svc = OptCV(SVC(), optimizer=hill_climbing)\n\n# 2. fitting the tuned estimator = tuning the hyperparameters\nX, y = load_iris(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n\ntuned_svc.fit(X_train, y_train)\n\n# 3. making predictions with the tuned estimator\ny_pred = tuned_svc.predict(X_test)"
   },
   {
    "cell_type": "markdown",

examples/test_examples.py

@@ -10,10 +10,8 @@
 don't break the examples.
 """
 
-import os
 import sys
 import subprocess
-import tempfile
 from pathlib import Path
 import pytest
 

pyproject.toml

@@ -7,7 +7,7 @@ where = ["src"]
 
 [project]
 name = "hyperactive"
-version = "4.8.1"
+version = "5.0.0"
 description = "An optimization and data collection toolbox for convenient and fast prototyping of computationally expensive models."
 readme = "README.md"
 requires-python = ">=3.9"

src/hyperactive/base/_experiment.py

@@ -1,4 +1,5 @@
 """Base class for experiment."""
+
 # copyright: hyperactive developers, MIT License (see LICENSE file)
 
 import numpy as np
@@ -22,7 +23,7 @@ def __init__(self):
         super().__init__()
 
     def __call__(self, params):
-        """Score parameters. Same as score call, returns only only a first element."""
+        """Score parameters. Same as score call, returns only a first element."""
         score, _ = self.score(params)
         return score
 
@@ -125,6 +126,15 @@ def score(self, params):
             sign = 1
         elif hib == "lower":
             sign = -1
+        elif hib == "mixed":
+            raise NotImplementedError(
+                "Score is undefined for mixed objectives. Override `score` or "
+                "set a concrete objective where higher or lower is better."
+            )
+        else:
+            raise ValueError(
+                f"Unknown value for tag 'property:higher_or_lower_is_better': {hib}"
+            )
 
         eval_res = self.evaluate(params)
         value = eval_res[0]