stabilize observability

Author: Liam-DeVoe

hypothesis-python/RELEASE.rst

@@ -0,0 +1,14 @@
+RELEASE_TYPE: minor
+
+This release adds |settings.observability|, which can be used to enable :ref:`observability <observability>`.
+
+When observability is enabled, Hypothesis writes data about each test case to the ``.hypothesis/observed`` directory in an analysis-ready `jsonlines <https://jsonlines.org/>`_ format. This data is intended to help users who want to dive deep into understanding their tests. It's also intended for people building tools or research on top of Hypothesis.
+
+Observability can be controlled in two ways:
+
+* via the new |settings.observability| argument,
+* or via the ``HYPOTHESIS_OBSERVABILITY`` environment variable.
+
+See :ref:`Configuring observability <observability-configuration>` for details.
+
+If you use VSCode, we recommend the `Tyche <https://github.com/tyche-pbt/tyche-extension>`__ extension, a PBT-specific visualization tool designed for Hypothesis's observability interface.

hypothesis-python/docs/changelog.rst

@@ -774,11 +774,11 @@ Further improve the performance of the constants-collection feature introduced i
 6.135.3 - 2025-06-08
 --------------------
 
-This release adds the experimental and unstable |OBSERVABILITY_CHOICES| option for :ref:`observability <observability>`. If set, the choice sequence is included in ``metadata.choice_nodes``, and choice sequence spans are included in ``metadata.choice_spans``.
+This release adds the experimental and unstable ``OBSERVABILITY_CHOICES`` option for :ref:`observability <observability>`. If set, the choice sequence is included in ``metadata.choice_nodes``, and choice sequence spans are included in ``metadata.choice_spans``.
 
 These are relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence and choice spans.
 
-We are actively working towards a better interface for this. Feel free to use |OBSERVABILITY_CHOICES| to experiment, but don't rely on it yet!
+We are actively working towards a better interface for this. Feel free to use ``OBSERVABILITY_CHOICES`` to experiment, but don't rely on it yet!
 
 .. _v6.135.2:
 

hypothesis-python/docs/prolog.rst

@@ -26,6 +26,7 @@
 .. |settings.suppress_health_check| replace:: :obj:`settings.suppress_health_check <hypothesis.settings.suppress_health_check>`
 .. |settings.stateful_step_count| replace:: :obj:`settings.stateful_step_count <hypothesis.settings.stateful_step_count>`
 .. |settings.backend| replace:: :obj:`settings.backend <hypothesis.settings.backend>`
+.. |settings.observability| replace:: :obj:`settings.observability <hypothesis.settings.observability>`
 
 .. |~settings.max_examples| replace:: :obj:`~hypothesis.settings.max_examples`
 .. |~settings.database| replace:: :obj:`~hypothesis.settings.database`
@@ -38,6 +39,7 @@
 .. |~settings.suppress_health_check| replace:: :obj:`~hypothesis.settings.suppress_health_check`
 .. |~settings.stateful_step_count| replace:: :obj:`~hypothesis.settings.stateful_step_count`
 .. |~settings.backend| replace:: :obj:`~hypothesis.settings.backend`
+.. |~settings.observability| replace:: :obj:`settings.observability <hypothesis.settings.observability>`
 
 .. |settings.register_profile| replace:: :func:`~hypothesis.settings.register_profile`
 .. |settings.get_profile| replace:: :func:`~hypothesis.settings.get_profile`
@@ -65,6 +67,8 @@
 .. |Verbosity.normal| replace:: :obj:`Verbosity.normal <hypothesis.Verbosity.normal>`
 .. |Verbosity.quiet| replace:: :obj:`Verbosity.quiet <hypothesis.Verbosity.quiet>`
 
+.. |ObservabilitySettings| replace:: :obj:`ObservabilitySettings <hypothesis.ObservabilitySettings>`
+
 .. |HypothesisException| replace:: :obj:`HypothesisException <hypothesis.errors.HypothesisException>`
 .. |HypothesisDeprecationWarning| replace:: :obj:`HypothesisDeprecationWarning <hypothesis.errors.HypothesisDeprecationWarning>`
 .. |Flaky| replace:: :obj:`Flaky <hypothesis.errors.Flaky>`
@@ -147,7 +151,6 @@
 .. |with_observability_callback| replace:: :data:`~hypothesis.internal.observability.with_observability_callback`
 .. |observability_enabled| replace:: :data:`~hypothesis.internal.observability.observability_enabled`
 .. |TESTCASE_CALLBACKS| replace:: :data:`~hypothesis.internal.observability.TESTCASE_CALLBACKS`
-.. |OBSERVABILITY_CHOICES| replace:: :data:`~hypothesis.internal.observability.OBSERVABILITY_CHOICES`
 .. |BUFFER_SIZE| replace:: :data:`~hypothesis.internal.conjecture.engine.BUFFER_SIZE`
 .. |MAX_SHRINKS| replace:: :data:`~hypothesis.internal.conjecture.engine.MAX_SHRINKS`
 .. |MAX_SHRINKING_SECONDS| replace:: :data:`~hypothesis.internal.conjecture.engine.MAX_SHRINKING_SECONDS`

hypothesis-python/docs/reference/api.rst

@@ -174,6 +174,8 @@ Settings
 .. autoclass:: hypothesis.Verbosity
     :members:
 
+.. autoclass:: hypothesis.ObservabilitySettings
+
 .. autoclass:: hypothesis.HealthCheck
    :undoc-members:
    :inherited-members:

hypothesis-python/docs/reference/integrations.rst

@@ -75,17 +75,10 @@ If you're interested in similar questions, `drop me an email`_!
 Observability
 -------------
 
-.. note::
+.. tip::
 
   The `Tyche <https://github.com/tyche-pbt/tyche-extension>`__ VSCode extension provides an in-editor UI for observability results generated by Hypothesis. If you want to *view* observability results, rather than programmatically consume or display them, we recommend using Tyche.
 
-.. warning::
-
-    This feature is experimental, and could have breaking changes or even be removed
-    without notice.  Try it out, let us know what you think, but don't rely on it
-    just yet!
-
-
 Motivation
 ~~~~~~~~~~
 
@@ -108,24 +101,23 @@ debuggers such as `rr <https://rr-project.org/>`__ or `pytrace <https://pytrace.
 because there's no good way to compare multiple traces from these tools and their
 Python support is relatively immature.
 
+.. _observability-configuration:
 
-Configuration
-~~~~~~~~~~~~~
+Configuring observability
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
-If you set the ``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY`` environment variable,
-Hypothesis will log various observations to jsonlines files in the
+The standard way to configure observability is with |settings.observability|.
+
+Alternatively, observability can be configured by setting the ``HYPOTHESIS_OBSERVABILITY`` environment variable. If ``HYPOTHESIS_OBSERVABILITY`` is set to one of ``True``, ``true``, or ``1``, |settings.observability| defaults to ``True``. Note that unlike |settings.observability|, ``HYPOTHESIS_OBSERVABILITY`` only configures whether observability is enabled or disabled, not additional options like |ObservabilitySettings|.
+
+When observability is enabled, Hypothesis will log various observations to jsonlines files in the
 ``.hypothesis/observed/`` directory.  You can load and explore these with e.g.
 :func:`pd.read_json(".hypothesis/observed/*_testcases.jsonl", lines=True) <pandas.read_json>`,
 or by using the :pypi:`sqlite-utils` and :pypi:`datasette` libraries::
 
     sqlite-utils insert testcases.db testcases .hypothesis/observed/*_testcases.jsonl --nl --flatten
     datasette serve testcases.db
 
-If you are experiencing a significant slow-down, you can try setting
-``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_NOCOVER`` instead; this will disable coverage information
-collection. This should not be necessary on Python 3.12 or later, where coverage collection is very fast.
-
-
 Collecting more information
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -181,7 +173,7 @@ While the observability format is agnostic to the property-based testing library
 Choices metadata
 ++++++++++++++++
 
-These additional metadata elements are included in ``metadata`` (as e.g. ``metadata["choice_nodes"]`` or ``metadata["choice_spans"]``), if and only if |OBSERVABILITY_CHOICES| is set.
+These additional metadata elements are included in ``metadata`` (as e.g. ``metadata["choice_nodes"]`` or ``metadata["choice_spans"]``), if and only if observability is configured to include choices (see |ObservabilitySettings|).
 
 .. jsonschema:: ./schema_metadata_choices.json
    :hide_key: /additionalProperties, /type

hypothesis-python/docs/reference/internals.rst

@@ -36,8 +36,6 @@ Observability
 .. autofunction:: hypothesis.internal.observability.observability_enabled
 
 .. autodata:: hypothesis.internal.observability.TESTCASE_CALLBACKS
-.. autodata:: hypothesis.internal.observability.OBSERVABILITY_COLLECT_COVERAGE
-.. autodata:: hypothesis.internal.observability.OBSERVABILITY_CHOICES
 
 Engine constants
 ----------------

hypothesis-python/docs/reference/schema_metadata_choices.json

@@ -3,7 +3,7 @@
     "properties": {
         "choice_nodes": {
             "type": ["array", "null"],
-            "description": ".. warning::\n\n  EXPERIMENTAL AND UNSTABLE. This attribute may change format or disappear without warning.\n\nThe sequence of choices made during this test case. This includes the choice value, as well as its constraints and whether it was forced or not.\n\nOnly present if |OBSERVABILITY_CHOICES| is ``True``.\n\n.. note::\n\n  The choice sequence is a relatively low-level implementation detail of Hypothesis, and is exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence.",
+            "description": ".. warning::\n\n  EXPERIMENTAL AND UNSTABLE. This attribute may change format without warning.\n\nThe sequence of choices made during this test case. This includes the choice value, as well as its constraints and whether it was forced or not.\n\n.. note::\n\n  Only present if observability is configured to include choices (see |ObservabilitySettings|).\n\n.. note::\n\n  The choice sequence is a relatively low-level implementation detail of Hypothesis, and is exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence.",
             "items": {
                 "type": "object",
                 "properties": {
@@ -31,7 +31,7 @@
         "choice_spans": {
             "type": "array",
             "items": {"type": "array"},
-            "description": ".. warning::\n\n  EXPERIMENTAL AND UNSTABLE. This attribute may change format or disappear without warning.\n\nThe semantically-meaningful spans of the choice sequence of this test case.\n\nEach span has the format ``[label, start, end, discarded]``, where:\n\n* ``label`` is an opaque integer-value string shared by all spans drawn from a particular strategy.\n* ``start`` and ``end`` are indices into the choice sequence for this span, such that ``choices[start:end]`` are the corresponding choices.\n* ``discarded`` is a boolean indicating whether this span was discarded (see |PrimitiveProvider.span_end|).\n\nOnly present if |OBSERVABILITY_CHOICES| is ``True``.\n\n.. note::\n\n  Spans are a relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| (and particularly |PrimitiveProvider.span_start| and |PrimitiveProvider.span_end|) for more details about spans."
+            "description": ".. warning::\n\n  EXPERIMENTAL AND UNSTABLE. This attribute may change format without warning.\n\nThe semantically-meaningful spans of the choice sequence of this test case.\n\nEach span has the format ``[label, start, end, discarded]``, where:\n\n* ``label`` is an opaque integer-value string shared by all spans drawn from a particular strategy.\n* ``start`` and ``end`` are indices into the choice sequence for this span, such that ``choices[start:end]`` are the corresponding choices.\n* ``discarded`` is a boolean indicating whether this span was discarded (see |PrimitiveProvider.span_end|).\n\n.. note::\n\n  Only present if observability is configured to include choices (see |ObservabilitySettings|).\n\n.. note::\n\n  Spans are a relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| (and particularly |PrimitiveProvider.span_start| and |PrimitiveProvider.span_end|) for more details about spans."
         }
     },
     "required": ["traceback", "reproduction_decorator", "predicates", "backend", "sys.argv", "os.getpid()", "imported_at", "data_status", "interesting_origin", "choice_nodes", "choice_spans"],

hypothesis-python/docs/reference/schema_observations.json

@@ -36,7 +36,7 @@
                 },
                 "coverage": {
                     "type": ["object", "null"],
-                    "description": "Mapping of filename to list of covered line numbers, if coverage information is available, or None if not.  Hypothesis deliberately omits stdlib and site-packages code.",
+                    "description": "Mapping of filename to list of covered line numbers, if coverage information is available, or None if not.  Hypothesis deliberately omits stdlib and site-packages code.\n\n.. note::\n\n  Only present if observability is configured to include coverage (see |ObservabilitySettings|).",
                     "additionalProperties": {
                         "type": "array",
                         "items": {"type": "integer", "minimum": 1},

hypothesis-python/src/hypothesis/__init__.py

@@ -17,7 +17,12 @@
 
 import _hypothesis_globals
 
-from hypothesis._settings import HealthCheck, Phase, Verbosity, settings
+from hypothesis._settings import (
+    HealthCheck,
+    Phase,
+    Verbosity,
+    settings,
+)
 from hypothesis.control import (
     assume,
     currently_in_test_context,
@@ -30,11 +35,13 @@
 from hypothesis.entry_points import run
 from hypothesis.internal.detection import is_hypothesis_test
 from hypothesis.internal.entropy import register_random
+from hypothesis.internal.observability import ObservabilitySettings
 from hypothesis.utils.conventions import infer
 from hypothesis.version import __version__, __version_info__
 
 __all__ = [
     "HealthCheck",
+    "ObservabilitySettings",
     "Phase",
     "Verbosity",
     "__version__",

hypothesis-python/src/hypothesis/_settings.py

@@ -35,6 +35,11 @@
     InvalidArgument,
 )
 from hypothesis.internal.conjecture.providers import AVAILABLE_PROVIDERS
+from hypothesis.internal.observability import (
+    ObservabilitySettings,
+    _ObservabilitySettings,
+    envvar_observability,
+)
 from hypothesis.internal.reflection import get_pretty_function_description
 from hypothesis.internal.validation import check_type, try_convert
 from hypothesis.utils.conventions import not_set
@@ -58,6 +63,7 @@
     "deadline",
     "print_blob",
     "backend",
+    "observability",
 ]
 
 
@@ -529,6 +535,14 @@ def _validate_backend(backend: str) -> str:
     return backend
 
 
+def _validate_observability(observability: Any) -> _ObservabilitySettings:
+    check_type((bool, ObservabilitySettings), observability, name="observability")
+
+    if isinstance(observability, bool):
+        return _ObservabilitySettings(enabled=observability)
+    return _ObservabilitySettings(enabled=True, options=observability)
+
+
 class settingsMeta(type):
     def __init__(cls, *args, **kwargs):
         super().__init__(*args, **kwargs)
@@ -569,7 +583,8 @@ class settings(metaclass=settingsMeta):
     |~settings.max_examples|, |~settings.derandomize|, |~settings.database|,
     |~settings.verbosity|, |~settings.phases|, |~settings.stateful_step_count|,
     |~settings.report_multiple_bugs|, |~settings.suppress_health_check|,
-    |~settings.deadline|, |~settings.print_blob|, and |~settings.backend|.
+    |~settings.deadline|, |~settings.print_blob|, |~settings.backend|, and
+    |~settings.observability|.
 
     A settings object can be applied as a decorator to a test function, in which
     case that test function will use those settings. A test may only have one
@@ -618,7 +633,7 @@ class settings(metaclass=settingsMeta):
             "default",
             max_examples=100,
             derandomize=False,
-            database=not_set,  # see settings.database for the default database
+            database=not_set,  # see settings.database for default behavior
             verbosity=Verbosity.normal,
             phases=tuple(Phase),
             stateful_step_count=50,
@@ -627,6 +642,7 @@ class settings(metaclass=settingsMeta):
             deadline=duration(milliseconds=200),
             print_blob=False,
             backend="hypothesis",
+            observability=False,
         )
 
         ci = settings.register_profile(
@@ -673,6 +689,7 @@ def __init__(
         deadline: int | float | datetime.timedelta | None = not_set,  # type: ignore
         print_blob: bool = not_set,  # type: ignore
         backend: str = not_set,  # type: ignore
+        observability: bool | ObservabilitySettings = not_set,  # type: ignore
     ) -> None:
         self._in_definition = True
 
@@ -699,10 +716,12 @@ def __init__(
             if derandomize is not_set  # type: ignore
             else _validate_choices("derandomize", derandomize, choices=[True, False])
         )
+
         if database is not not_set:  # type: ignore
             database = _validate_database(database)
         self._database = database
         self._cached_database = None
+
         self._verbosity = (
             self._fallback.verbosity  # type: ignore
             if verbosity is not_set  # type: ignore
@@ -746,6 +765,15 @@ def __init__(
             else _validate_backend(backend)
         )
 
+        if observability is not_set:  # type: ignore
+            self._observability = (
+                envvar_observability
+                if self._fallback is None
+                else self._fallback.observability
+            )
+        else:
+            self._observability = _validate_observability(observability)
+
         self._in_definition = False
 
     @property
@@ -1027,6 +1055,35 @@ def backend(self):
         """
         return self._backend
 
+    @property
+    def observability(self):
+        """
+        Controls the :ref:`observability <observability>` behavior of Hypothesis.
+
+        Observability may be enabled or disabled by passing ``True`` or ``False``.
+        You can also pass |ObservabilitySettings| instead to further configure
+        observability behavior.
+
+        For example:
+
+        .. code-block:: python
+
+            from hypothesis import settings, ObservabilitySettings
+
+            # enables observability
+            settings(observability=True)
+
+            # enables observability, with choice sequence data
+            settings(observability=ObservabilitySettings(choices=True))
+
+        If not set, |settings.observability| is inherited from the
+        :ref:`HYPOTHESIS_OBSERVABILITY <observability-configuration>`
+        environment variable.
+
+        |settings.observability| is ``False`` by default.
+        """
+        return self._observability
+
     def __call__(self, test: T) -> T:
         """Make the settings object (self) an attribute of the test.
 
@@ -1199,6 +1256,7 @@ def note_deprecation(
     deadline=duration(milliseconds=200),
     print_blob=False,
     backend="hypothesis",
+    observability=not_set,  # type: ignore
 )
 settings.register_profile("default", default)
 settings.load_profile("default")