config: load and validate generated config models (#4898)

* config: generate model code from json schema

Proposing that the first step towards implementing OpenTelemetry Configuration is to
produce the model code from the json schema. I did a quick search for tools available
to do this and came across datamodel-codegen which seems to do what i expected.

Will open following pull requests (in draft) to use this model code, i just want
to keep these as clearly separated as possible to make reviewing them easier.

Signed-off-by: alex boten <223565+codeboten@users.noreply.github.com>

* update tox command’s deps and allowlist

* add use-union-operator to datamodel-codegen and regenerate models file

* add changelog

* disable union-operator and set target python to 3.10

* ignore generated file for linting

* fix TypeAlias import for python 3.9 in generated models file

* update uv.lock with new dev dependencies

* run precommit

Signed-off-by: alex boten <223565+codeboten@users.noreply.github.com>

* config: add yam/json file loading and env var substitution

* Fix pyright config structure in pyproject.toml

datamodel-codegen section was inserted between [tool.pyright] and its
include/exclude config, causing pyright to check entire repo (599 files)
instead of just included paths. Moved datamodel-codegen section after
pyright config.

* Fix typecheck and pylint errors

- Fix re.sub callback return type (must return str, not str | None)
- Rename DOLLAR_PLACEHOLDER to dollar_placeholder (pylint naming)
- Rename f to temp_file/config_file (pylint naming)
- Update uv.lock

* Add types-PyYAML and file-configuration extra to typecheck

Fixes yaml import warning by installing PyYAML type stubs and the
file-configuration optional dependencies.

* run precommit

* config: add jsonschema validation and vendor OTel config schema

- Bundle schema.json (v1.0.0-rc.3) alongside models.py
- Add jsonschema >= 4.0 to file-configuration optional extra
- Validate parsed config against schema before constructing model,
  with field path included in error messages for nested violations
- Switch datamodel-codegen to use local schema (drop [http] extra)
- Add schema validation tests (wrong type, missing required, nested
  path, enum violation)

Assisted-by: Claude Sonnet 4.6

* config: fix pylint warnings in _loader.py

- Replace global statement with list cache in _get_schema
- Extract _validate_schema helper to reduce branches/statements
  in load_config_file

Assisted-by: Claude Sonnet 4.6

* fix: bump typing_extensions to 4.12.0 for Python 3.13+ compat

jsonschema's `referencing` dep uses TypeVar defaults which requires
typing_extensions>=4.12.0 on Python 3.13+ (4.10.0 raises AttributeError
on TypeVar.__default__).

Assisted-by: Claude Sonnet 4.6

* config: address review feedback - simplify env substitution and bump deps

- replace $$ placeholder trick with single-pass regex matching both $$
  and \${VAR} patterns, correctly handling edge case of $$\${VAR}
- bump pyyaml to 6.0.3 and jsonschema to 4.26.0 in test-requirements
- add PR link to CHANGELOG entry

Co-authored-by: Riccardo Magliocchetti <riccardo.magliocchetti@gmail.com>
Assisted-by: Claude Sonnet 4.6

* fix: downgrade jsonschema to 4.25.1 for Python 3.9 compatibility

jsonschema 4.26.0 requires Python>=3.10.

Assisted-by: Claude Sonnet 4.6

---------

Signed-off-by: alex boten <223565+codeboten@users.noreply.github.com>
Co-authored-by: alex boten <223565+codeboten@users.noreply.github.com>
Co-authored-by: Riccardo Magliocchetti <riccardo.magliocchetti@gmail.com>

Author: 3 people

CHANGELOG.md

@@ -12,6 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Unreleased
 
+- `opentelemetry-sdk`: Add file configuration support with YAML/JSON loading, environment variable substitution, and schema validation against the vendored OTel config JSON schema
+  ([#4898](https://github.com/open-telemetry/opentelemetry-python/pull/4898))
 - Fix intermittent CI failures in `getting-started` and `tracecontext` jobs caused by GitHub git CDN SHA propagation lag by installing contrib packages from the already-checked-out local copy instead of a second git clone
   ([#4958](https://github.com/open-telemetry/opentelemetry-python/pull/4958))
 - `opentelemetry-sdk`: fix type annotations on `MetricReader` and related types

opentelemetry-sdk/pyproject.toml

@@ -32,6 +32,12 @@ dependencies = [
   "typing-extensions >= 4.5.0",
 ]
 
+[project.optional-dependencies]
+file-configuration = [
+  "pyyaml >= 6.0",
+  "jsonschema >= 4.0",
+]
+
 [project.entry-points.opentelemetry_environment_variables]
 sdk = "opentelemetry.sdk.environment_variables"
 

opentelemetry-sdk/src/opentelemetry/sdk/_configuration/file/__init__.py

@@ -0,0 +1,41 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""OpenTelemetry SDK File Configuration.
+
+This module provides support for configuring the OpenTelemetry SDK
+using declarative configuration files (YAML or JSON).
+
+Example:
+    >>> from opentelemetry.sdk._configuration.file import load_config_file
+    >>> config = load_config_file("otel-config.yaml")
+    >>> print(config.file_format)
+    '1.0-rc.3'
+"""
+
+from opentelemetry.sdk._configuration.file._env_substitution import (
+    EnvSubstitutionError,
+    substitute_env_vars,
+)
+from opentelemetry.sdk._configuration.file._loader import (
+    ConfigurationError,
+    load_config_file,
+)
+
+__all__ = [
+    "load_config_file",
+    "substitute_env_vars",
+    "ConfigurationError",
+    "EnvSubstitutionError",
+]

opentelemetry-sdk/src/opentelemetry/sdk/_configuration/file/_env_substitution.py

@@ -0,0 +1,86 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Environment variable substitution for configuration files."""
+
+import logging
+import os
+import re
+
+_logger = logging.getLogger(__name__)
+
+
+class EnvSubstitutionError(Exception):
+    """Raised when environment variable substitution fails.
+
+    This occurs when a ${VAR} reference is found but the environment
+    variable is not set and no default value is provided.
+    """
+
+
+def substitute_env_vars(text: str) -> str:
+    """Substitute environment variables in configuration text.
+
+    Supports the following syntax:
+    - ${VAR}: Substitute with environment variable VAR. Raises error if not found.
+    - ${VAR:-default}: Substitute with VAR if set, otherwise use default value.
+    - $$: Escape sequence for literal $.
+
+    Args:
+        text: Configuration text with potential ${VAR} placeholders.
+
+    Returns:
+        Text with environment variables substituted.
+
+    Raises:
+        EnvSubstitutionError: If a required environment variable is not found.
+
+    Examples:
+        >>> os.environ['SERVICE_NAME'] = 'my-service'
+        >>> substitute_env_vars('name: ${SERVICE_NAME}')
+        'name: my-service'
+        >>> substitute_env_vars('name: ${MISSING:-default}')
+        'name: default'
+        >>> substitute_env_vars('price: $$100')
+        'price: $100'
+    """
+    # Pattern matches $$ (escape sequence) or ${VAR_NAME} / ${VAR_NAME:-default_value}
+    # Handling both in a single pass ensures $$ followed by ${VAR} works correctly
+    pattern = r"\$\$|\$\{([A-Za-z_][A-Za-z0-9_]*)(:-([^}]*))?\}"
+
+    def replace_var(match) -> str:
+        if match.group(1) is None:
+            # Matched $$, return literal $
+            return "$"
+
+        var_name = match.group(1)
+        has_default = match.group(2) is not None
+        default_value = match.group(3) if has_default else None
+
+        value = os.environ.get(var_name)
+
+        if value is None:
+            if has_default:
+                return default_value or ""
+            _logger.error(
+                "Environment variable '%s' not found and no default provided",
+                var_name,
+            )
+            raise EnvSubstitutionError(
+                f"Environment variable '{var_name}' not found and no default provided"
+            )
+
+        return value
+
+    return re.sub(pattern, replace_var, text)

opentelemetry-sdk/src/opentelemetry/sdk/_configuration/file/_loader.py

@@ -0,0 +1,223 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Configuration file loading and parsing."""
+
+import importlib.resources
+import json
+import logging
+from pathlib import Path
+from typing import Any
+
+from opentelemetry.sdk._configuration.file._env_substitution import (
+    substitute_env_vars,
+)
+from opentelemetry.sdk._configuration.models import OpenTelemetryConfiguration
+
+try:
+    import yaml
+except ImportError as exc:
+    raise ImportError(
+        "File configuration requires pyyaml. "
+        "Install with: pip install opentelemetry-sdk[file-configuration]"
+    ) from exc
+
+try:
+    import jsonschema
+except ImportError as exc:
+    raise ImportError(
+        "File configuration requires jsonschema. "
+        "Install with: pip install opentelemetry-sdk[file-configuration]"
+    ) from exc
+
+_schema_cache: list[dict] = []
+
+
+def _get_schema() -> dict:
+    if not _schema_cache:
+        schema_path = (
+            importlib.resources.files("opentelemetry.sdk._configuration")
+            / "schema.json"
+        )
+        _schema_cache.append(
+            json.loads(schema_path.read_text(encoding="utf-8"))
+        )
+    return _schema_cache[0]
+
+
+_logger = logging.getLogger(__name__)
+
+
+class ConfigurationError(Exception):
+    """Raised when configuration file loading, parsing, or validation fails.
+
+    This includes errors from:
+    - File not found or inaccessible
+    - Invalid YAML/JSON syntax
+    - Schema validation failures
+    - Environment variable substitution errors
+    """
+
+
+def load_config_file(file_path: str) -> OpenTelemetryConfiguration:
+    """Load and parse an OpenTelemetry configuration file.
+
+    Supports YAML and JSON formats. Performs environment variable substitution
+    before parsing.
+
+    Args:
+        file_path: Path to the configuration file (.yaml, .yml, or .json).
+
+    Returns:
+        Parsed OpenTelemetryConfiguration object.
+
+    Raises:
+        ConfigurationError: If file cannot be read, parsed, or validated.
+        EnvSubstitutionError: If required environment variable is missing.
+
+    Examples:
+        >>> config = load_config_file("otel-config.yaml")
+        >>> print(config.tracer_provider)
+    """
+    path = Path(file_path)
+
+    if not path.exists():
+        _logger.error("Configuration file not found: %s", file_path)
+        raise ConfigurationError(f"Configuration file not found: {file_path}")
+
+    if not path.is_file():
+        _logger.error("Configuration path is not a file: %s", file_path)
+        raise ConfigurationError(
+            f"Configuration path is not a file: {file_path}"
+        )
+
+    try:
+        with open(path, encoding="utf-8") as config_file:
+            content = config_file.read()
+    except (OSError, IOError) as exc:
+        _logger.exception("Failed to read configuration file: %s", file_path)
+        raise ConfigurationError(
+            f"Failed to read configuration file: {file_path}"
+        ) from exc
+
+    # Perform environment variable substitution
+    try:
+        content = substitute_env_vars(content)
+    except Exception as exc:
+        raise ConfigurationError(
+            f"Environment variable substitution failed: {exc}"
+        ) from exc
+
+    # Parse based on file extension
+    suffix = path.suffix.lower()
+    try:
+        if suffix in (".yaml", ".yml"):
+            data = yaml.safe_load(content)
+        elif suffix == ".json":
+            data = json.loads(content)
+        else:
+            _logger.error("Unsupported file format: %s", suffix)
+            raise ConfigurationError(
+                f"Unsupported file format: {suffix}. Use .yaml, .yml, or .json"
+            )
+    except yaml.YAMLError as exc:
+        _logger.exception("Failed to parse YAML from %s", file_path)
+        raise ConfigurationError(f"Failed to parse YAML: {exc}") from exc
+    except json.JSONDecodeError as exc:
+        _logger.exception("Failed to parse JSON from %s", file_path)
+        raise ConfigurationError(f"Failed to parse JSON: {exc}") from exc
+
+    if data is None:
+        _logger.error("Configuration file is empty: %s", file_path)
+        raise ConfigurationError("Configuration file is empty")
+
+    if not isinstance(data, dict):
+        _logger.error(
+            "Configuration must be a mapping/object, got %s",
+            type(data).__name__,
+        )
+        raise ConfigurationError(
+            f"Configuration must be a mapping/object, got {type(data).__name__}"
+        )
+
+    _validate_schema(data)
+
+    # Convert to OpenTelemetryConfiguration model
+    try:
+        config = _dict_to_model(data)
+    except Exception as exc:
+        _logger.exception(
+            "Failed to validate configuration from %s", file_path
+        )
+        raise ConfigurationError(
+            f"Failed to validate configuration: {exc}"
+        ) from exc
+
+    return config
+
+
+def _validate_schema(data: dict) -> None:
+    """Validate configuration dict against the OTel configuration JSON schema.
+
+    Raises:
+        ConfigurationError: If the data does not conform to the schema.
+    """
+    try:
+        jsonschema.validate(
+            instance=data,
+            schema=_get_schema(),
+            cls=jsonschema.Draft202012Validator,
+        )
+    except jsonschema.ValidationError as exc:
+        raise ConfigurationError(
+            f"Configuration does not match schema: {exc.message} "
+            f"(at {' -> '.join(str(p) for p in exc.absolute_path)})"
+            if exc.absolute_path
+            else f"Configuration does not match schema: {exc.message}"
+        ) from exc
+    except jsonschema.SchemaError as exc:
+        raise ConfigurationError(
+            f"Invalid configuration schema: {exc.message}"
+        ) from exc
+
+
+def _dict_to_model(data: dict[str, Any]) -> OpenTelemetryConfiguration:
+    """Convert dictionary to OpenTelemetryConfiguration model.
+
+    Uses the generated dataclass from models.py. This provides basic
+    validation through dataclass field types.
+
+    Args:
+        data: Parsed configuration dictionary.
+
+    Returns:
+        OpenTelemetryConfiguration instance.
+
+    Raises:
+        TypeError: If data doesn't match expected structure.
+        ValueError: If values are invalid.
+    """
+    # Construct the top-level model from the validated dict. Nested fields
+    # are stored as dicts rather than their dataclass types; factory functions
+    # in later PRs will handle the full recursive conversion when building
+    # SDK objects.
+    try:
+        config = OpenTelemetryConfiguration(**data)
+        return config
+    except TypeError as exc:
+        # Provide more helpful error message
+        raise TypeError(
+            f"Configuration structure is invalid. "
+            f"Check that all required fields are present and correctly typed: {exc}"
+        ) from exc