feat(agent-config): allow extensible configuration via ConfigExtension trait (#111)

* feat(agent-config): allow extensible configuration via ConfigExtension trait

Introduces a generic `Config<E: ConfigExtension>` type that lets consumers
define additional configuration fields without modifying or copy-pasting the
core crate. Includes a unified `Source` type for dual extraction from both
env vars and YAML, a `merge_fields!` macro to reduce merge boilerplate, and
moves Lambda-specific fields out of the core Config struct.

Also restructures the crate to use a conventional `src/` layout and adds a
README documenting the extension API.

* refactor(agent-config): organize crate into sources/ and deserializers/ modules

Move config source implementations (env, yaml) into `src/sources/` and
type definitions with custom deserialization into `src/deserializers/`.
Re-exports at the crate root preserve all existing import paths.

* refactor(agent-config): move inline deserializer helpers to deserializers/helpers.rs

Extracts all generic deserializer functions (deserialize_optional_string,
deserialize_with_default, duration parsers, key-value parsers, etc.) from
lib.rs into src/deserializers/helpers.rs. Re-exported at the crate root
so all existing import paths continue to work.

* refactor(agent-config): reorder lib.rs so Config struct is visible first

Reorganize lib.rs so an engineer opening the file immediately sees the
Config struct and its fields, followed by the loading entry points, then
the extension trait, builder, and macros. Sections are separated with
headers for quick scanning.

* fix(agent-config): make NoExtensionSource deserialize from map data

Change NoExtensionSource from a unit struct to an empty struct so serde
accepts map-shaped data from figment (env vars / YAML) instead of
expecting null/unit. Prevents spurious warning logs on every get_config()
call when no extension is used.

* docs(agent-config): improve ConfigExtension trait and Source docs

Address reviewer feedback: document field name collision behavior,
clarify Source type requirements and their runtime failure modes,
and expand the README with collision and flat-field explanations.

Author: duncanista

crates/datadog-agent-config/Cargo.toml

@@ -4,9 +4,6 @@ version = "0.1.0"
 edition.workspace = true
 license.workspace = true
 
-[lib]
-path = "mod.rs"
-
 [dependencies]
 figment = { version = "0.10", default-features = false, features = ["yaml", "env"] }
 libdd-trace-obfuscation = { git = "https://github.com/DataDog/libdatadog", rev = "8c88979985154d6d97c0fc2ca9039682981eacad" }

crates/datadog-agent-config/README.md

@@ -0,0 +1,117 @@
+# datadog-agent-config
+
+Shared configuration crate for Datadog serverless agents. Provides a typed `Config` struct with built-in loading from environment variables (`DD_*`) and YAML files (`datadog.yaml`), with environment variables taking precedence.
+
+## Core features
+
+- **Typed config struct** with fields for site, API key, proxy, logs, APM, metrics, DogStatsD, OTLP, and trace propagation
+- **Two built-in sources**: `EnvConfigSource` (reads `DD_*` / `DATADOG_*` env vars) and `YamlConfigSource` (reads `datadog.yaml`)
+- **Graceful deserialization**: every field uses forgiving deserializers that fall back to defaults on bad input, so one misconfigured value never crashes the whole config
+- **Extensible via `ConfigExtension`**: consumers can define additional configuration fields without modifying this crate
+
+## Quick start
+
+```rust
+use std::path::Path;
+use datadog_agent_config::get_config;
+
+let config = get_config(Path::new("/var/task"));
+println!("site: {}", config.site);
+println!("api_key: {}", config.api_key);
+```
+
+## Extensible configuration
+
+Consumers that need additional fields (e.g., Lambda-specific settings) implement the `ConfigExtension` trait instead of forking or copy-pasting the crate.
+
+### 1. Define the extension and its source
+
+```rust
+use datadog_agent_config::{
+    ConfigExtension, merge_fields,
+    deserialize_optional_string, deserialize_optional_bool_from_anything,
+};
+use serde::Deserialize;
+
+#[derive(Debug, PartialEq, Clone)]
+pub struct MyExtension {
+    pub custom_flag: bool,
+    pub custom_name: String,
+}
+
+impl Default for MyExtension {
+    fn default() -> Self {
+        Self { custom_flag: false, custom_name: String::new() }
+    }
+}
+
+/// Source struct for deserialization.
+///
+/// REQUIRED: `#[serde(default)]` on the struct + graceful deserializers on each
+/// field. Without these, a missing or malformed value fails the entire extension
+/// extraction — fields silently fall back to defaults with a warning log.
+#[derive(Debug, Clone, Default, Deserialize)]
+#[serde(default)]
+pub struct MySource {
+    #[serde(deserialize_with = "deserialize_optional_bool_from_anything")]
+    pub custom_flag: Option<bool>,
+    #[serde(deserialize_with = "deserialize_optional_string")]
+    pub custom_name: Option<String>,
+}
+
+impl ConfigExtension for MyExtension {
+    type Source = MySource;
+
+    fn merge_from(&mut self, source: &MySource) {
+        merge_fields!(self, source,
+            string: [custom_name],
+            value:  [custom_flag],
+        );
+    }
+}
+```
+
+### 2. Load config with the extension
+
+```rust
+use std::path::Path;
+use datadog_agent_config::{Config, get_config_with_extension};
+
+type MyConfig = Config<MyExtension>;
+
+let config: MyConfig = get_config_with_extension(Path::new("/var/task"));
+
+// Core fields
+println!("site: {}", config.site);
+
+// Extension fields
+println!("custom_flag: {}", config.ext.custom_flag);
+println!("custom_name: {}", config.ext.custom_name);
+```
+
+Extension fields are populated from both `DD_*` environment variables and `datadog.yaml` using dual extraction: the core fields and extension fields are extracted independently from the same figment instance, so they don't interfere with each other.
+
+### Flat fields only
+
+The single `Source` type is used for both env var and YAML extraction. This works because Figment uses a single key-value namespace per provider, so flat fields map naturally to both `DD_*` env vars and top-level YAML keys. If you need nested YAML structures (e.g., `lambda: { enhanced_metrics: true }`) that differ from the flat env var layout, you'd need separate source structs — implement `merge_from` with a nested source struct and handle the mapping manually.
+
+### Field name collisions
+
+Extension fields are extracted independently from the same figment as core fields. If an extension defines a field with the same name as a core field (e.g., `api_key`), both get their own copy — they don't interfere, but the extension copy does **not** override the core value. Avoid shadowing core field names to prevent confusion.
+
+### merge_fields! macro
+
+The `merge_fields!` macro reduces boilerplate in `merge_from` by batching fields by merge strategy:
+
+- `string`: merges `Option<String>` into `String` (sets value if `Some`)
+- `value`: merges `Option<T>` into `T` (sets value if `Some`)
+- `option`: merges `Option<T>` into `Option<T>` (overwrites if `Some`)
+
+Custom merge logic (e.g., OR-ing two boolean fields together) goes after the macro call in the same method.
+
+## Config loading precedence
+
+1. `Config::default()` (hardcoded defaults)
+2. `datadog.yaml` values (lower priority)
+3. `DD_*` environment variables (highest priority)
+4. Post-processing defaults (site, proxy, logs/APM URL construction)