docs(open_feature): make scoped agent guidance discoverable

AGENTS.md

@@ -30,7 +30,7 @@ This repository is the source code of a Ruby gem created by Datadog to provide D
 - @lib/datadog/kit - shared product features
 - @lib/datadog/data_streams - Data Streams Monitoring
 - @lib/datadog/di - dynamic instrumentation (`docs/DynamicInstrumentation.md`)
-- @lib/datadog/open_feature - an implementation of OpenFeature Provider https://openfeature.dev/docs/reference/sdks/server/ruby. See `lib/datadog/open_feature/AGENTS.md` for the coding guide for this subtree.
+- @lib/datadog/open_feature - an implementation of OpenFeature Provider https://openfeature.dev/docs/reference/sdks/server/ruby. Before modifying OpenFeature code, specs, or signatures, read and follow `lib/datadog/open_feature/AGENTS.md`.
 - @lib/datadog/opentelemetry - support OpenTelemetry API for tracing and metrics (`docs/OpenTelemetry.md`)
 - @lib/datadog/profiling - profiling
 - @lib/datadog/tracing - distributed tracing

tools/agent-evals/open_feature_guidance/README.md

@@ -0,0 +1,36 @@
+# OpenFeature agent guidance evals
+
+These lightweight evals check that a Codex session started at the repository root recognizes `lib/datadog/open_feature/AGENTS.md` as governing guidance for OpenFeature work.
+
+The first phase covers awareness only. Separate cases verify the root route for changes under:
+
+- `lib/datadog/open_feature/`;
+- `spec/datadog/open_feature/`; and
+- `sig/datadog/open_feature/`.
+
+These cases do not evaluate whether the agent follows the scoped guide's content.
+
+## Next steps
+
+After [#6032](https://github.com/DataDog/dd-trace-rb/pull/6032) makes the repository's RuboCop configuration directly discoverable and enforceable, add a separate content-adherence phase that verifies agents find and run those lint rules. Keep the executable lint configuration as the source of truth instead of duplicating individual cops in agent guidance.
+
+Run all cases from any directory in the checkout:
+
+```bash
+ruby tools/agent-evals/open_feature_guidance/run.rb
+```
+
+To compare discovery before and after a routing change, point the same cases at another checkout:
+
+```bash
+ruby tools/agent-evals/open_feature_guidance/run.rb --root /path/to/pre-change-checkout
+```
+
+Run one case or select a model:
+
+```bash
+ruby tools/agent-evals/open_feature_guidance/run.rb --case lib_awareness
+ruby tools/agent-evals/open_feature_guidance/run.rb --model MODEL_ID
+```
+
+Each case launches a fresh, ephemeral, read-only `codex exec` session rooted at the repository. The runner asserts that both the root and scoped instruction files are identified, and writes no results into the checkout. It is intentionally not wired into CI or LLMObs yet.

tools/agent-evals/open_feature_guidance/cases/lib_awareness.json

@@ -0,0 +1,9 @@
+{
+  "name": "lib_awareness",
+  "description": "OpenFeature production work discovers the scoped guide from the repository root.",
+  "prompt": "Start at the repository root and do not modify files. You are about to change lib/datadog/open_feature/provider.rb. Before beginning the work, return instruction_files as the repository-relative paths of all AGENTS.md files that govern this task, ordered from broadest to most specific.",
+  "expected_instruction_files": [
+    "AGENTS.md",
+    "lib/datadog/open_feature/AGENTS.md"
+  ]
+}

tools/agent-evals/open_feature_guidance/cases/sig_awareness.json

@@ -0,0 +1,9 @@
+{
+  "name": "sig_awareness",
+  "description": "OpenFeature signature work discovers the scoped guide from the repository root.",
+  "prompt": "Start at the repository root and do not modify files. You are about to change sig/datadog/open_feature/provider.rbs. Before beginning the work, return instruction_files as the repository-relative paths of all AGENTS.md files that govern this task, ordered from broadest to most specific.",
+  "expected_instruction_files": [
+    "AGENTS.md",
+    "lib/datadog/open_feature/AGENTS.md"
+  ]
+}

tools/agent-evals/open_feature_guidance/cases/spec_awareness.json

@@ -0,0 +1,9 @@
+{
+  "name": "spec_awareness",
+  "description": "OpenFeature spec work discovers the scoped guide from the repository root.",
+  "prompt": "Start at the repository root and do not modify files. You are about to change spec/datadog/open_feature/provider_spec.rb. Before beginning the work, return instruction_files as the repository-relative paths of all AGENTS.md files that govern this task, ordered from broadest to most specific.",
+  "expected_instruction_files": [
+    "AGENTS.md",
+    "lib/datadog/open_feature/AGENTS.md"
+  ]
+}

tools/agent-evals/open_feature_guidance/output.schema.json

@@ -0,0 +1,17 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "title": "OpenFeature agent guidance eval response",
+  "type": "object",
+  "additionalProperties": false,
+  "required": [
+    "instruction_files"
+  ],
+  "properties": {
+    "instruction_files": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    }
+  }
+}

tools/agent-evals/open_feature_guidance/run.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+require "json"
+require "open3"
+require "optparse"
+require "tmpdir"
+
+DEFAULT_ROOT = File.expand_path("../../..", __dir__)
+CASES_DIR = File.join(__dir__, "cases")
+OUTPUT_SCHEMA = File.join(__dir__, "output.schema.json")
+
+options = {cases: [], model: nil, root: DEFAULT_ROOT}
+OptionParser.new do |parser|
+  parser.banner = "Usage: ruby tools/agent-evals/open_feature_guidance/run.rb [options]"
+  parser.on("--case NAME", "Run one named case (repeatable)") { |name| options[:cases] << name }
+  parser.on("--model MODEL", "Pass a model override to codex exec") { |model| options[:model] = model }
+  parser.on("--root PATH", "Evaluate another repository checkout") { |root| options[:root] = root }
+end.parse!
+
+evaluation_root = File.expand_path(options[:root])
+unless File.directory?(evaluation_root)
+  warn "Unknown evaluation root: #{evaluation_root}"
+  exit 2
+end
+
+case_paths =
+  if options[:cases].empty?
+    Dir[File.join(CASES_DIR, "*.json")].sort
+  else
+    options[:cases].map { |name| File.join(CASES_DIR, "#{name}.json") }
+  end
+
+missing_case_paths = case_paths.reject { |path| File.file?(path) }
+unless missing_case_paths.empty?
+  warn "Unknown eval case(s): #{missing_case_paths.map { |path| File.basename(path, ".json") }.join(", ")}"
+  exit 2
+end
+
+def run_codex(prompt, output_path, model, root)
+  command = [
+    "codex", "exec",
+    "--ephemeral",
+    "--ignore-user-config",
+    "--sandbox", "read-only",
+    "--color", "never",
+    "--cd", root,
+    "--output-schema", OUTPUT_SCHEMA,
+    "--output-last-message", output_path
+  ]
+  command.concat(["--model", model]) if model
+  command << prompt
+
+  Open3.capture3(*command, chdir: root)
+end
+
+def evaluate(case_config, result)
+  instruction_files = result.fetch("instruction_files")
+
+  case_config.fetch("expected_instruction_files").reject do |path|
+    instruction_files.include?(path)
+  end
+end
+
+failures = []
+
+case_paths.each do |case_path|
+  case_config = JSON.parse(File.read(case_path))
+  name = case_config.fetch("name")
+
+  Dir.mktmpdir("open-feature-agent-eval-#{name}") do |tmpdir|
+    output_path = File.join(tmpdir, "result.json")
+    stdout, stderr, status = run_codex(case_config.fetch("prompt"), output_path, options[:model], evaluation_root)
+
+    unless status.success?
+      failures << [name, "codex exec failed (#{status.exitstatus})\n#{stderr}\n#{stdout}"]
+      next
+    end
+
+    begin
+      result = JSON.parse(File.read(output_path))
+      missing_files = evaluate(case_config, result)
+    rescue JSON::ParserError, KeyError, Errno::ENOENT => e
+      failures << [name, "invalid structured result: #{e.message}"]
+      next
+    end
+
+    if missing_files.empty?
+      puts "PASS #{name}"
+    else
+      details = [
+        "missing instruction files: #{missing_files.join(", ")}",
+        "result: #{JSON.pretty_generate(result)}"
+      ]
+      failures << [name, details.join("\n")]
+    end
+  end
+end
+
+unless failures.empty?
+  failures.each do |name, details|
+    warn "FAIL #{name}"
+    warn details
+  end
+  exit 1
+end
+
+puts "#{case_paths.length} eval(s) passed"