feat(agentic): Phase 4 — self-improving orchestration (epic #1544) (#1556)

* feat(agentic): Phase 4 — trajectory capture (self-improvement foundation) (#1544)

The raw material every self-improvement mechanism learns from:

- AgentTrajectory: a captured run (transcript, final answer, iterations, usage, agent name, optional evaluator-assigned Reward, metadata).

- ITrajectoryStore (Add/Get/GetAll/Query/Clear) + InMemoryTrajectoryStore.

- TracingAgent<T> (IAgent<T>): transparently records each run of any wrapped agent (executor/supervisor/swarm/memory) to the store without changing behavior; supports per-experiment metadata.

- 5 tests (green net10.0 + net471): records runs, accumulates + queryable, metadata, get-by-id + reward annotation, clear. net471 nullability fix on GetAsync (Task.FromResult<AgentTrajectory?>). No null-forgiving.

Stacked on Phase 3 (#1552). Next: eval harness scoring trajectories with the existing reward models, then learned routing / prompt-opt / online LoRA. Part of epic #1544 (Phase 4).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(agentic): Phase 4 — evaluation harness (score trajectories) (#1544)

Turns captured runs into a measurable quality signal — the continuous-eval step the learners optimize against:

- ITrajectoryEvaluator (trajectory-native scoring seam; higher = better) + DelegateTrajectoryEvaluator (custom scoring functions: exact-match, JSON validity, cost penalties, LLM-as-judge, or an adapter over the reasoning reward models).

- EvaluationReport (count + mean/min/max reward + pass-rate at a threshold) — the scoreboard a self-improvement loop watches.

- TrajectoryEvaluationRunner: scores trajectories (writing AgentTrajectory.Reward) and aggregates the report; EvaluateAsync(list) + EvaluateStoreAsync(store).

- 9 tests total (green net10.0 + net471): exact-match grading + reward annotation, pass-rate threshold, whole-store eval with write-back, empty-set report. No null-forgiving.

Reward models (OutcomeRewardModel/etc.) are reasoning-chain-specific + internal; they plug in via an ITrajectoryEvaluator adapter (follow-up) rather than coupling the harness to reasoning types. Part of epic #1544 (Phase 4).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(agentic): Phase 4 — learned routing (contextual reward-weighted bandit) (#1544)

Replaces a hand-tuned router with one that learns from graded history which agent performs best — the orchestration improves as rewarded runs accumulate:

- LearnedAgentRouter<T> (an IAgent<T>): LearnFrom(trajectories) builds per-agent (optionally per-context) mean-reward stats; routing exploits the best agent, explores unseen candidates first (optimistic), and takes a random agent with probability explorationRate. Optional contextKey learns different best-agents per task type.

- Composable: drops in anywhere an agent is expected and, wrapped by TracingAgent, its routed runs feed back into the store — closing the self-improvement loop.

- 5 tests (green net10.0 + net471): routes to highest-reward agent after learning, explores unseen first, learns per-context (math vs prose), closed-loop capture under the chosen agent, empty/duplicate guards. No null-forgiving.

Part of epic #1544 (Phase 4).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(agentic): Phase 4 — prompt auto-optimization (DSPy-like) (#1544)

Evaluation-driven prompt search: pick the system prompt that scores best on a labeled eval set.

- PromptOptimizer<T>.OptimizeAsync(candidatePrompts, agentFactory, cases): builds an agent per candidate prompt, runs the eval cases, scores each, returns the best prompt + ranked candidates. Default scorer = case-insensitive substring match vs expected; overridable (e.g., reuse an ITrajectoryEvaluator / reward model / length penalty).

- PromptEvalCase (input + expected) + ScoredPrompt + PromptOptimizationResult.

- The optimizer is the SELECTION half; candidate prompts (the SEARCH half) can come from a hand-written set, an LLM proposer, or AiDotNet's existing genetic/beam/annealing prompt optimizers feeding their population in.

- 4 tests (green net10.0 + net471): selects best prompt (ranked), custom scorer, single candidate, empty-input guards. Tests use a real AgentExecutor whose scripted model keys off the system prompt (no manual result construction, no null-forgiving).

Part of epic #1544 (Phase 4).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(agentic): Phase 4 — reward-filtered fine-tuning dataset (online LoRA data prep) (#1544)

The data-preparation half of online LoRA self-improvement (reward-filtered behavior cloning):

- RewardFilteredDatasetBuilder.Build(trajectories): keeps graded runs at/above a reward threshold and distills each into a (prompt, completion) pair — prompt = conversation up to the final turn, completion = final answer; skips ungraded/degenerate runs.

- FineTuningExample (prompt/completion/reward) + FineTuningDataset (examples + mean reward) — ready to hand to the repo's LoRA/FineTuning trainer to produce an adapter for the local model.

- 3 tests (green net10.0 + net471): keeps only high-reward runs (prompt excludes completion), skips ungraded/too-short, threshold controls inclusion. No null-forgiving.

The actual LoRA training (dataset -> adapter on the local model) is the model-layer step performed by the existing trainer; this is the agentic step that decides WHAT to learn from. Part of epic #1544 (Phase 4).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: franklinic <franklin@ivorycloud.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: 3 people

src/Agentic/SelfImproving/AgentTrajectory.cs

@@ -0,0 +1,89 @@
+using AiDotNet.Agentic.Models;
+
+// Disambiguate from the legacy AiDotNet.PromptEngineering.Templates.ChatMessage (global using).
+using ChatMessage = AiDotNet.Agentic.Models.ChatMessage;
+
+namespace AiDotNet.Agentic.SelfImproving;
+
+/// <summary>
+/// A captured record of one agent run — the structured "trajectory" the self-improving layer learns from.
+/// It pairs what the agent did (the full message transcript, the final answer, step count, token usage) with
+/// an optional quality <see cref="Reward"/> assigned later by an evaluator.
+/// </summary>
+/// <remarks>
+/// <para>
+/// Trajectories are the raw material for every self-improvement mechanism: continuous evaluation scores them,
+/// learned routers/tool policies train on them, prompt optimizers measure prompts against them, and
+/// reward-filtered trajectories become fine-tuning (LoRA) data. Capturing a run is non-invasive — a
+/// <see cref="TracingAgent{T}"/> wraps any agent and records each run without changing its behavior.
+/// </para>
+/// <para><b>For Beginners:</b> Think of this as the flight recorder for an agent: it saves exactly what was
+/// said and done on a run, plus (once graded) how good the outcome was. Collect many of these and the system
+/// can learn what works and get better over time.
+/// </para>
+/// </remarks>
+public sealed class AgentTrajectory
+{
+    /// <summary>
+    /// Initializes a new trajectory.
+    /// </summary>
+    /// <param name="id">The unique id for this trajectory.</param>
+    /// <param name="agentName">The name of the agent that produced the run.</param>
+    /// <param name="messages">The full conversation transcript the run produced.</param>
+    /// <param name="finalText">The agent's final answer.</param>
+    /// <param name="iterations">The number of model calls the run took.</param>
+    /// <param name="usage">Aggregate token usage, when available.</param>
+    /// <param name="reward">An optional quality score (assigned by an evaluator). <c>null</c> until graded.</param>
+    /// <param name="metadata">Optional key/value metadata.</param>
+    /// <exception cref="ArgumentNullException">Thrown when <paramref name="id"/>, <paramref name="agentName"/>, <paramref name="messages"/>, or <paramref name="finalText"/> is <c>null</c>.</exception>
+    public AgentTrajectory(
+        string id,
+        string agentName,
+        IReadOnlyList<ChatMessage> messages,
+        string finalText,
+        int iterations,
+        ChatUsage? usage = null,
+        double? reward = null,
+        IReadOnlyDictionary<string, string>? metadata = null)
+    {
+        Guard.NotNullOrWhiteSpace(id);
+        Guard.NotNull(agentName);
+        Guard.NotNull(messages);
+        Guard.NotNull(finalText);
+        Id = id;
+        AgentName = agentName;
+        Messages = messages;
+        FinalText = finalText;
+        Iterations = iterations;
+        Usage = usage;
+        Reward = reward;
+        Metadata = metadata;
+    }
+
+    /// <summary>Gets the unique id for this trajectory.</summary>
+    public string Id { get; }
+
+    /// <summary>Gets the name of the agent that produced the run.</summary>
+    public string AgentName { get; }
+
+    /// <summary>Gets the full conversation transcript the run produced.</summary>
+    public IReadOnlyList<ChatMessage> Messages { get; }
+
+    /// <summary>Gets the agent's final answer.</summary>
+    public string FinalText { get; }
+
+    /// <summary>Gets the number of model calls the run took.</summary>
+    public int Iterations { get; }
+
+    /// <summary>Gets aggregate token usage, or <c>null</c> when not reported.</summary>
+    public ChatUsage? Usage { get; }
+
+    /// <summary>
+    /// Gets or sets the quality score for this trajectory, assigned by an evaluator (higher is better), or
+    /// <c>null</c> while ungraded.
+    /// </summary>
+    public double? Reward { get; set; }
+
+    /// <summary>Gets optional key/value metadata, or <c>null</c> when none was supplied.</summary>
+    public IReadOnlyDictionary<string, string>? Metadata { get; }
+}

src/Agentic/SelfImproving/DelegateTrajectoryEvaluator.cs

@@ -0,0 +1,34 @@
+namespace AiDotNet.Agentic.SelfImproving;
+
+/// <summary>
+/// An <see cref="ITrajectoryEvaluator"/> backed by a user-supplied scoring function — the general-purpose
+/// hook for custom rewards (exact-match against a labeled answer, regex/JSON validity, cost penalties, or any
+/// combination).
+/// </summary>
+/// <remarks>
+/// <para><b>For Beginners:</b> The do-it-yourself grader: you provide a small function that looks at a run and
+/// returns a score, and this turns it into an evaluator the rest of the system can use.
+/// </para>
+/// </remarks>
+public sealed class DelegateTrajectoryEvaluator : ITrajectoryEvaluator
+{
+    private readonly Func<AgentTrajectory, double> _score;
+
+    /// <summary>
+    /// Initializes a new evaluator from a synchronous scoring function.
+    /// </summary>
+    /// <param name="score">The function that scores a trajectory (higher is better).</param>
+    /// <exception cref="ArgumentNullException">Thrown when <paramref name="score"/> is <c>null</c>.</exception>
+    public DelegateTrajectoryEvaluator(Func<AgentTrajectory, double> score)
+    {
+        Guard.NotNull(score);
+        _score = score;
+    }
+
+    /// <inheritdoc/>
+    public Task<double> EvaluateAsync(AgentTrajectory trajectory, CancellationToken cancellationToken = default)
+    {
+        Guard.NotNull(trajectory);
+        return Task.FromResult(_score(trajectory));
+    }
+}

src/Agentic/SelfImproving/EvaluationReport.cs

@@ -0,0 +1,51 @@
+namespace AiDotNet.Agentic.SelfImproving;
+
+/// <summary>
+/// Aggregate statistics from evaluating a set of trajectories: how many were scored, the reward
+/// distribution, and the fraction that met a pass threshold. This is the scoreboard a self-improvement loop
+/// watches to know whether it is getting better.
+/// </summary>
+/// <remarks>
+/// <para><b>For Beginners:</b> The report card for a batch of runs — average score, best and worst, and what
+/// percentage "passed". Compare reports before and after a change to see whether the agents improved.
+/// </para>
+/// </remarks>
+public sealed class EvaluationReport
+{
+    /// <summary>
+    /// Initializes a new report.
+    /// </summary>
+    /// <param name="count">The number of trajectories scored.</param>
+    /// <param name="meanReward">The mean reward.</param>
+    /// <param name="minReward">The minimum reward.</param>
+    /// <param name="maxReward">The maximum reward.</param>
+    /// <param name="passRate">The fraction of trajectories with reward &gt;= <paramref name="passThreshold"/> (0–1).</param>
+    /// <param name="passThreshold">The threshold used to compute <paramref name="passRate"/>.</param>
+    public EvaluationReport(int count, double meanReward, double minReward, double maxReward, double passRate, double passThreshold)
+    {
+        Count = count;
+        MeanReward = meanReward;
+        MinReward = minReward;
+        MaxReward = maxReward;
+        PassRate = passRate;
+        PassThreshold = passThreshold;
+    }
+
+    /// <summary>Gets the number of trajectories scored.</summary>
+    public int Count { get; }
+
+    /// <summary>Gets the mean reward across scored trajectories (0 when none).</summary>
+    public double MeanReward { get; }
+
+    /// <summary>Gets the minimum reward (0 when none).</summary>
+    public double MinReward { get; }
+
+    /// <summary>Gets the maximum reward (0 when none).</summary>
+    public double MaxReward { get; }
+
+    /// <summary>Gets the fraction of trajectories meeting the pass threshold (0–1).</summary>
+    public double PassRate { get; }
+
+    /// <summary>Gets the threshold used to compute <see cref="PassRate"/>.</summary>
+    public double PassThreshold { get; }
+}

src/Agentic/SelfImproving/FineTuningDataset.cs

@@ -0,0 +1,35 @@
+namespace AiDotNet.Agentic.SelfImproving;
+
+/// <summary>
+/// A set of reward-filtered <see cref="FineTuningExample"/>s ready to hand to a LoRA / fine-tuning trainer —
+/// the bridge from "the agent did well on these runs" to "make the local model better at them".
+/// </summary>
+/// <remarks>
+/// <para><b>For Beginners:</b> A study packet of known-good question/answer pairs. Feed it to the existing
+/// LoRA fine-tuning trainer and the local model improves at the kinds of task it already handled well —
+/// learning from its own successes (reward-filtered behavior cloning).
+/// </para>
+/// </remarks>
+public sealed class FineTuningDataset
+{
+    /// <summary>
+    /// Initializes a new dataset.
+    /// </summary>
+    /// <param name="examples">The fine-tuning examples.</param>
+    /// <exception cref="ArgumentNullException">Thrown when <paramref name="examples"/> is <c>null</c>.</exception>
+    public FineTuningDataset(IReadOnlyList<FineTuningExample> examples)
+    {
+        Guard.NotNull(examples);
+        Examples = examples;
+        MeanReward = examples.Count == 0 ? 0.0 : examples.Average(e => e.Reward);
+    }
+
+    /// <summary>Gets the fine-tuning examples.</summary>
+    public IReadOnlyList<FineTuningExample> Examples { get; }
+
+    /// <summary>Gets the number of examples.</summary>
+    public int Count => Examples.Count;
+
+    /// <summary>Gets the mean reward across the examples (0 when empty).</summary>
+    public double MeanReward { get; }
+}

src/Agentic/SelfImproving/FineTuningExample.cs

@@ -0,0 +1,38 @@
+namespace AiDotNet.Agentic.SelfImproving;
+
+/// <summary>
+/// One supervised fine-tuning example distilled from a high-reward trajectory: the prompt the agent saw and
+/// the completion it produced that earned a good score.
+/// </summary>
+/// <remarks>
+/// <para><b>For Beginners:</b> A "do it like this" training pair — the question plus a known-good answer —
+/// harvested from a run that scored well. Many of these teach a local model to imitate its own best behavior.
+/// </para>
+/// </remarks>
+public sealed class FineTuningExample
+{
+    /// <summary>
+    /// Initializes a new example.
+    /// </summary>
+    /// <param name="prompt">The input/context the agent was given.</param>
+    /// <param name="completion">The (good) completion to learn.</param>
+    /// <param name="reward">The reward the originating trajectory earned.</param>
+    /// <exception cref="ArgumentNullException">Thrown when <paramref name="prompt"/> or <paramref name="completion"/> is <c>null</c>.</exception>
+    public FineTuningExample(string prompt, string completion, double reward)
+    {
+        Guard.NotNull(prompt);
+        Guard.NotNull(completion);
+        Prompt = prompt;
+        Completion = completion;
+        Reward = reward;
+    }
+
+    /// <summary>Gets the input/context the agent was given.</summary>
+    public string Prompt { get; }
+
+    /// <summary>Gets the completion to learn.</summary>
+    public string Completion { get; }
+
+    /// <summary>Gets the reward the originating trajectory earned.</summary>
+    public double Reward { get; }
+}

src/Agentic/SelfImproving/ITrajectoryEvaluator.cs

@@ -0,0 +1,28 @@
+namespace AiDotNet.Agentic.SelfImproving;
+
+/// <summary>
+/// Scores a captured <see cref="AgentTrajectory"/>, producing the reward signal the self-improving layer
+/// optimizes against (higher is better). This is the seam between "how good was that run?" and every learning
+/// mechanism that consumes the answer.
+/// </summary>
+/// <remarks>
+/// <para>
+/// Evaluators range from simple (exact-match against a known answer, length/cost penalties) to sophisticated
+/// (an LLM-as-judge, or an adapter over the reasoning reward models). Keeping the contract trajectory-native
+/// means routing/prompt-optimization/fine-tuning all share one definition of quality and can be swapped
+/// without touching the learners.
+/// </para>
+/// <para><b>For Beginners:</b> A grader. You hand it one recorded run and it returns a score saying how good
+/// the outcome was. Collect scores across many runs and the system can tell which behaviors to reinforce.
+/// </para>
+/// </remarks>
+public interface ITrajectoryEvaluator
+{
+    /// <summary>
+    /// Scores a trajectory. Higher is better; the scale is the evaluator's own (commonly 0–1).
+    /// </summary>
+    /// <param name="trajectory">The trajectory to score.</param>
+    /// <param name="cancellationToken">Token used to cancel the evaluation.</param>
+    /// <returns>The reward score.</returns>
+    Task<double> EvaluateAsync(AgentTrajectory trajectory, CancellationToken cancellationToken = default);
+}

src/Agentic/SelfImproving/ITrajectoryStore.cs

@@ -0,0 +1,49 @@
+namespace AiDotNet.Agentic.SelfImproving;
+
+/// <summary>
+/// Stores captured <see cref="AgentTrajectory"/> records so the self-improving layer can replay, evaluate,
+/// and learn from past agent runs.
+/// </summary>
+/// <remarks>
+/// <para><b>For Beginners:</b> This is the logbook of everything the agents have done. Other parts of the
+/// system read from it to grade past runs and figure out how to do better next time.
+/// </para>
+/// </remarks>
+public interface ITrajectoryStore
+{
+    /// <summary>
+    /// Adds a trajectory and returns its id.
+    /// </summary>
+    /// <param name="trajectory">The trajectory to store.</param>
+    /// <param name="cancellationToken">Token used to cancel the operation.</param>
+    /// <returns>The stored trajectory's id.</returns>
+    Task<string> AddAsync(AgentTrajectory trajectory, CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Gets a trajectory by id, or <c>null</c> when not found.
+    /// </summary>
+    /// <param name="id">The trajectory id.</param>
+    /// <param name="cancellationToken">Token used to cancel the operation.</param>
+    Task<AgentTrajectory?> GetAsync(string id, CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Returns all stored trajectories, oldest first.
+    /// </summary>
+    /// <param name="cancellationToken">Token used to cancel the operation.</param>
+    Task<IReadOnlyList<AgentTrajectory>> GetAllAsync(CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Returns the trajectories matching a predicate, oldest first.
+    /// </summary>
+    /// <param name="predicate">The filter to apply.</param>
+    /// <param name="cancellationToken">Token used to cancel the operation.</param>
+    Task<IReadOnlyList<AgentTrajectory>> QueryAsync(
+        Func<AgentTrajectory, bool> predicate,
+        CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Removes all stored trajectories.
+    /// </summary>
+    /// <param name="cancellationToken">Token used to cancel the operation.</param>
+    Task ClearAsync(CancellationToken cancellationToken = default);
+}