fix(llm): stabilize JSON-object generation (force tool_choice + native response_format + robust planner parsing) (#78)

* fix(llm): stabilize JSON-object generation

Users reported unstable JSON-object generation. Two root causes:

1. Structured output never used provider-native guarantees: the synthetic
   `emit_<schema>` tool was only *offered* (no `tool_choice`), and Strict/Json
   modes collapsed to Tool universally — so the model could emit prose or
   malformed args ("parse-and-pray").
2. The planner / pre-analysis JSON path used a naive first-`{`/last-`}` slice
   with no fence handling and no repair, hard-erroring on fenced/prosey output.

Fix (Tier 1 + Tier 2):
- LlmClient: additive `native_structured_support()`, `complete_structured()`,
  `complete_streaming_structured()` with default impls (non-breaking — existing
  clients/mocks keep working).
- structured.rs: capability-aware `resolve_mode` + `StructuredDirective`. Force
  `tool_choice` (Tool/Auto), and request native `response_format`
  (Strict→json_schema+strict, Json→json_object) on capable providers, falling
  back to forced Tool mode otherwise.
- anthropic/openai/zhipu: honor the directive (Anthropic forced tool_choice;
  OpenAI tool_choice + response_format; Zhipu delegates to its inner client).
- llm_planner: reuse the robust shared extractor + add one repair retry in
  `pre_analyze`.
- generate_object: stop pre-collapsing Strict/Json to Tool (engine resolves it).

Tests:
- Deep adversarial unit tests: capability/directive routing, provider
  wire-format (tool_choice/response_format), adversarial JSON extraction,
  planner fence/prose/brace-in-string + repair-retry. 1811 lib tests green,
  fmt + clippy clean.
- New `#[ignore]` real-LLM integration test (tests/test_structured_json_real_llm.rs).
  Validated end-to-end against gpt-4o via the real gateway in .a3s/config.acl:
  forced tool_choice 5/5 stable (0 repairs), json_object ok, pre_analyze ok.

Notes:
- Strict json_schema is opt-in only; Auto/Tool never send response_format.
- Follow-up: override complete_streaming_structured on providers so the
  streaming structured path also forces the directive (today it uses the
  non-forcing default).

* fix(llm): force structured directive on the streaming path too

Follow-up to the blocking-path fix: providers now override
`complete_streaming_structured` so streaming structured generation also forces
`tool_choice` / sets native `response_format`, instead of falling back to the
non-forcing default.

- anthropic/openai: extract `send_streaming` from `complete_streaming`; the
  trait methods become thin wrappers, and `complete_streaming_structured`
  applies the directive before executing. The large streaming parsers are
  unchanged.
- zhipu: already delegates to its inner client (no change).
- tests: RecordingClient records the streaming directive; new unit test asserts
  `generate_streaming` forces the tool; new `#[ignore]` real-LLM streaming case.
  Validated against gpt-4o: streaming forced tool_choice -> 8 partials, valid
  object (5/5 integration cases pass).

---------

Co-authored-by: Claude <claude@anthropic.com>

Author: ZhiXiao-Lin

core/src/llm/anthropic.rs

@@ -1,6 +1,7 @@
 //! Anthropic Claude LLM client
 
 use super::http::{default_http_client, normalize_base_url, HttpClient};
+use super::structured;
 use super::types::*;
 use super::LlmClient;
 use crate::retry::{AttemptOutcome, RetryConfig};
@@ -152,17 +153,24 @@ impl AnthropicClient {
     }
 }
 
-#[async_trait]
-impl LlmClient for AnthropicClient {
-    async fn complete(
-        &self,
-        messages: &[Message],
-        system: Option<&str>,
-        tools: &[ToolDefinition],
-    ) -> Result<LlmResponse> {
+impl AnthropicClient {
+    /// Apply a structured-output directive to an Anthropic request.
+    ///
+    /// Anthropic supports forced tool choice (`tool_choice`) but has no
+    /// `response_format`, so only `force_tool` is honored.
+    fn apply_directive(
+        request: &mut serde_json::Value,
+        directive: &structured::StructuredDirective,
+    ) {
+        if let Some(tool) = &directive.force_tool {
+            request["tool_choice"] = serde_json::json!({ "type": "tool", "name": tool });
+        }
+    }
+
+    /// Execute a fully-built (non-streaming) request body.
+    async fn send_request(&self, request_body: serde_json::Value) -> Result<LlmResponse> {
         {
             let request_started_at = Instant::now();
-            let request_body = self.build_request(messages, system, tools);
             let url = format!("{}/v1/messages", self.base_url);
 
             let headers = vec![
@@ -259,17 +267,70 @@ impl LlmClient for AnthropicClient {
             Ok(llm_response)
         }
     }
+}
+
+#[async_trait]
+impl LlmClient for AnthropicClient {
+    async fn complete(
+        &self,
+        messages: &[Message],
+        system: Option<&str>,
+        tools: &[ToolDefinition],
+    ) -> Result<LlmResponse> {
+        self.send_request(self.build_request(messages, system, tools))
+            .await
+    }
+
+    async fn complete_structured(
+        &self,
+        messages: &[Message],
+        system: Option<&str>,
+        tools: &[ToolDefinition],
+        directive: &structured::StructuredDirective,
+    ) -> Result<LlmResponse> {
+        let mut request_body = self.build_request(messages, system, tools);
+        Self::apply_directive(&mut request_body, directive);
+        self.send_request(request_body).await
+    }
+
+    fn native_structured_support(&self) -> structured::NativeStructuredSupport {
+        structured::NativeStructuredSupport::ForcedTool
+    }
 
     async fn complete_streaming(
         &self,
         messages: &[Message],
         system: Option<&str>,
         tools: &[ToolDefinition],
         cancel_token: CancellationToken,
+    ) -> Result<mpsc::Receiver<StreamEvent>> {
+        self.send_streaming(self.build_request(messages, system, tools), cancel_token)
+            .await
+    }
+
+    async fn complete_streaming_structured(
+        &self,
+        messages: &[Message],
+        system: Option<&str>,
+        tools: &[ToolDefinition],
+        directive: &structured::StructuredDirective,
+        cancel_token: CancellationToken,
+    ) -> Result<mpsc::Receiver<StreamEvent>> {
+        let mut request_body = self.build_request(messages, system, tools);
+        Self::apply_directive(&mut request_body, directive);
+        self.send_streaming(request_body, cancel_token).await
+    }
+}
+
+impl AnthropicClient {
+    /// Execute a fully-built streaming request body (sets `stream: true`).
+    async fn send_streaming(
+        &self,
+        mut request_body: serde_json::Value,
+        cancel_token: CancellationToken,
     ) -> Result<mpsc::Receiver<StreamEvent>> {
         {
             let request_started_at = Instant::now();
-            let mut request_body = self.build_request(messages, system, tools);
             request_body["stream"] = serde_json::json!(true);
 
             let url = format!("{}/v1/messages", self.base_url);
@@ -739,4 +800,40 @@ mod tests {
         assert_eq!(req["max_tokens"], 16_000);
         assert_eq!(req["thinking"]["budget_tokens"], 8_000);
     }
+
+    #[test]
+    fn test_apply_directive_forces_tool_choice() {
+        let mut req = serde_json::json!({ "model": "m", "messages": [] });
+        let directive = structured::StructuredDirective {
+            force_tool: Some("emit_person".to_string()),
+            response_format: None,
+        };
+        AnthropicClient::apply_directive(&mut req, &directive);
+        assert_eq!(req["tool_choice"]["type"], "tool");
+        assert_eq!(req["tool_choice"]["name"], "emit_person");
+    }
+
+    #[test]
+    fn test_apply_directive_ignores_response_format() {
+        // Anthropic has no response_format; both a response_format-only and an
+        // empty directive must be no-ops.
+        let mut req = serde_json::json!({ "model": "m" });
+        AnthropicClient::apply_directive(
+            &mut req,
+            &structured::StructuredDirective {
+                force_tool: None,
+                response_format: Some(structured::ResponseFormat::JsonObject),
+            },
+        );
+        assert!(req.get("response_format").is_none());
+        assert!(req.get("tool_choice").is_none());
+    }
+
+    #[test]
+    fn test_native_structured_support_is_forced_tool() {
+        assert_eq!(
+            make_client().native_structured_support(),
+            structured::NativeStructuredSupport::ForcedTool
+        );
+    }
 }

core/src/llm/mod.rs

@@ -48,6 +48,44 @@ pub trait LlmClient: Send + Sync {
         tools: &[ToolDefinition],
         cancel_token: CancellationToken,
     ) -> Result<mpsc::Receiver<StreamEvent>>;
+
+    /// Report the strongest provider-native structured-output enforcement this
+    /// client supports. Used by [`structured`] to decide whether to force a
+    /// tool call, request a native `response_format`, or fall back to
+    /// prompt-and-parse. Defaults to no native support.
+    fn native_structured_support(&self) -> structured::NativeStructuredSupport {
+        structured::NativeStructuredSupport::None
+    }
+
+    /// Complete a conversation while honoring a structured-output directive
+    /// (forced `tool_choice` and/or native `response_format`).
+    ///
+    /// The default implementation ignores the directive and behaves exactly
+    /// like [`LlmClient::complete`], so existing clients keep working unchanged;
+    /// providers that support native structured output override this.
+    async fn complete_structured(
+        &self,
+        messages: &[Message],
+        system: Option<&str>,
+        tools: &[ToolDefinition],
+        _directive: &structured::StructuredDirective,
+    ) -> Result<LlmResponse> {
+        self.complete(messages, system, tools).await
+    }
+
+    /// Streaming counterpart of [`LlmClient::complete_structured`]. Defaults to
+    /// [`LlmClient::complete_streaming`], ignoring the directive.
+    async fn complete_streaming_structured(
+        &self,
+        messages: &[Message],
+        system: Option<&str>,
+        tools: &[ToolDefinition],
+        _directive: &structured::StructuredDirective,
+        cancel_token: CancellationToken,
+    ) -> Result<mpsc::Receiver<StreamEvent>> {
+        self.complete_streaming(messages, system, tools, cancel_token)
+            .await
+    }
 }
 
 // Include test modules — these reference internal types via crate paths