Skip to content

fix(agentscope): start streaming LLM spans before model calls#242

Merged
sipercai merged 1 commit into
mainfrom
liuyu/fix/agentscope-v2-llm-sdk-dedup
Jul 9, 2026
Merged

fix(agentscope): start streaming LLM spans before model calls#242
sipercai merged 1 commit into
mainfrom
liuyu/fix/agentscope-v2-llm-sdk-dedup

Conversation

@sipercai

@sipercai sipercai commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR fixes AgentScope v2 streaming model-call telemetry so the framework LLM span starts before the underlying model handler is invoked. This keeps the util-genai context active during the provider call, which prevents duplicate provider LLM spans in full-agent installations while preserving TTFT, token usage, stream finalization, and captured input/output messages on the framework span.

Fixes # (N/A)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • tox -c tox-loongsuite.ini -e py312-test-loongsuite-instrumentation-agentscope-latest
  • tox -c tox-loongsuite.ini -e py312-test-loongsuite-instrumentation-agentscope-oldest
  • tox -c tox-loongsuite.ini -e lint-loongsuite-instrumentation-agentscope
  • PRE_COMMIT_HOME="$(mktemp -d)" tox -e precommit
  • git diff --check
  • Weaver live-check on a generated AgentScope v2 streaming JSON span sample, using the loongsuite-genai advice profile
  • Downstream full-agent live smoke for AgentScope v2 streaming confirmed one framework LLM span and one provider HTTP span, with no duplicate provider LLM span

Validation Evidence

Spec and Scope

  • Linked issue/spec: N/A. Fixes the observed duplicate AgentScope v2 streaming LLM telemetry when framework and provider instrumentation are both active.
  • Approved spec/comment: direct maintainer request in this thread.
  • Changed surface: loongsuite-instrumentation-agentscope v2 middleware, tests, and package changelog.

Local Checks

Check Command Result Notes
Static readiness LoongSuite pipeline static PR readiness checker pass No readiness blockers reported.
Change detector Pull-request event simulation for changed files pass `packages=
Precommit PRE_COMMIT_HOME="$(mktemp -d)" tox -e precommit pass Fresh pre-commit hook cache.
Focused tests tox -c tox-loongsuite.ini -e py312-test-loongsuite-instrumentation-agentscope-latest pass 15 passed. Covers AgentScope v2 streaming span ordering and content capture.
Focused tests tox -c tox-loongsuite.ini -e py312-test-loongsuite-instrumentation-agentscope-oldest pass 94 passed. Maintains older AgentScope compatibility.
Focused lint tox -c tox-loongsuite.ini -e lint-loongsuite-instrumentation-agentscope pass Ruff check passed.
External review Codex/Qoder review loop pass Deep review found no blocking findings; advisory generator-lifecycle risks are deferred as non-blocking.
Privacy scan Branch diff scan for credentials, private trace IDs, local paths, and internal artifacts pass No sensitive content found in the PR diff.

Real E2E Matrix

Scenario Status Command or Demo Evidence
non-streaming pass py312-test-loongsuite-instrumentation-agentscope-latest AgentScope v2 non-streaming E2E test passed.
streaming pass py312-test-loongsuite-instrumentation-agentscope-latest; downstream full-agent live smoke Streaming E2E passed; live smoke confirmed one framework LLM span plus provider HTTP span, without duplicate provider LLM span.
concurrency pass py312-test-loongsuite-instrumentation-agentscope-latest Concurrent AgentScope v2 E2E test produced isolated agent and LLM spans.
agent/tool/ReAct pass py312-test-loongsuite-instrumentation-agentscope-latest Tool and ReAct telemetry tests passed.
tool-heavy pass py312-test-loongsuite-instrumentation-agentscope-latest Multi-tool ReAct telemetry test passed.
error path pass py312-test-loongsuite-instrumentation-agentscope-latest Non-streaming and streaming model error-path tests passed.

Telemetry and Weaver

Check Status Command or Artifact Notes
Span tree / span kinds pass Focused AgentScope v2 tests and downstream full-agent live smoke Streaming path now emits a single framework LLM span under the agent span and a provider HTTP child span.
Content capture modes pass py312-test-loongsuite-instrumentation-agentscope-latest New streaming test verifies gen_ai.input.messages and gen_ai.output.messages in SPAN_ONLY mode; NO_CONTENT behavior is unchanged by this diff.
Concurrency isolation pass py312-test-loongsuite-instrumentation-agentscope-latest Concurrent AgentScope v2 E2E test passed.
Weaver live-check pass weaver registry live-check -r <loongsuite-semantic-conventions>/model --input-source <generated AgentScope v2 streaming JSON sample> --input-format json --advice-profile loongsuite-genai --skip-policies true Exit 0, no violations. Only expected development-stability advice was reported.

CI

  • GitHub checks: not started yet; this PR has not been opened at the time of local validation.
  • Known unrelated failures: none known.
  • Follow-up needed: inspect GitHub checks after the PR is opened.

Does This PR Require a Core Repo Change?

  • Yes. - Link to PR:
  • No.

Checklist:

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

@ralf0131 ralf0131 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR fixes two issues in the AgentScope v2 instrumentation:

  1. Streaming LLM span lifecycle — The span is now always started before the model handler runs (via handler.start_llm), instead of deferring it to first-token time for streaming models. This ensures TTFT and the full stream lifecycle are captured on the framework LLM span. The started flag and all its conditional branches are correctly removed, simplifying the control flow.

  2. String content guard_blocks_to_parts now handles None (returns []) and bare str (wraps in Text), so LLM input/output message attributes are populated when AgentScope passes string content instead of block objects.

Tests are solid: test_v2_streaming_model_call_starts_llm_span_before_model_handler verifies the span is active during the model handler, and test_v2_streaming_model_call_captures_input_and_output_content verifies content capture end-to-end.

LGTM — clean simplification, good test coverage, CI green.


Automated review by github-manager-bot

@sipercai sipercai merged commit be4f7d3 into main Jul 9, 2026
22 of 27 checks passed
@sipercai sipercai deleted the liuyu/fix/agentscope-v2-llm-sdk-dedup branch July 9, 2026 06:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants