Skip to content

feat: support Vertex Lyria 3 interactions relay()#4934

Open
SwillerDawn wants to merge 1 commit into
QuantumNous:mainfrom
NaciTech:feat/lyria003-adapter
Open

feat: support Vertex Lyria 3 interactions relay()#4934
SwillerDawn wants to merge 1 commit into
QuantumNous:mainfrom
NaciTech:feat/lyria003-adapter

Conversation

@SwillerDawn
Copy link
Copy Markdown

@SwillerDawn SwillerDawn commented May 18, 2026

⚠️ 提交说明 / PR Notice

Important

  • 请提供人工撰写的简洁摘要,避免直接粘贴未经整理的 AI 输出。
    支持 lyria-3-pro-preview 与 lyria-3-clip-preview 通过Vertex 原生路径 /v1beta/models/{model}:interactions 调用。

📝 变更描述 / Description

(简述:做了什么?为什么这样改能生效?请基于你对代码逻辑的理解来写,避免粘贴未经整理的内容)

🚀 变更类型 / Type of change

  • 🐛 Bug 修复 (Bug fix) - 请关联对应 Issue,避免将设计取舍、理解偏差或预期不一致直接归类为 bug
  • [x ] ✨ 新功能 (New feature) - 重大特性建议先通过 Issue 沟通
  • ⚡ 性能优化 / 重构 (Refactor)
  • 📝 文档更新 (Documentation)

🔗 关联任务 / Related Issue

  • Closes # (如有)

✅ 提交前检查项 / Checklist

  • [x ] 人工确认: 我已亲自整理并撰写此描述,没有直接粘贴未经处理的 AI 输出。
  • [x ] 非重复提交: 我已搜索现有的 IssuesPRs,确认不是重复提交。
  • Bug fix 说明: 若此 PR 标记为 Bug fix,我已提交或关联对应 Issue,且不会将设计取舍、预期不一致或理解偏差直接归类为 bug。
  • [x ] 变更理解: 我已理解这些更改的工作原理及可能影响。
  • [x ] 范围聚焦: 本 PR 未包含任何与当前任务无关的代码改动。
  • [x ] 本地验证: 已在本地运行并通过测试或手动验证,维护者可以据此复核结果。
  • [x ] 安全合规: 代码中无敏感凭据,且符合项目代码规范。

📸 运行证明 / Proof of Work

(请在此粘贴截图、关键日志或测试报告,以证明变更生效)
image

curl http://localhost:3000/v1beta/models/lyria-3-pro-preview:interactions \
  --request POST \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "lyria-3-pro-preview",
    "input": [
      {
        "type": "text",
        "text": "Compose a full song with intro, build-up, chorus, and outro. Uplifting, modern pop style."
      },
      {
        "type": "image",
        "mime_type": "image/png",
        "uri": "https://ark-project.tos-cn-beijing.volces.com/doc_image/r2v_tea_pic1.jpg"
      }
    ]
  }'

Summary by CodeRabbit

  • New Features

    • End-to-end support for native Gemini/Vertex interaction requests with passthrough of upstream payloads
    • Added two preview model variants with default pricing: lyria-3-pro-preview and lyria-3-clip-preview
  • Behavior & Validation

    • Interactions endpoints route to a dedicated flow with stricter request validation and clearer error responses
  • Usage

    • Improved token billing estimation for interaction payloads; streaming disabled for native interactions

Review Change Stack

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 18, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 047377a3-6e24-4bdc-935e-d19d54cd0191

📥 Commits

Reviewing files that changed from the base of the PR and between b577512 and 425d605.

📒 Files selected for processing (5)
  • dto/gemini.go
  • relay/channel/vertex/adaptor.go
  • relay/gemini_handler.go
  • relay/helper/valid_request.go
  • setting/ratio_setting/model_ratio.go
🚧 Files skipped from review as they are similar to previous changes (4)
  • relay/helper/valid_request.go
  • relay/gemini_handler.go
  • dto/gemini.go
  • relay/channel/vertex/adaptor.go

Walkthrough

Adds support for native Vertex/Gemini interaction payloads: a new GeminiInteractionRequest DTO with custom JSON handling and token estimation, request validation for interactions payloads, a dedicated interaction handler, adaptor URL/response routing for Vertex interactions, and pricing defaults for new lyria-3-* preview models.

Changes

Gemini Interaction Flow

Layer / File(s) Summary
GeminiInteractionRequest DTO and marshalling
dto/gemini.go
GeminiInteractionRequest preserves native Vertex payloads with custom UnmarshalJSON/MarshalJSON that extracts/reinjects model, provides token-count estimation from recursive input text extraction, implements IsStream (always false), and SetModelName updates mapped and native model fields.
Interaction request validation
relay/helper/valid_request.go
Router detects :interactions path and calls GetAndValidateGeminiInteractionRequest, which unmarshals into GeminiInteractionRequest, requires payload and payload["input"], and enforces input is a non-empty array.
Gemini interaction handler orchestration
relay/gemini_handler.go
GeminiHelper detects *dto.GeminiInteractionRequest and routes to GeminiInteractionHelper, which validates lyria-3-* model prefixes, applies model mapping/param overrides, initializes the adaptor by ApiType, marshals and sends the interaction, maps errors/status, processes adaptor response, and consumes quota.
Vertex adaptor interactions endpoint and response handling
relay/channel/vertex/adaptor.go
GetRequestURL builds :interactions endpoint for Lyria models (API-key or service-account ProjectID); DoResponse routes interactions to doInteractionResponse; isLyriaInteractionRequest classifies interactions; doInteractionResponse preserves upstream response bytes, validates 2xx status, sets Content-Type (default application/json for Lyria models), writes raw bytes to client, and returns minimal usage or an error for unsupported models.
Pricing defaults for Lyria models
setting/ratio_setting/model_ratio.go
Adds lyria-3-pro-preview and lyria-3-clip-preview entries to defaultModelPrice.

Sequence Diagram

sequenceDiagram
  participant Client
  participant RequestValidator
  participant GeminiHandler
  participant Adaptor
  participant VertexAPI
  participant QuotaService
  Client->>RequestValidator: POST interactions with native payload
  RequestValidator->>RequestValidator: Detect interactions path and unmarshal payload
  RequestValidator-->>GeminiHandler: Validated GeminiInteractionRequest
  GeminiHandler->>GeminiHandler: Validate lyria-3 model prefix and apply mapping
  GeminiHandler->>Adaptor: Initialize by ApiType and DoRequest
  Adaptor->>VertexAPI: POST interactions endpoint
  VertexAPI-->>Adaptor: 2xx and native response bytes
  Adaptor->>Adaptor: Validate status and read raw bytes
  Adaptor-->>Client: Write raw bytes to response
  Adaptor-->>GeminiHandler: Return usage
  GeminiHandler->>QuotaService: Consume usage
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~40 minutes

Possibly related PRs

  • QuantumNous/new-api#2222: Adds default pricing entries for lyria-3-*-preview models related to the pricing map extended here.

Poem

🐰 In tunnels of bytes the interactions spring,

I hug the payloads tight and guard each string.
Models mapped and JSON kept intact,
Responses routed home — no streaming fact.
Hooray, the relay hums — hop, hop, click-clack!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'feat: support Vertex Lyria 3 interactions relay' accurately summarizes the main change: adding support for Vertex's native interactions endpoint for Lyria 3 models. It clearly identifies the primary feature being added.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@dto/gemini.go`:
- Around line 27-30: Change the GeminiInteractionRequest.Model field from string
to *string (keeping `json:"model,omitempty"`) so omitted vs explicit-empty is
preserved when parsing and re-marshaling; update any constructors, accessors and
JSON helpers that reference GeminiInteractionRequest.Model (and the similar
optional scalar fields noted around lines 73-82) to handle a nil pointer (e.g.,
use string dereference checks, helper getters that return empty string when nil,
and when setting the field allocate a new string), and ensure any relay/convert
code that builds or forwards the DTO treats Model as optional by checking for
nil before including it.

In `@relay/channel/vertex/adaptor.go`:
- Around line 178-184: The service-account branch currently returns a base
interactions URL without the model path; change the returned URL in adaptor.go
so it appends the model-specific endpoint "models/{model}:interactions" instead
of just "/interactions". Use the decoded credentials (adc or
a.AccountCredentials) to fetch the model identifier (e.g., adc.Model or
a.AccountCredentials.Model) and return fmt.Sprintf("%s/models/%s:interactions",
BuildAPIBaseURL(info.ChannelBaseUrl, OpenSourceAPIVersion, adc.ProjectID,
"global"), <model>) so Vertex calls hit the correct models/{model}:interactions
endpoint.

In `@relay/gemini_handler.go`:
- Around line 244-256: The code currently type-asserts resp with
resp.(*http.Response) without guarding for nil or wrong type, which can panic;
update the logic around statusCodeMappingStr/httpResp and the call to
adaptor.DoResponse: first check that resp is non-nil and of type *http.Response
(e.g., via httpResp, ok := resp.(*http.Response)); if not, construct and return
a structured relay error via service.RelayErrorHandler (using
c.Request.Context()) and call service.ResetStatusCode before returning; only
call adaptor.DoResponse(httpResp, info) when the assert succeeded and httpResp
is non-nil.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f14591b5-1221-435f-87fa-5a7d34a09ea2

📥 Commits

Reviewing files that changed from the base of the PR and between 5dd0d3b and 700a8a2.

📒 Files selected for processing (5)
  • dto/gemini.go
  • relay/channel/vertex/adaptor.go
  • relay/gemini_handler.go
  • relay/helper/valid_request.go
  • setting/ratio_setting/model_ratio.go

Comment thread dto/gemini.go
Comment thread relay/channel/vertex/adaptor.go
Comment thread relay/gemini_handler.go
@SwillerDawn SwillerDawn force-pushed the feat/lyria003-adapter branch 2 times, most recently from b577512 to 37492d1 Compare May 18, 2026 06:54
@SwillerDawn SwillerDawn deleted the feat/lyria003-adapter branch May 18, 2026 06:55
@SwillerDawn SwillerDawn restored the feat/lyria003-adapter branch May 18, 2026 07:09
@SwillerDawn SwillerDawn reopened this May 18, 2026
@SwillerDawn SwillerDawn force-pushed the feat/lyria003-adapter branch from 37492d1 to 425d605 Compare May 18, 2026 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant