Skip to content

feat(s3): add short access links and SDK core#7281

Merged
c121914yu merged 25 commits into
labring:mainfrom
xqvvu:xqvvu/s3-shortlink-sdk-wrapper
Jul 16, 2026
Merged

feat(s3): add short access links and SDK core#7281
c121914yu merged 25 commits into
labring:mainfrom
xqvvu:xqvvu/s3-shortlink-sdk-wrapper

Conversation

@xqvvu

@xqvvu xqvvu commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add short S3 download alias URLs (/api/system/file/d/<alias>.<exp>.<sig>) and short upload session URLs (/api/system/file/u/<token>).
  • Extract the access-link protocol into @fastgpt-sdk/storage/access-link so other trusted services can reuse HMAC signing, TTL bucketing, alias reuse, upload token hashing, verification, and revoke logic.
  • Keep FastGPT runtime concerns in service adapters: Mongo stores, route builders, proxy handlers, and error mapping.
  • Preserve legacy JWT download/upload route compatibility for already-issued links while new signing paths issue short access links.
  • Update dataset/search/quote preview URL generation to use async shortlink signing from the previous shortlink implementation.

Tests

  • pnpm exec vitest run -c vitest.config.mts test/sdk/storage/accessLink.test.ts
  • pnpm --filter @fastgpt-sdk/storage build
  • pnpm --filter @fastgpt/app typecheck
  • From packages/service: pnpm exec vitest run -c vitest.config.ts test/common/s3/accessLink.test.ts test/common/s3/token.test.ts test/core/dataset/utils.test.ts test/core/dataset/data/controller.test.ts
  • From projects/app: pnpm exec vitest run -c vitest.config.ts test/api/system/file/accessLink.test.ts test/api/system/file/sourceContentType.test.ts
  • git diff --check

Notes

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 18.42% 1632 / 8859
🔵 Statements 18.41% 1713 / 9304
🔵 Functions 16.2% 342 / 2110
🔵 Branches 15.14% 761 / 5026
File CoverageNo changed files found.
Generated in workflow #1482 for commit 9901bbe by the Vitest Coverage Report Action

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

Admin Preview Image Ready!

ghcr.io/labring/fastgpt-pr:admin_9901bbec9ffc644b454768f2034e550fcecc969c

🕒 Time: 2026-07-16 19:04:59 (UTC+8)

@xqvvu xqvvu force-pushed the xqvvu/s3-shortlink-sdk-wrapper branch from 5bfe7be to 1518715 Compare July 8, 2026 06:37
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

Build Successful - Preview fastgpt Image for this PR:

ghcr.io/labring/fastgpt-pr:fastgpt_9901bbec9ffc644b454768f2034e550fcecc969c

🕒 Time: 2026-07-16 18:57:41 (UTC+8)

@xqvvu xqvvu force-pushed the xqvvu/s3-shortlink-sdk-wrapper branch 5 times, most recently from e597ef1 to 93ec562 Compare July 15, 2026 06:47
@xqvvu xqvvu force-pushed the xqvvu/s3-shortlink-sdk-wrapper branch from e08c553 to 3f36136 Compare July 16, 2026 08:00
@xqvvu xqvvu force-pushed the xqvvu/s3-shortlink-sdk-wrapper branch from 210408f to c849f1e Compare July 16, 2026 09:40

@c121914yu c121914yu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

结论:当前不建议合并。核心短链协议的 SDK/runtime 分层方向合理,但最新 head c849f1e 存在确定性构建阻塞、配置回归和下载语义问题。

必须修改

  1. FastGPT 镜像无法构建。 presignDraftChatFilePostUrl.ts、新旧 system file route 共 5 个文件引用了最新 main 已删除的 @fastgpt/service/type/next,应改用 @fastgpt/next/type。当前 Docker build 已在 presignDraftChatFilePostUrl.ts:1 失败。
  2. Pro 未适配共享契约。 pro/admin/src/service/core/ai/auxiliaryGeneration/chatAgentHelper/service.ts:316 调用 persistChatFiles 时缺少 sourceType/sourceId/chatId,Admin 镜像构建失败。
  3. Pro tests 无法解析 SDK subpath。 pro/admin/vitest.config.ts:8@fastgpt-sdk/storage 映射到 index.ts,导致 @fastgpt-sdk/storage/access-link 被拼成 index.ts/access-link 并报 ENOTDIR。请补精确 alias 或统一从 package root 导入。
  4. 默认环境模板不可用。 projects/app/.env.template:92 在 MinIO 且 external/CDN endpoint 为空时默认启用 short-redirect,下载必然 500;:111-112 又把 Mongo URI 错误断成两行。建议与 service 默认统一为 short-proxy 并恢复完整 URI。
  5. 短链过期和撤销会被缓存绕过。 projects/app/src/service/common/s3/proxy.ts:314 返回 public, max-age=31536000。浏览器或共享缓存命中后不会重新校验 expiry、disabledAt 或对象删除状态。私有短链至少需要 no-store,或将缓存上限约束到剩余 TTL 并明确撤销语义。
  6. 下载 mode API 不兼容且语义未真正实现。 OpenAPI 将旧 proxy 删除,已有客户端会得到 400;同时 short-proxyshort-redirect 生成相同 URL,最终传输方式只读取全局 env,单次调用传入的 mode 不能强制对应行为。请保留旧值映射,并明确 mode 是签发模式还是传输模式。
  7. short-redirect 的 HEAD 会失败。 route 接受 HEAD,但 proxy.ts:403 始终生成 GET presigned URL;S3 签名包含 HTTP method,客户端跟随 HEAD 302 后会验签失败。应为 HEAD 返回 metadata,或生成对应的 HEAD 签名。
  8. 真实存储 404 会落成 500。 当前错误映射只识别 CommonErrEnum.fileNotFound,AWS/OSS 的 Head/Get 异常不会命中。现有测试用 undefined/null mock,未覆盖真实 adapter 错误。

范围与性能

PR 自身的 .agents/design/common/s3/pr-split-plan.md 明确要求把短链、上传校验、ChatBox abort 拆成 3 个 PR,但当前三者以及 Sandbox/Invoke 文件生命周期全部混入。后者已经修改共享契约并导致 Pro 构建失败,建议按文档拆分后再审。

批量 alias API 已实现,但 collection/export.ts:114 仍最多对 50,000 行逐条处理 q/a,data/v2/list.ts:61 也逐字段、逐图片签发。请按 export chunk/page 汇总 key 后调用 createS3DownloadAccessUrls

另外请删除 dispatchChatCompletion.ts:75console.dir,并合并重复的 downloadAlias/entity.ts 与实际 store adapter。

验证

git diff --check、global/service/app unit tests 通过;FastGPT-Pro-Test、Preview Admin Image、Preview FastGPT Image 三项 CI 失败。补齐后还需要覆盖 redirect HEAD、真实 adapter 404、缓存后的过期/撤销、旧 mode: proxy 兼容和大批量导出。

@c121914yu c121914yu merged commit 81d7b93 into labring:main Jul 16, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants