对备份功能进行进一步修复#3845
Conversation
- Add backup module with Storage interface - Implement LocalStorage, S3Storage, and WebDAVStorage - Add backup scheduler for automatic scheduled backups - Add Management API endpoints for backup operations - Update config structure with BackupConfig - Add backup configuration to config.example.yaml Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Fix variable naming conflict (err vs errCreate) - Rename logDirectory to getLogDirectory to avoid conflict - Remove dependency on logging.DefaultLogDir Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Backend changes: - Add Manager.Restore() method to extract files from backup zip - Add Manager.extractFile() to handle individual file extraction - Add RestoreBackup API handler (POST /v0/management/backup/restore) - Support multipart/form-data file upload - Extract config.yaml and auths/* from backup - Skip logs during restore to preserve current logs - Register /backup/restore route in server.go File extraction logic: - auths/* files -> restore to auth directory - config.yaml -> restore to config file path - logs/* files -> skip (don't overwrite current logs) - Unknown files -> skip with warning Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Critical fixes: 1. S3 endpoint protocol cleaning - Remove https:// and http:// prefixes automatically - Remove trailing slashes and path components - Minio client requires bare hostname only 2. Multi-storage backend support - Parse comma-separated storage types (local,s3,webdav) - Try each storage backend in order - Return first successfully created backend - Prevents failures when one backend is misconfigured 3. Better error handling - Storage creation errors no longer block other backends - Last error is returned if all backends fail Tested with: - Local storage: ✅ Create, List, Restore working - S3 storage: ✅ Endpoint cleaning works (auth issue separate) - Multi-storage: ✅ Fallback to first available Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Major improvements: 1. Chinese localization - All API error messages now in Chinese - Test connection: '连接成功' instead of 'connection successful' - All error/success messages translated 2. Multi-storage upload support - Parse storage as 'local,s3,webdav' - Upload to ALL configured storage backends - Report success count in message - Continue on individual storage failures 3. New functions - createBackupManagerForType(): create manager for specific type - createBackupStorageByType(): create storage for specific type - Improved error handling with lastErr tracking 4. Better user feedback - '备份创建成功,已上传到 N 个存储后端' - Logs which backend succeeded/failed - Continues uploading even if one backend fails All messages now display correctly in Chinese UI! Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Cli-Proxy-API-Management-Center is a separate repository and should not be tracked in the backend repository. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Remove timeout violations (S3/WebDAV operations) - Fix multi-storage backend logic (List/Delete/Download) - Improve scheduler concurrency safety - Execute backup immediately on scheduler start - Add comprehensive config validation - Fix UpdateSchedule race condition
Critical security fixes: - Fix Zip Slip vulnerability (directory traversal attack prevention) - Validate file paths and check directory boundaries during restore High priority fixes: - Fix log file filtering (filter files first, then take last 10) - Track critical file restoration failures - Return error if config.yaml fails to restore Medium priority fixes: - Optimize backup creation (create ZIP once, upload to all storages) - Fix double JSON response in PutBackupConfig - Test all storage backends in TestBackupConnection - Add time import for BackupInfo timestamp Improvements: - Better error messages for restoration failures - Individual storage backend test results - Reduced redundant ZIP creation Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixed compilation errors: - Removed duplicate code block (lines 145-155) - Removed extra closing braces causing syntax errors - Code now compiles successfully Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements a comprehensive backup and restore system supporting local, S3, and WebDAV storage backends, complete with automated scheduling and management API endpoints. While the implementation is feature-rich, the code review highlights critical security and concurrency concerns. These include a Zip Slip path traversal vulnerability during backup restoration, multiple data races in the API handlers and scheduler due to improper mutex locking, and fragile XML parsing in the WebDAV storage backend.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if strings.HasPrefix(file.Name, "auths/") { | ||
| absAuthDir, err := filepath.Abs(m.authDir) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to resolve auth directory: %w", err) | ||
| } | ||
| if !strings.HasPrefix(absTarget, absAuthDir) { | ||
| return fmt.Errorf("attempted to extract file outside auth directory: %s", file.Name) | ||
| } | ||
| } |
There was a problem hiding this comment.
安全漏洞 (Zip Slip / 路径遍历绕过)：使用 strings.HasPrefix(absTarget, absAuthDir) 来验证解压目标路径是否在授权目录内存在安全漏洞。如果 absAuthDir 为 /app/auths，而恶意构造的 absTarget 为 /app/auths-secret/file，strings.HasPrefix 仍然会返回 true，从而绕过安全检查，允许将文件写入同级的敏感目录。\n\n建议使用 filepath.Rel 来安全地计算相对路径，并检查其是否以 .. 开头，或者确保在比较前为 absAuthDir 加上路径分隔符后缀。
| if strings.HasPrefix(file.Name, "auths/") { | |
| absAuthDir, err := filepath.Abs(m.authDir) | |
| if err != nil { | |
| return fmt.Errorf("failed to resolve auth directory: %w", err) | |
| } | |
| if !strings.HasPrefix(absTarget, absAuthDir) { | |
| return fmt.Errorf("attempted to extract file outside auth directory: %s", file.Name) | |
| } | |
| } | |
| if strings.HasPrefix(file.Name, "auths/") { | |
| absAuthDir, err := filepath.Abs(m.authDir) | |
| if err != nil { | |
| return fmt.Errorf("failed to resolve auth directory: %w", err) | |
| } | |
| rel, err := filepath.Rel(absAuthDir, absTarget) | |
| if err != nil || strings.HasPrefix(rel, "..") { | |
| return fmt.Errorf("attempted to extract file outside auth directory: %s", file.Name) | |
| } | |
| } |
| h.mu.Lock() | ||
| h.cfg.Backup.Enabled = newConfig.Enabled | ||
| h.cfg.Backup.Schedule = strings.TrimSpace(newConfig.Schedule) | ||
| h.cfg.Backup.Storage = strings.TrimSpace(string(newConfig.Storage)) | ||
| h.cfg.Backup.LocalDir = strings.TrimSpace(newConfig.LocalDir) | ||
| h.cfg.Backup.MaxBackups = newConfig.MaxBackups | ||
|
|
||
| // Update S3 config | ||
| h.cfg.Backup.S3.Endpoint = strings.TrimSpace(newConfig.S3.Endpoint) | ||
| h.cfg.Backup.S3.Region = strings.TrimSpace(newConfig.S3.Region) | ||
| h.cfg.Backup.S3.Bucket = strings.TrimSpace(newConfig.S3.Bucket) | ||
| h.cfg.Backup.S3.Path = strings.TrimSpace(newConfig.S3.Path) | ||
| h.cfg.Backup.S3.AccessKey = strings.TrimSpace(newConfig.S3.AccessKey) | ||
| h.cfg.Backup.S3.SecretKey = strings.TrimSpace(newConfig.S3.SecretKey) | ||
| h.cfg.Backup.S3.UseSSL = newConfig.S3.UseSSL | ||
|
|
||
| // Update WebDAV config | ||
| h.cfg.Backup.WebDAV.URL = strings.TrimSpace(newConfig.WebDAV.URL) | ||
| h.cfg.Backup.WebDAV.Username = strings.TrimSpace(newConfig.WebDAV.Username) | ||
| h.cfg.Backup.WebDAV.Password = strings.TrimSpace(newConfig.WebDAV.Password) | ||
| h.cfg.Backup.WebDAV.Path = strings.TrimSpace(newConfig.WebDAV.Path) | ||
| h.mu.Unlock() | ||
|
|
||
| if !h.persist(c) { | ||
| return | ||
| } |
There was a problem hiding this comment.
并发安全与数据一致性问题：在 PutBackupConfig 中，锁 h.mu 在第 60 行被提前释放，而写入磁盘的 h.persist(c) 在锁外执行。由于 h.persist 需要读取整个 h.cfg 结构体并将其序列化为 YAML，如果在执行 h.persist 时有其他协程并发修改 h.cfg（例如其他配置更新接口），将会导致数据竞态（Data Race），甚至可能导致序列化输出的配置文件处于不一致的中间状态。\n\n建议将 h.persist(c) 放在锁内执行，以确保配置更新和持久化的原子性与并发安全。
h.mu.Lock()
defer h.mu.Unlock()
h.cfg.Backup.Enabled = newConfig.Enabled
h.cfg.Backup.Schedule = strings.TrimSpace(newConfig.Schedule)
h.cfg.Backup.Storage = strings.TrimSpace(string(newConfig.Storage))
h.cfg.Backup.LocalDir = strings.TrimSpace(newConfig.LocalDir)
h.cfg.Backup.MaxBackups = newConfig.MaxBackups
// Update S3 config
h.cfg.Backup.S3.Endpoint = strings.TrimSpace(newConfig.S3.Endpoint)
h.cfg.Backup.S3.Region = strings.TrimSpace(newConfig.S3.Region)
h.cfg.Backup.S3.Bucket = strings.TrimSpace(newConfig.S3.Bucket)
h.cfg.Backup.S3.Path = strings.TrimSpace(newConfig.S3.Path)
h.cfg.Backup.S3.AccessKey = strings.TrimSpace(newConfig.S3.AccessKey)
h.cfg.Backup.S3.SecretKey = strings.TrimSpace(newConfig.S3.SecretKey)
h.cfg.Backup.S3.UseSSL = newConfig.S3.UseSSL
// Update WebDAV config
h.cfg.Backup.WebDAV.URL = strings.TrimSpace(newConfig.WebDAV.URL)
h.cfg.Backup.WebDAV.Username = strings.TrimSpace(newConfig.WebDAV.Username)
h.cfg.Backup.WebDAV.Password = strings.TrimSpace(newConfig.WebDAV.Password)
h.cfg.Backup.WebDAV.Path = strings.TrimSpace(newConfig.WebDAV.Path)
if !h.persist(c) {
return
}| storage, err := h.createBackupStorageByType(storageType) | ||
| if err != nil { | ||
| log.WithError(err).Warnf("failed to create storage for %s", storageType) | ||
| lastErr = err | ||
| continue | ||
| } |
There was a problem hiding this comment.
并发安全问题：在 CreateBackup 中，第 113 行直接调用了 h.createBackupStorageByType(storageType)，而该方法内部会读取共享的 h.cfg 配置。此时未持有锁 h.mu，会与 PutBackupConfig 等写操作产生数据竞态。\n\n建议在调用 createBackupStorageByType 前加锁，以确保并发安全。
| storage, err := h.createBackupStorageByType(storageType) | |
| if err != nil { | |
| log.WithError(err).Warnf("failed to create storage for %s", storageType) | |
| lastErr = err | |
| continue | |
| } | |
| h.mu.Lock() | |
| storage, err := h.createBackupStorageByType(storageType) | |
| h.mu.Unlock() | |
| if err != nil { | |
| log.WithError(err).Warnf("failed to create storage for %s", storageType) | |
| lastErr = err | |
| continue | |
| } |
| func (s *WebDAVStorage) parseWebDAVResponse(body string) []BackupInfo { | ||
| var backups []BackupInfo | ||
|
|
||
| // Simple string-based parsing - look for .zip files | ||
| lines := strings.Split(body, "\n") | ||
| var currentFile string | ||
| var currentSize int64 | ||
| var currentTime time.Time | ||
|
|
||
| for _, line := range lines { | ||
| line = strings.TrimSpace(line) | ||
|
|
||
| // Extract filename from href or displayname | ||
| if strings.Contains(line, ".zip") { | ||
| if strings.Contains(line, "<d:href>") || strings.Contains(line, "<D:href>") { |
There was a problem hiding this comment.
解析健壮性问题：使用按行分割（strings.Split(body, "\n")）和简单的字符串匹配来解析 WebDAV PROPFIND XML 响应非常脆弱。如果 WebDAV 服务器返回的 XML 响应是单行（未格式化的紧凑 XML），或者使用了不同的命名空间前缀（例如 DAV:、d:、D:），该解析器将无法正确提取备份列表，甚至可能丢失数据。\n\n建议使用 Go 标准库的 encoding/xml 进行结构化解析，以确保兼容各种 WebDAV 服务端实现。
| func (s *Scheduler) UpdateSchedule(ctx context.Context, schedule string, maxBackups int) error { | ||
| s.mu.Lock() | ||
| wasRunning := s.running | ||
| s.schedule = schedule | ||
| s.maxBackups = maxBackups | ||
| s.mu.Unlock() | ||
|
|
||
| if wasRunning { | ||
| s.Stop() | ||
| } | ||
|
|
||
| if schedule != "" { | ||
| return s.Start(ctx) | ||
| } | ||
|
|
||
| return nil | ||
| } |
There was a problem hiding this comment.
锁粒度与竞态条件：在 UpdateSchedule 中，为了避免调用 s.Stop() 和 s.Start() 时发生死锁，代码在第 90 行提前释放了锁 s.mu.Unlock()。然而，这会导致在释放锁到重新加锁的间隙中，其他协程可能会并发调用 Start 或 Stop，从而引发竞态条件。\n\n建议重构代码，提供内部未加锁的辅助方法（如 stop() 和 start(ctx)），并在 UpdateSchedule 中全程持有锁，调用这些辅助方法以确保线程安全。
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 01047414bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| // NewScheduler creates a new backup scheduler. | ||
| func NewScheduler(manager *Manager, schedule string, maxBackups int) *Scheduler { |
There was a problem hiding this comment.
This creates the scheduler API/config surface, but it is never instantiated: a repo-wide rg "NewScheduler|Backup.Enabled|Backup.Schedule" only finds this constructor and the management config setter. As a result, setting backup.enabled: true with any schedule persists successfully but no goroutine ever calls Start, so automatic backups never run; only manual /backup/create works.
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| // Create backup manager | ||
| manager, err := h.createBackupManager() |
There was a problem hiding this comment.
Manager.Restore does not use the storage backend, but this path builds one through createBackupManager() before restoring. If the current backup storage is misconfigured or unavailable (for example backup.storage: s3 with missing/rotated credentials), an uploaded ZIP restore fails at manager creation even though the restore only needs configFilePath, authDir, and logsDir; construct a storage-agnostic manager here instead.
Useful? React with 👍 / 👎.
Critical Security Fixes:
1. Zip Slip path traversal - use filepath.Rel instead of HasPrefix
- HasPrefix('/app/auths', '/app/auths-secret') incorrectly passes
- Now properly validates relative path doesn't start with '..'
Critical Concurrency Fixes:
2. Data race in PutBackupConfig
- Use defer h.mu.Unlock() to keep lock during persist()
- Prevents concurrent modifications during config serialization
3. Unprotected config read in CreateBackup
- Lock h.mu before calling createBackupStorageByType()
- Read maxBackups once with lock held, use local copy
Improvements:
4. Remove unnecessary storage dependency in RestoreBackup
- Restore only needs paths, not storage backend
- Avoids failure when storage misconfigured
- Create manager with nil storage
All critical security and concurrency issues resolved!
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BREAKING CHANGE: ampcode configuration, management endpoints, provider routing, and X-Amp-Thread-Id session affinity are no longer supported
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9d1d6d7f2b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| h.cfg.Backup.WebDAV.Password = strings.TrimSpace(newConfig.WebDAV.Password) | ||
| h.cfg.Backup.WebDAV.Path = strings.TrimSpace(newConfig.WebDAV.Path) | ||
|
|
||
| if !h.persist(c) { |
There was a problem hiding this comment.
When PUT/PATCH /backup/config reaches this call, the handler is still holding h.mu from the defer h.mu.Unlock() above, but persist() in handler.go also calls h.mu.Lock(). Because sync.Mutex is not reentrant, the request blocks forever after mutating the in-memory backup config and keeps the management mutex held, causing later management operations that need h.mu to hang as well; call persistLocked() here or release the lock before persisting.
Useful? React with 👍 / 👎.
| continue | ||
| } | ||
|
|
||
| data, err := manager.Download(name) |
There was a problem hiding this comment.
For S3 and WebDAV backends, the storage implementations do not perform the local backend's traversal/base-name check, yet this handler passes the query name directly into manager.Download (and the delete handler does the same). With backup.storage: s3 and a configured prefix, a request such as name=../other.zip is normalized by S3Storage.objectName outside the backup prefix, and WebDAV builds the URL with the same raw path component, so authenticated backup download/delete calls can access non-backup objects under the same credentials; validate that name is a single .zip base filename before invoking the storage backend.
Useful? React with 👍 / 👎.
P1 Critical Fixes: 1. Mutex deadlock in PutBackupConfig - Use persistLocked() instead of persist() - Prevents double-locking h.mu causing request hang 2. Missing config.yaml validation - Fail backup creation if config.yaml cannot be archived - Prevents silent backup corruption P2 Security & Quality Fixes: 3. Insecure file permissions - Local backups now use 0600 (was 0644) - Restored files use 0600 to protect credentials 4. Path traversal in S3/WebDAV - Validate filename doesn't contain '..' or path separators - Prevent escaping configured backup directory 5. S3 listing errors ignored - Track and return listing errors instead of silently ignoring - Prevents reporting success with incomplete results All P1 and P2 issues from latest Codex review resolved! Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9d4371713
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| mgmt.GET("/backup/config", s.mgmt.GetBackupConfig) | ||
| mgmt.PUT("/backup/config", s.mgmt.PutBackupConfig) | ||
| mgmt.PATCH("/backup/config", s.mgmt.PutBackupConfig) |
There was a problem hiding this comment.
Registering PATCH /backup/config to the full replacement handler makes partial requests destructive: for example, PATCH /backup/config with only {"enabled":true} binds zero values for every omitted field, and PutBackupConfig then overwrites storage, paths, credentials, and use-ssl before persisting. Other PATCH endpoints in this API apply partial updates, so clients can accidentally erase backup storage settings; either remove this route or implement pointer-based patch semantics.
Useful? React with 👍 / 👎.
| h.cfg.Backup.Enabled = newConfig.Enabled | ||
| h.cfg.Backup.Schedule = strings.TrimSpace(newConfig.Schedule) | ||
| h.cfg.Backup.Storage = strings.TrimSpace(string(newConfig.Storage)) | ||
| h.cfg.Backup.LocalDir = strings.TrimSpace(newConfig.LocalDir) | ||
| h.cfg.Backup.MaxBackups = newConfig.MaxBackups |
There was a problem hiding this comment.
When PUT /backup/config receives an enabled S3/WebDAV config with missing required fields, or an invalid schedule such as not-a-duration, this path copies and persists it without invoking the newly added backup.Config.Validate(). The API returns success even though later backup creation or scheduler startup will fail, so reject invalid backup configs before mutating h.cfg.
Useful? React with 👍 / 👎.
| @@ -0,0 +1,272 @@ | |||
| # 备份功能开发说明 | |||
There was a problem hiding this comment.
The repo-scoped AGENTS.md says “New Markdown docs should be in English unless the file is explicitly language-specific”; this newly added Markdown file is not language-specific but is written in Chinese from the heading onward, and the other new backup PR Markdown docs follow the same pattern. This violates the repository documentation convention, so translate these docs or make them explicitly language-specific.
Useful? React with 👍 / 👎.
Remove AMP integration support
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f77a023af0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if root == nil || root.Kind != yaml.MappingNode { | ||
| return | ||
| } | ||
| removeMapKey(root, "ampcode") |
There was a problem hiding this comment.
In deployments with existing ampcode settings, the next management config save now runs this remover and deletes the whole Amp configuration. I checked the repo for AmpCode, ampcode, and X-Amp-Thread-Id; after the internal/api/modules/amp deletion and removal of the NewLegacy registration in NewServer, this pruning is the only remaining Amp reference, so Amp CLI /api/provider/... and management proxy routes disappear and existing configs are silently erased. If this PR is for backup fixes, keep the Amp config/routes or migrate them rather than stripping them here.
Useful? React with 👍 / 👎.
| objectCh := s.client.ListObjects(ctx, s.bucket, minio.ListObjectsOptions{ | ||
| Prefix: s.prefix, | ||
| Recursive: true, |
There was a problem hiding this comment.
When backup.s3.path is set to a prefix such as backups, listing with Prefix: s.prefix also returns objects under adjacent prefixes like backups-old/ or backups2/. In buckets shared with other data, /backup/list can expose those unrelated .zip objects, and retention cleanup can make decisions based on them; normalize non-empty prefixes to include a trailing slash or filter object.Key against the exact backup directory boundary.
Useful? React with 👍 / 👎.
| return | ||
| } | ||
|
|
||
| close(s.stopCh) |
There was a problem hiding this comment.
When UpdateSchedule restarts a running scheduler, Stop closes stopCh but Start reuses the same already-closed channel. The restarted goroutine exits immediately on <-s.stopCh, so subsequent scheduled backups never run, and a later Stop will panic by closing the closed channel again; recreate stopCh before starting after a stop.
Useful? React with 👍 / 👎.
5 participants
根据代码审查结果对功能进行进一步的升级和稳定性修复
🔴 关键安全问题（已修复）
Zip Slip 漏洞 ✅
添加路径验证，检测 .. 序列
验证目标路径在允许目录内
防止目录遍历攻击
🟠 高优先级问题（已修复）
日志文件过滤缺陷 ✅
先过滤出文件，再取最后 10 个
避免目录计入导致丢失日志
静默恢复失败 ✅
跟踪关键文件（config.yaml）恢复状态
失败时返回错误而非成功
聚合错误信息
🟡 中优先级问题（已修复）
冗余 ZIP 创建 ✅
创建一次 ZIP，上传到所有存储
避免多次创建浪费资源
确保所有存储的备份一致
双重 JSON 响应 ✅
移除 PutBackupConfig 中重复的响应
persist() 已写入响应后直接返回
单一后端连接测试 ✅
测试所有配置的存储后端
返回每个后端的测试结果
报告部分失败情况