Skip to content

storage: support local filesystem storage with localPath config#985

Open
huanghaoyuanhhy wants to merge 1 commit into
zilliztech:mainfrom
huanghaoyuanhhy:feat/local-storage-support
Open

storage: support local filesystem storage with localPath config#985
huanghaoyuanhhy wants to merge 1 commit into
zilliztech:mainfrom
huanghaoyuanhhy:feat/local-storage-support

Conversation

@huanghaoyuanhhy
Copy link
Copy Markdown
Collaborator

@huanghaoyuanhhy huanghaoyuanhhy commented Mar 17, 2026

Fixes #964

Summary

Support Milvus Standalone local storage (storageType: local) by adding localPath config that maps to Milvus localStorage.path. This enables backup/restore for Milvus deployed via the official standalone docker method which uses COMMON_STORAGETYPE=local.

Changes

  • Add LocalPath field to storage.Config struct
  • Add minio.localPath / minio.backupLocalPath config fields with env var support (MINIO_LOCAL_PATH / MINIO_BACKUP_LOCAL_PATH), backupLocalPath inherits from localPath if not set
  • Rewrite LocalClient with baseDir (localPath/bucket), absPath(key) and relPath(abs) to correctly resolve relative keys
  • BucketExist now checks if baseDir directory exists; CreateBucket creates it via os.MkdirAll
  • Update all LocalClient tests to use Config{LocalPath: tempDir, Bucket: "test-bucket"} pattern
  • Document new fields in configs/backup.yaml
  • Add configs/backup-local.yaml example config for Milvus Standalone local storage
  • Add e2e CI test using docker run with embedded etcd, matching official deployment

Breaking Changes

LocalClient now resolves all keys relative to localPath/bucket instead of treating them as absolute paths. Users who previously relied on absolute-path keys with storageType: local must set localPath to the correct base directory (the host path mapping to Milvus localStorage.path).

User Config Example

minio:
  storageType: "local"
  localPath: "volumes/milvus/data"      # host path to Milvus localStorage.path
  bucketName: "a-bucket"
  rootPath: "files"

  backupStorageType: "local"
  backupLocalPath: "volumes/backup"
  backupBucketName: "milvus-backup"
  backupRootPath: "backup"

@sre-ci-robot sre-ci-robot added the kind/improvement Changes related to something improve, likes ut and code refactor label Mar 17, 2026
@sre-ci-robot
Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: huanghaoyuanhhy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Mar 17, 2026

Codecov Report

❌ Patch coverage is 86.90476% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.46%. Comparing base (adae7f7) to head (7cbb895).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
internal/storage/local.go 86.95% 3 Missing and 3 partials ⚠️
core/restore/coll_task.go 76.92% 3 Missing ⚠️
internal/storage/factory.go 0.00% 2 Missing ⚠️

❌ Your project status has failed because the head coverage (28.46%) is below the target coverage (80.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #985      +/-   ##
==========================================
+ Coverage   28.23%   28.46%   +0.23%     
==========================================
  Files         104      104              
  Lines        9835     9871      +36     
==========================================
+ Hits         2777     2810      +33     
- Misses       6856     6858       +2     
- Partials      202      203       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mergify mergify Bot added ci-passed and removed ci-passed labels Mar 17, 2026
@huanghaoyuanhhy huanghaoyuanhhy force-pushed the feat/local-storage-support branch from 37959a9 to 178e2f7 Compare March 17, 2026 10:55
@mergify mergify Bot added the ci-passed label Mar 17, 2026
@huanghaoyuanhhy huanghaoyuanhhy force-pushed the feat/local-storage-support branch from 178e2f7 to 9065097 Compare March 17, 2026 11:24
@mergify mergify Bot removed the ci-passed label Mar 17, 2026
@huanghaoyuanhhy huanghaoyuanhhy force-pushed the feat/local-storage-support branch 4 times, most recently from b179e25 to 4276810 Compare March 18, 2026 05:57
@mergify mergify Bot added the ci-passed label Mar 18, 2026
@huanghaoyuanhhy huanghaoyuanhhy force-pushed the feat/local-storage-support branch from 4276810 to 7c7938d Compare March 18, 2026 06:26
@mergify mergify Bot added ci-passed and removed ci-passed labels Mar 18, 2026
@huanghaoyuanhhy huanghaoyuanhhy force-pushed the feat/local-storage-support branch from 7c7938d to be89699 Compare March 18, 2026 07:20
@mergify mergify Bot removed the ci-passed label Mar 18, 2026
@huanghaoyuanhhy huanghaoyuanhhy force-pushed the feat/local-storage-support branch from be89699 to e6567f6 Compare March 18, 2026 08:15
@mergify mergify Bot added ci-passed and removed ci-passed labels Mar 18, 2026
@huanghaoyuanhhy huanghaoyuanhhy force-pushed the feat/local-storage-support branch from b28e9d3 to 91a7ce9 Compare April 17, 2026 04:19
Support Milvus standalone with local storage (COMMON_STORAGETYPE=local)
by mapping minio.localPath / minio.backupLocalPath to the host directory
that backs Milvus localStorage.path.

LocalClient now resolves keys against baseDir = localPath/bucket so all
keys returned by ListPrefix are relative, matching the S3 client.

The CI test dumps the milvus and backup volume trees around backup and
restore, and keeps restore temp files, so failures expose what is
actually on disk where Milvus expects to read it.

Signed-off-by: huanghaoyuanhhy <haoyuan.huang@zilliz.com>
@huanghaoyuanhhy huanghaoyuanhhy force-pushed the feat/local-storage-support branch from 91a7ce9 to 7cbb895 Compare April 17, 2026 06:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved dco-passed kind/improvement Changes related to something improve, likes ut and code refactor size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: milvus-backup does not support backing up from local storage to MinIO for Milvus standalone?

3 participants