feat: Support XNet accelerated transfer and enhance CLI commands (#113)

* Add Xnet upload and  download

* Add Xnet upload and  download

* Download compatible with old version

* enhance tips msg

* fix ut

* fix ut

* Add system info

* Adaptation setup

* enhance code

* override the default cache dir for hf

* Add cli  doc

* remove version and  __init__ version

* remove version

* fix conflic int transformer and hunggingface-hub

* fix databaset load error with databaset version 4.4.1

* fix finetune issue

* fix finetune issue

* Add download cli  for single file

* Add the feature doc in README.md/README-CN.md

* Add the feature doc in README.md/README-CN.md

---------

Co-authored-by: cemeng <cemengzhang@yntengyun.com>

Author: CementZhang

.env.example

@@ -0,0 +1,10 @@
+# csghub user token
+CSGHUB_TOKEN=your_csghub_token
+# csghub domain
+CSGHUB_DOMAIN==https://hub.csghub.com
+# test repo and types
+CSGHUB_TEST_REPOS='
+[
+   {"repo_id": "ns/model-repo", "type": "model", "disable_xnet": "false", "size_kb": 200},
+   {"repo_id": "ns/dataset-repo", "type": "dataset", "disable_xnet": "true"}
+]'

.gitignore

@@ -11,4 +11,6 @@ dist/
 .idea
 .pypirc
 .vscode/
+.trae/
 .DS_Store
+/.env

README.md

@@ -28,6 +28,11 @@ The main functions are:
 1. Repo downloading（model/dataset）
 2. Repo information query（Compatible with huggingface）
 
+**XNet Accelerated Transfer (New!)**: Next-generation storage and version control technology for large-scale AI/ML data.
+- **Storage Optimization**: Significantly reduces storage costs (tested savings > 50%) via intelligent Content-Defined Chunking and deduplication.
+- **High-Speed Transfer**: Incremental updates ensure only changed data chunks are transferred, boosting upload/download speeds by multiples.
+- **Enabled by Default**: Automatically optimizes upload, download, and storage for LFS large files. To disable, set the environment variable `CSGHUB_DISABLE_XNET=true` to fallback to standard LFS mode.
+
 ## Get My Token
 
 Visit [OpenCSG](https://opencsg.com/), click on Sign Up in the top right corner to complete the user registration process. Use the successfully registered username and password to log in to [OpenCSG](https://opencsg.com/). After logging in, find [Access Token](https://opencsg.com/settings/access-token) under Account Settings to obtain the token.
@@ -96,4 +101,4 @@ For detailed SDK usage examples, including model/dataset downloading, file uploa
 1. Interacting with CSGHub via command-line tools
 2. Management operations such as creation and modification of CSGHub repositories
 3. Model deployment locally or online
-4. Model fine-tuning locally or online
+4. Model fine-tuning locally or online

README_cn.md

@@ -28,6 +28,11 @@ CSGHub SDK 是一个强大的 Python 客户端，专门设计用于与 CSGHub 
 1. 仓库下载（模型/数据集）
 2. 仓库信息查询（与huggingface兼容）
 
+**XNet 加速传输（新！）**：面向大规模 AI/ML 数据的下一代存储与版本控制技术。
+- **存储优化**：通过智能分块（Content-Defined Chunking）与数据去重，大幅降低存储成本（实测节省 50% 以上）。
+- **加速传输**：增量更新机制，仅传输变化的数据块，上传/下载速度提升数倍。
+- **默认开启**：针对 LFS 大文件的上传、下载与存储自动优化。如需禁用，设置环境变量 `CSGHUB_DISABLE_XNET=true` 即可降级为普通 LFS 模式。
+
 ## 获取Token
 
 浏览器访问[OpenCSG](https://opencsg.com/)，点击右上角`注册`完成用户注册过程，使用已经注册成功的用户和密码登录[OpenCSG](https://opencsg.com/)，登录成功后在`账号设置`中找到[`Access Token`](https://opencsg.com/settings/access-token)来获取token。

doc/cli.md

@@ -17,6 +17,9 @@ $env:CSGHUB_TOKEN="your_access_token"
 # download model
 csghub-cli download OpenCSG/csg-wukong-1B
 
+# download single file
+csghub-cli download OpenCSG/csg-wukong-1B config.json
+
 # download model with allow patterns '*.json' and ignore '*_config.json' pattern of files
 csghub-cli download OpenCSG/csg-wukong-1B --allow-patterns "*.json" --ignore-patterns "tokenizer.json"
 
@@ -67,6 +70,12 @@ csghub-cli upload wanghh2000/m01 /Users/hhwang/temp/jsonl test/files -k xxxxxx
 
 # auto upload large file in multi-part mode by 'git push' under working directory
 csghub-cli lfs-enable-largefiles ./
+
+# print csghub-cli version
+csghub-cli version
+
+# print csghub-cli environment for debug or report issue with `csghub-cli env`
+csghub-cli env
 ```
 
 ### Example for Window Powershell

doc/cli_cn.md

@@ -17,6 +17,9 @@ $env:CSGHUB_TOKEN="your_access_token"
 # 模型下载
 csghub-cli download OpenCSG/csg-wukong-1B
 
+# 下载单个文件
+csghub-cli download OpenCSG/csg-wukong-1B config.json
+
 # 模型下载时允许'*.json'模式的文件并忽略'*_config.json'模式的文件
 csghub-cli download OpenCSG/csg-wukong-1B --allow-patterns "*.json" --ignore-patterns "*_config.json"
 
@@ -67,6 +70,12 @@ csghub-cli upload wanghh2000/m01 /Users/hhwang/temp/jsonl test/files -k xxxxxx
 
 # 在当前工作目录启用大文件分片上传功能
 csghub-cli lfs-enable-largefiles ./
+
+# 查看csghub-sdk版本号
+csghub-cli version
+
+# 打印csghub-cli环境变量，如果需要提issue，请执行此命令，并且复制输出到issue中。
+csghub-cli env
 ```
 
 ### 示例命令使用在Window Powershell环境

examples/run_finetune_bert.py

@@ -35,7 +35,7 @@ def data_collator() -> Any:
 outputDir = "/Users/hhwang/temp/ff"
 args = TrainingArguments(
     outputDir,
-    evaluation_strategy="steps",
+    eval_strategy="steps",
     save_strategy="steps",
     logging_strategy="steps",
     logging_steps = 2,
@@ -57,7 +57,7 @@ def data_collator() -> Any:
     args,
     train_dataset=train_dataset,
     eval_dataset=eval_dataset,
-    tokenizer=tokenizer,
+    data_collator=data_collator(),
 )
 
 trainResult = trainer.train()

pycsghub/api_client/__init__.py

@@ -0,0 +1,15 @@
+from typing import Optional
+
+from pycsghub.utils import disable_xnet, get_endpoint, get_token_to_send
+from .api_client import CsghubApi
+from .api_client_interface import HubApi
+from .api_client_xet import CsgXnetApi
+
+def get_csghub_api(token: Optional[str] = None,
+                   endpoint: Optional[str] = None,
+                   user_name: Optional[str] = None) -> HubApi:
+    token = get_token_to_send(token)
+    endpoint = get_endpoint(endpoint=endpoint)
+    return CsghubApi(token=token, endpoint=endpoint, user_name=user_name) if disable_xnet() else CsgXnetApi(token=token,
+                                                                                                            endpoint=endpoint,
+                                                                                                            user_name=user_name)

pycsghub/api_client/api_client.py

@@ -0,0 +1,228 @@
+from __future__ import annotations
+
+from fnmatch import fnmatch
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Union
+
+from huggingface_hub.utils import filter_repo_objects
+
+from pycsghub.commit_ops import build_payload, CommitOperationAdd, CommitOperationDelete
+from pycsghub.csghub_api import CsgHubApi
+from pycsghub.file_download import file_download as csghub_file_download
+from pycsghub.snapshot_download import snapshot_download as csghub_snapshot_download
+from pycsghub.utils import get_endpoint, get_repo_info
+from pycsghub.cmd import repo
+
+class CsghubApi:
+    def __init__(self, token: Optional[str] = None, endpoint: Optional[str] = None, user_name: Optional[str] = None):
+        self._api = CsgHubApi(token=token, endpoint=endpoint)
+        self._token = token
+        self._endpoint = endpoint
+        self._user_name = user_name
+    
+    def hf_hub_download(
+        self,
+        repo_id: str,
+        filename: str,
+        subfolder: Optional[str] = None,
+        repo_type: Optional[str] = None,
+        revision: Optional[str] = None,
+        library_name: Optional[str] = None,
+        library_version: Optional[str] = None,
+        cache_dir: Optional[Union[str, Path]] = None,
+        local_dir: Optional[Union[str, Path]] = None,
+        user_agent: Optional[Union[Dict, str]] = None,
+        force_download: bool = False,
+        force_filename: Optional[str] = None,
+        proxies: Optional[Dict] = None,
+        etag_timeout: float = 10,
+        resume_download: bool = False,
+        token: Optional[Union[bool, str]] = None,
+        local_files_only: bool = False,
+        headers: Optional[Dict[str, str]] = None,
+        endpoint: Optional[str] = None,
+        **kwargs
+    ) -> Union[str, Path]:
+        # Adapt to csghub_file_download signature
+        # Note: subfolder handling in csghub might be different, usually path in filename
+        final_filename = f"{subfolder}/{filename}" if subfolder else filename
+        
+        # Extract dry_run/quiet/source from kwargs if present
+        dry_run = kwargs.get('dry_run', False)
+        quiet = kwargs.get('quiet', False)
+        source = kwargs.get('source', None)
+        
+        return csghub_file_download(
+            repo_id=repo_id,
+            file_name=final_filename,
+            revision=revision,
+            repo_type=repo_type,
+            cache_dir=cache_dir,
+            local_dir=local_dir,
+            local_files_only=local_files_only,
+            headers=headers,
+            endpoint=endpoint or self._endpoint,
+            token=token or self._token,
+            force_download=force_download,
+            dry_run=dry_run,
+            quiet=quiet,
+            source=source
+        )
+    
+    def snapshot_download(
+        self,
+        repo_id: str,
+        revision: Optional[str] = None,
+        repo_type: Optional[str] = None,
+        cache_dir: Optional[Union[str, Path]] = None,
+        local_dir: Optional[Union[str, Path]] = None,
+        library_name: Optional[str] = None,
+        library_version: Optional[str] = None,
+        user_agent: Optional[Union[Dict, str]] = None,
+        proxies: Optional[Dict] = None,
+        etag_timeout: float = 10,
+        resume_download: bool = False,
+        token: Optional[Union[bool, str]] = None,
+        local_files_only: bool = False,
+        allow_patterns: Optional[Union[List[str], str]] = None,
+        ignore_patterns: Optional[Union[List[str], str]] = None,
+        max_workers: int = 8,
+        tqdm_class: Optional[Any] = None,
+        # Extra args for CSGHub compatibility or enhancements
+        endpoint: Optional[str] = None,
+        source: Optional[str] = None,
+        quiet: Optional[bool] = False,
+        dry_run: Optional[bool] = False,
+        force_download: Optional[bool] = False,
+    ):
+        # Delegate to pycsghub.snapshot_download
+        return csghub_snapshot_download(
+            repo_id=repo_id,
+            repo_type=repo_type,
+            revision=revision,
+            cache_dir=cache_dir,
+            local_dir=local_dir,
+            local_files_only=local_files_only,
+            allow_patterns=allow_patterns,
+            ignore_patterns=ignore_patterns,
+            endpoint=endpoint or self._endpoint,
+            token=token or self._token,
+            source=source,
+            dry_run=dry_run,
+            force_download=force_download,
+            max_workers=max_workers,
+            quiet=quiet
+        )
+    
+    def create_repo(
+        self,
+        repo_id: str,
+        repo_type: Optional[str] = None,
+        exist_ok: bool = False,
+        private: bool = False,
+        **kwargs
+    ):
+        # Reusing Repository logic for creation
+        from pycsghub.repository import Repository
+        repo = Repository(
+            repo_id=repo_id,
+            upload_path="..",
+            repo_type=repo_type,
+            token=self._token,
+            endpoint=self._endpoint,
+            auto_create=False
+        )
+        try:
+            repo.create_new_repo()
+        except Exception:
+            if not exist_ok:
+                raise
+        return repo
+    
+    def repo_info(
+        self,
+        repo_id: str,
+        revision: Optional[str] = None,
+        repo_type: Optional[str] = None,
+        token: Optional[str] = None,
+    ):
+        return get_repo_info(
+            repo_id=repo_id,
+            revision=revision,
+            repo_type=repo_type,
+            token=token or self._token,
+            endpoint=self._endpoint
+        )
+    
+    def create_branch(
+        self,
+        repo_id: str,
+        branch: str,
+        repo_type: Optional[str] = None,
+        exist_ok: bool = False
+    ):
+        try:
+            # Reusing Repository logic until CsgHubApi supports create_branch directly or via new endpoint
+            from pycsghub.repository import Repository
+            repo = Repository(
+                repo_id=repo_id,
+                upload_path="..",
+                repo_type=repo_type,
+                branch_name=branch,
+                token=self._token,
+                endpoint=self._endpoint
+            )
+            return repo.create_new_branch()
+        except Exception:
+            if not exist_ok:
+                raise
+    
+    def upload_file(
+        self,
+        *,
+        path_or_fileobj: Union[str, Path, bytes],
+        path_in_repo: str,
+        repo_id: str,
+        repo_type: Optional[str] = None,
+        revision: Optional[str] = None,
+        commit_message: Optional[str] = None,
+        commit_description: Optional[str] = None,
+        create_pr: Optional[bool] = None,
+        parent_commit: Optional[str] = None,
+    ) -> dict:
+        return repo.upload_files(
+            repo_id=repo_id,
+            repo_type=repo_type,
+            repo_file=path_or_fileobj,
+            path_in_repo=path_in_repo,
+            revision=revision,
+            endpoint=self._endpoint,
+            token=self._token,
+        )
+    
+    def upload_folder(
+        self,
+        *,
+        repo_id: str,
+        folder_path: Union[str, Path],
+        path_in_repo: Optional[str] = None,
+        repo_type: Optional[str] = None,
+        revision: Optional[str] = None,
+        commit_message: Optional[str] = None,
+        commit_description: Optional[str] = None,
+        allow_patterns: Optional[Union[List[str], str]] = None,
+        ignore_patterns: Optional[Union[List[str], str]] = None,
+        delete_patterns: Optional[Union[List[str], str]] = None,
+        create_pr: Optional[bool] = None,
+        parent_commit: Optional[str] = None,
+    ) -> dict:
+        return repo.upload_folder(
+            repo_id=repo_id, 
+            repo_type=repo_type,
+            local_path=folder_path,
+            path_in_repo=path_in_repo,
+            revision=revision,
+            endpoint=self._endpoint,
+            token=self._token,
+            user_name=self._user_name,
+        )