[Bug] 404 Not Found when uploading model using xnet (xet-write-token endpoint missing)

[sfeng1996]

Version Information

• csghub-server: v1.16.1-ce
• csghub-cli: 0.8.4

Upload Command

csghub-cli --log-level DEBUG upload root/Qwen3.5-9B . -k <apikey> -e <endpoint>

Error Message

HTTPStatusError: Client error '404 Not Found' for url 'http://csghub.10.200.9.249.nip.io:30081/hf/api/models/root/Qwen3.5-9B/xet-write-token/main'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

[CementZhang]

Hi @sfeng1996, thanks for reporting this. 👍

From the version information you provided:

• csghub-server: v1.16.1-ce
• csghub-cli: 0.8.4

The CE version of CSGHub does not include the xnet feature. The xet-write-token endpoint is only available in the Enterprise Edition (EE).

Because of this, when the CLI tries to request the xet-write-token API, the server cannot find the endpoint and returns 404 Not Found.

Could you please confirm:

1. Whether the repository root/Qwen3.5-9B was manually modified in the database to enable xnet?
2. Or whether there were any custom changes made to enable xnet-related fields or settings?

If you want to use xnet-based upload, it requires the EE version where the corresponding API endpoints are implemented.

Thanks!

[sfeng1996]

Hi @CementZhang,

Thank you for your detailed response!

Regarding your questions:

1. Database Modification: Yes, I did manually modify xnet_enabled to true in the database to try and enable the feature.
2. Test Results: As you suspected, it still returned a 404. I've checked the code and confirmed that in the CE version, api_ce.go does not register the xnet API routes (such as xet-write-token). Therefore, even if enabled in the database, the routing layer cannot find the corresponding handlers.
3. Deployment Advice: Since the xnet feature is not supported in the CE version, is it correct to assume that we do not need to deploy any xnet-related components when performing a private deployment of the CE version?

One more follow-up regarding performance:
I observed that csghub-cli upload is significantly faster than csghub-cli upload-large-folder and even git push.
Initially, I thought this was due to xnet, but now that we've confirmed the CE version doesn't have xnet functionality, I'm curious from a server-side logic perspective: Why is csghub-cli upload so much faster than the other methods? Are there major differences in the underlying implementation (e.g., using different API paths or processing mechanisms)?

Looking forward to your insights. Thanks again!

[CementZhang]

Hi @sfeng1996,

Thanks again for your detailed explanation.

Regarding the performance difference between the upload methods, I did a brief analysis of the implementations and would like to summarize my understanding below.

1. csghub-cli upload

This command uses the Hugging Face Hub library as the underlying implementation.

Key characteristics:

• File-level deduplication
  It computes the SHA256 of all local files using multiple threads, then sends a preupload request to compare the file tree with the server.
  Only new or modified files are uploaded, while identical files are skipped (instant deduplication).

• High concurrency & multipart upload
  It uses a mature thread pool mechanism to perform multipart uploads for LFS objects concurrently, which allows much better network throughput utilization.

2. csghub-cli upload-large-folder

This is a custom multi-threaded upload pipeline implemented by the OpenCSG team in Python.

Although it also uploads via HTTP APIs, it uses a multi-stage queue architecture:

hash queue -> preupload queue -> multipart upload queue

This pipeline enables efficient parallel processing for large numbers of files.

3. Why both are faster than git push (Git-LFS)?

For repositories containing large files (e.g., model weights):

• Git-LFS typically treats each file as a single upload task, with relatively limited concurrency.

• In contrast, both:

  • csghub-cli upload
  • csghub-cli upload-large-folder

  implement multipart uploads and higher concurrency, allowing large files to be uploaded in parallel chunks, which significantly improves throughput.

Therefore, even without XNet, these two upload methods can still be noticeably faster than a standard git push.

---

We recommend using the csghub-cli upload method directly to upload files; there is no need to manually determine whether a file is an LFS object, as the tool automatically handles this detection internally to ensure optimal upload performance.

[sfeng1996]

Hi @CementZhang,

Thank you for the detailed comparison!

Regarding your point about csghub-cli upload, I've conducted some additional tests and observed that it does not seem to support breakpoint resumption (resume upload). When an upload is interrupted and restarted, the progress starts from the beginning for each file.

Here is the log from my test:

[root@aicloud249 model]# time csghub-cli upload root/Qwen3.5-9B-origin ./Qwen3.5-9B-origin -k 11cb8323a249421e8fece0bc7b1dcc1d -e http://csghub.10.200.9.249.nip.io:30081/
model.safetensors-00004-of-00004.safetensors: 100%|██████████████████████████████████████████████| 3.33G/3.33G [00:34<00:00, 95.3MB/s]
model.safetensors-00001-of-00004.safetensors:  91%|██████████████████████████████████████████▉    | 4.82G/5.28G [00:46<00:03, 127MB/s^Upload 4 LFS files:   0%|                                                                                       | 0/4 [00:46<?, ?it/s]
... (interrupted by KeyboardInterrupt)

real    1m4.276s
user    0m24.405s
sys     0m15.913s

[root@aicloud249 model]# time csghub-cli upload root/Qwen3.5-9B-origin ./Qwen3.5-9B-origin -k 11cb8323a249421e8fece0bc7b1dcc1d -e http://csghub.10.200.9.249.nip.io:30081/
model.safetensors-00001-of-00004.safetensors:  13%|██████▏                                         | 674M/5.28G [00:06<00:40, 115MB/s^Upload 4 LFS files:   0%|                                                                                       | 0/4 [00:07<?, ?it/s]
model.safetensors-00001-of-00004.safetensors:  16%|███████▍                                        | 824M/5.28G [00:08<00:40, 111MB/s^

As shown in the log:

1. In the first attempt, model.safetensors-00001-of-00004.safetensors reached 91% before I interrupted it.
2. In the second attempt, the upload for the same file started from 0% again (the log shows 13% and 16% shortly after starting).

Is this the intended behavior for the upload command, or is there a way to enable resume-on-interrupt functionality?

Thanks!

[CementZhang]

Hi @sfeng1996,

Thanks for the rigorous testing! You are absolutely correct: csghub-cli upload does not support breakpoint resumption (resume-on-interrupt) in its current implementation.

Here is the technical breakdown of why this happens and how it differs from upload-large-folder:

1. Why csghub-cli upload cannot resume

The upload command (in non-XNet mode) inherits directly from the huggingface_hub library's HfApi.

• Stateless Design: The underlying LFS upload logic is designed to be stateless. It does not persist any "session state" or upload_id to your local disk.
• Linear Execution: When you run the command, it requests a new upload session from the server. It iterates through the file parts linearly and uploads them one by one. It has no mechanism to check if a specific part was already uploaded in a previous (failed) session.

2. The Trade-off: Speed vs. Resilience

• csghub-cli upload:
  • Pros: Extremely fast (optimized HTTP Commit API, minimal overhead).
  • Cons: No resume capability. If it fails, you must restart the specific file.
• csghub-cli upload-large-folder:
  • Pros: Supports breakpoint resumption. It maintains local .metadata files to track upload_id and completed parts. If interrupted, it skips already uploaded chunks.
  • Cons: Slower (due to Python GIL and complex queue management overhead).

3. Recommendation

If you are in a network environment where interruptions are frequent and you need to upload very large files (e.g., >10GB), I recommend using csghub-cli upload-large-folder. It is specifically designed for resilience.

---

Currently, the csghub-cli upload command is transitioning toward an "xnet" upload mode. This mode essentially serves as an advanced version of resumable uploads. Although it has not yet been open-sourced, you can stay tuned for future updates regarding xnet.

[sfeng1996]

Hi @CementZhang,

Thank you for the detailed technical breakdown!

Since csghub-cli upload is significantly faster than upload-large-folder, it would be a huge performance boost if it supported breakpoint resumption. Are there any plans in the community roadmap to implement this feature for the upload command?

Additionally, regarding the XNet protocol you mentioned, does the team have any intention or timeline to open-source it to the community version in the future? This would be very beneficial for users with large model deployments in private environments.

Looking forward to your insights!

[CementZhang]

Hi @sfeng1996,

Thank you for your suggestion.

Supporting resume capability for csghub-cli upload is indeed a valuable improvement, and we agree that it could significantly enhance reliability and performance in unstable network environments. We will consider upgrading this functionality in the future.

If you’re interested, contributions are very welcome! You are welcome to submit improvements or proposals to the repository:
https://github.com/OpenCSGs/csghub-sdk

Regarding XNet, there is currently no clear plan to open source it at this time.

Thanks again for your feedback!

[CementZhang]

Enhance PR: #881