Steps to reproduce
- Submit a run with a large repo diff (e.g. create a 50MB file)
- Observe the run stuck in pulling and an error in the container logs:
time=2025-05-28T18:10:15.600273Z level=error msg=Unexpected API error err=[http.go:79 api.(*Server).uploadCodePostHandler] unexpected EOF
The problem may or may not occur depending on the network speed/state. Reduce the client timeout to reproduce it:
|
resp = requests.post(self._url("/api/upload_code"), data=file, timeout=REQUEST_TIMEOUT) |
#2673 made it possible to configure upload code size limit, thus allowing for large uploads. unexpected EOF may happen due to the client dropping connection after the timeout (which is 9s now). Clearly, 9s is not sufficient for large uploads and should be increased (e.g. to 1m).
Actual behaviour
No response
Expected behaviour
No response
dstack version
master
Server logs
Additional information
No response
Steps to reproduce
The problem may or may not occur depending on the network speed/state. Reduce the client timeout to reproduce it:
dstack/src/dstack/_internal/server/services/runner/client.py
Line 112 in 5d57e19
#2673 made it possible to configure upload code size limit, thus allowing for large uploads.
unexpected EOFmay happen due to the client dropping connection after the timeout (which is 9s now). Clearly, 9s is not sufficient for large uploads and should be increased (e.g. to 1m).Actual behaviour
No response
Expected behaviour
No response
dstack version
master
Server logs
Additional information
No response