Environment
- Deployment Manifests (Helm/YAML): v0.1.10 (or older)
- Controller Manager Image: v0.1.13
- Client SDK: v0.1.8
Bug Description
After upgrading the OpenSandbox server image to v0.1.13, the controller-manager pod fails to start and enters a CrashLoopBackOff state. As a result, all API requests to the sandbox service fail immediately, resulting in TCP connection drops.
Clients using the Python SDK receive the following exception:
httpx.RemoteProtocolError: Server disconnected without sending a response
Raw curl commands return:
curl: (52) Empty reply from server
Checking the pod logs reveals that the binary is crashing instantly due to an unrecognized command-line flag:
$ kubectl logs opensandbox-controller-manager-58b457c97b-bm7hk -n opensandbox-system
flag provided but not defined: -kube-client-qps
Usage of /workspace/server:
-enable-file-log
Enable log output to file
-enable-http2
If set, HTTP/2 will be enabled for the metrics and webhook servers
...
Root Cause Analysis
The Go binary in the v0.1.13 image no longer defines or accepts the -kube-client-qps command-line flag. However, because our deployment manifests (Helm chart or Kubernetes Deployment YAML) are still from an older version (e.g., v0.1.10), they are still passing -kube-client-qps in the container's args.
When the v0.1.13 binary starts and parses the arguments, the flag package throws an error for the undefined flag, prints the usage text, and exits with a fatal error. This brings down the entire controller-manager pod, leaving the service without an active backend.
Expected Behavior
- Backward Compatibility: If a flag is removed, it is usually better to keep it defined but ignored (with a deprecation warning log) for a few minor releases. This ensures backward compatibility during rolling upgrades or when users mix slightly older Helm charts with newer images.
- Documentation: Breaking changes like the removal of command-line flags (e.g.,
-kube-client-qps) should be prominently documented in the Release Notes so users are aware that they must update their Helm charts/Deployment manifests simultaneously with the image.
Steps to Reproduce
- Deploy OpenSandbox using older manifests (e.g., v0.1.10) which include the
-kube-client-qps flag in the deployment container args.
- Update the container image tag of the
opensandbox-controller-manager deployment to v0.1.13.
- Check the pod logs to observe the
flag provided but not defined: -kube-client-qps crash.
- Attempt to hit the service endpoint and observe the immediate TCP connection drop.
Environment
Bug Description
After upgrading the OpenSandbox server image to
v0.1.13, the controller-manager pod fails to start and enters aCrashLoopBackOffstate. As a result, all API requests to the sandbox service fail immediately, resulting in TCP connection drops.Clients using the Python SDK receive the following exception:
httpx.RemoteProtocolError: Server disconnected without sending a responseRaw
curlcommands return:curl: (52) Empty reply from serverChecking the pod logs reveals that the binary is crashing instantly due to an unrecognized command-line flag:
$ kubectl logs opensandbox-controller-manager-58b457c97b-bm7hk -n opensandbox-system flag provided but not defined: -kube-client-qps Usage of /workspace/server: -enable-file-log Enable log output to file -enable-http2 If set, HTTP/2 will be enabled for the metrics and webhook servers ...Root Cause Analysis
The Go binary in the
v0.1.13image no longer defines or accepts the-kube-client-qpscommand-line flag. However, because our deployment manifests (Helm chart or Kubernetes Deployment YAML) are still from an older version (e.g.,v0.1.10), they are still passing-kube-client-qpsin the container'sargs.When the
v0.1.13binary starts and parses the arguments, theflagpackage throws an error for the undefined flag, prints the usage text, and exits with a fatal error. This brings down the entire controller-manager pod, leaving the service without an active backend.Expected Behavior
-kube-client-qps) should be prominently documented in the Release Notes so users are aware that they must update their Helm charts/Deployment manifests simultaneously with the image.Steps to Reproduce
-kube-client-qpsflag in the deployment containerargs.opensandbox-controller-managerdeployment tov0.1.13.flag provided but not defined: -kube-client-qpscrash.