dev-server: docker image 1.14 not removing all overrides

[ajworkos]

Is this a support request?

No

Describe the bug

Using the docker image with tag 1.14 results in an error when attempting to remove all overrides

It feels like the ui code is new but the ldcli go code is old in that image. The UI code calls the rest endpoint DELETE /dev/projects/:projectKey/overrides which returns a 404.

When I pull from this repo locally and start the dev server using go run main.go dev-server, the same DELETE endpoint returns 204

To reproduce

• Start a docker container using launchdarkly/ldcli:1.14.0 with the configuration in this docker compose file
• Add an override
• Click "Remove all overrides" and note the 404 response in the browser console

To compare

• pull this repo and run go run main.go dev-server with the same configuration as the docker container was started
• open the UI, add an override
• Click "Remove all overrides" and note the 204 response in the browser console

Expected behavior
Expect a 204, not a 404

Logs

CLI version

launchdarkly/ldcli:1.14.0

When I output the version of ldcli inside the container I get

ldcli version 1.14.0

OS/platform

launchdarkly/ldcli:1.14.0

Additional context

N/A

[ajworkos]

Oh, just saw this PR - related?

[mike-zorn]

Huh. I just tried the same test and it was working for me: I got a 204 instead of a 404.

My logs

❯ docker run -p 8765:8765 launchdarkly/ldcli:1.14.0 dev-server start --access-token 'REDACTED' --base-uri 'REDACTED' --dev-stream-uri 'REDACTED' --project default --source staging --sync-once
2025/05/29 20:27:57 Using database at /root/.local/state/ldcli/dev_server.db
2025/05/29 20:27:57 Using database at /root/.local/state/ldcli/dev_server.db
2025/05/29 20:27:57 Using database at /root/.local/state/ldcli/dev_server.db
2025/05/29 20:27:57 Initial project [default] with env [staging]
2025/05/29 20:27:57 GetSdkKey - projectKey: default, environmentKey: staging
[LaunchDarkly] 2025/05/29 20:27:57 INFO: Starting LaunchDarkly client 7.8.0
[LaunchDarkly] 2025/05/29 20:27:57 INFO: Starting LaunchDarkly streaming connection
[LaunchDarkly] 2025/05/29 20:27:57 INFO: Waiting up to 5000 milliseconds for LaunchDarkly client to start...
[LaunchDarkly] 2025/05/29 20:27:57 INFO: Connecting to LaunchDarkly stream
[LaunchDarkly] 2025/05/29 20:27:59 INFO: LaunchDarkly streaming is active
[LaunchDarkly] 2025/05/29 20:27:59 INFO: Initialized LaunchDarkly client
[LaunchDarkly] 2025/05/29 20:27:59 INFO: Closing LaunchDarkly client
2025/05/29 20:27:59 Fetching all flags for project 'default'
2025/05/29 20:28:09 Successfully synced Initial project [default]
2025/05/29 20:28:09 Server running on 0.0.0.0:8765
2025/05/29 20:28:09 Access the UI for toggling overrides at http://localhost:8765/ui or by running `ldcli dev-server ui`
2025/05/29 20:28:17 Notify: event {FlagKey:ab.testing ProjectKey:default FlagState:{Value:true Version:199}} to observers
192.168.65.1 - - [29/May/2025:20:28:17 +0000] "PUT /dev/projects/default/overrides/ab.testing HTTP/1.1" 200 31 "http://localhost:8765/ui/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"
192.168.65.1 - - [29/May/2025:20:28:19 +0000] "DELETE /dev/projects/default/overrides HTTP/1.1" 204 0 "http://localhost:8765/ui/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"

Can you share the request log from your container?

Oh, just saw #540 - related?

I don't think so: there was an issue with publishing to homebrew in 1.14.0, but we were able to publish to docker successfully.

[ajworkos]

Relevant log in the container

2025-05-29 18:27:18 ld_dev_server  | 172.18.0.1 - - [29/May/2025:21:27:18 +0000] "DELETE /dev/projects/default/overrides HTTP/1.1" 404 0 "http://localhost:8765/ui/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36"

I figured out the bug.

Reproduction:

1. Start a container synced to a project
2. Create an override for a flag
3. Delete that flag from your launch darkly instance
4. Sync your local dev server to launch darkly
5. You'll notice the Remove All Overrides button is still highlighted because an override exists, but there's no flag to back it. When you click the override button, you'll get a 404.

If I try ./ldcli dev-server remove-override --flag flag-name I get a 404 as well

The only way I can get out of this state is to restore the flag in launch darkly, sync the local server, remove the override, and then remove the flag and sync again.

[mike-zorn]

Ohhh! Interesting. We end up with a zombie override when flags are deleted and then the zombie will "poison" the remove all overrides because it'll try to remove an override for a flag that doesn't exist. I think we'll need to update the sync to prune overrides for deleted flags.