fix: slow workspace list refresh when many workspaces are stopped#328
Merged
Conversation
A customer reported that refreshing the workspace list could take around ten seconds. The client logs showed that most of that time was spent expanding the auth header command: stopped workspaces do not include their resources in the /workspaces response, so we were issuing a separate resource lookup for each stopped workspace just to recover its agents, and every one of those HTTP calls paid the header command expansion cost (roughly half a second per call — about 9.7 seconds for the 21 stopped workspaces in the customer's search). Instead of trying to make those calls faster, we questioned why they were needed at all. Agents only matter when there is something to connect to, and nothing can connect to a stopped workspace: the SSH configuration is only useful for running workspaces, and the agent list arrives for free in the /workspaces response the moment a workspace is running. So stopped (and otherwise not-running) workspaces are now registered as workspace-only environments, identified by workspace name alone, with no resource lookup at all. When a workspace comes up, the poller replaces the workspace-only row with one row per agent and regenerates the SSH configuration, which now covers running workspaces only. While reviewing the features that interact with stopped workspaces I found and fixed three problems this change would have introduced: - deep links (URI handling) became racy: the agent environment and its SSH config entry now exist only after the poller observes the running workspace, so a fixed two-second delay was no longer enough. The protocol handler now nudges the poller and waits for the environment to actually appear (with a timeout) before showing the environment page and launching the IDE. - an environment without an agent used to hand Toolbox an SSH contents view with an empty host. It now returns a manual contents view with empty IDE and project lists, so Toolbox never attempts to connect to a blank host. - Toolbox closes environments that disappear from the provider list by cancelling their wrapper scope without ever firing the AfterDisconnectHook. This one took a lot of time to figure out and was only possible by verifying against the Toolbox bytecode. Since stop/start transitions now replace environment instances, a connected environment leaked its network metrics poll job. The provider now disposes dropped instances explicitly, both during poll refreshes and on close/logout. Two known issues remain and I think we can consider them acceptable: (1) the environment id changes across a stop/start cycle, so an environment page left open while the workspace stops can go throw a weird error about Toolbox not being able to find it; (2) a workspace that goes from running and ssh connected to stopped and then back to running, no longer auto-connects - resolves DEVEX-372 - resolves DEVEX-374
The reason was never reported to the user.
code-asher
approved these changes
Jul 6, 2026
code-asher
left a comment
code-asher
left a comment
Member
There was a problem hiding this comment.
Looks good to me!
I remember we also discussed getting rid of the wildcard toggle setting and always using wildcard, are we still going to do that?
|
|
||
| val resources = ws.latestBuild.resources.ifEmpty { | ||
| client.resources(ws) | ||
| } |
Member
There was a problem hiding this comment.
I think we can get rid of this, we should always have resources for a running workspace.
Collaborator
Author
I'll have to confirm that with Atif. But we still have to support older versions of Coder that don't have wildcard support. In those cases we will have to use coder vscodessh, but then we get into other issues 🤔 |
Member
Ah yeah good point, we would have to bump the minimum supported version to get rid of it entirely. 😢 I think using |
jeremyruppel
reviewed
Jul 6, 2026
running workspaces should already include the agents.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A customer reported that refreshing the workspace list could take around ten seconds. The client logs showed that most of that time was spent expanding the auth header command: stopped workspaces do not include their resources in the /workspaces response, so we were issuing a separate resource lookup for each stopped workspace just to recover its agents, and every one of those HTTP calls paid the header command expansion cost (roughly half a second per call — about 9.7 seconds for the 21 stopped workspaces in the customer's search).
Instead of trying to make those calls faster, we questioned why they were needed at all. Agents only matter when there is something to connect to, and nothing can connect to a stopped workspace: the SSH configuration is only useful for running workspaces, and the agent list arrives for free in the /workspaces response the moment a workspace is running. So stopped (and otherwise not-running) workspaces are now registered as workspace-only environments, identified by workspace name alone, with no resource lookup at all. When a workspace comes up, the poller replaces the workspace-only row with one row per agent and regenerates the SSH configuration, which now covers running workspaces only.
While reviewing the features that interact with stopped workspaces I found and fixed three problems this change would have introduced:
deep links (URI handling) became racy: the agent environment and its SSH config entry now exist only after the poller observes the running workspace, so a fixed two-second delay was no longer enough. The protocol handler now nudges the poller and waits for the environment to actually appear (with a timeout) before showing the environment page and launching the IDE.
an environment without an agent used to hand Toolbox an SSH contents view with an empty host. It now returns a manual contents view with empty IDE and project lists, so Toolbox never attempts to connect to a blank host.
Toolbox closes environments that disappear from the provider list by cancelling their wrapper scope without ever firing the AfterDisconnectHook. This one took a lot of time to figure out and was only possible by verifying against the Toolbox bytecode. Since stop/start transitions now replace environment instances, a connected environment leaked its network metrics poll job. The provider now disposes dropped instances explicitly, both during poll refreshes and on close/logout.
Two known issues remain and I think we can consider them acceptable:
(1) the environment id changes across a stop/start cycle, so an environment page left open while the workspace stops can go throw a weird error about Toolbox not being able to find it;
(2) a workspace that goes from running and ssh connected to stopped and then back to running, no longer auto-connects