[JENKINS-76064] PowerShell build step never starts and hangs indefinitely

[jenkins-infra-bot]

We will see something like the following in the console output:

[build] $ powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass -File C:\Users\user\AppData\Local\Temp\jenkins14472050587877702380.ps1

But the step will never start.

When we login/RDP to the windows agent and try to run the following, a console window opens and hangs:

powershell.exe -NoProfile -NonInteractive

And once this happens on one Jenkins agent, it seems to immediately spread to all other agents - so they all begin hanging on any PowerShell build step (even in the middle of existing ones), no matter how old each agent is.  This does not appear to be an agent imaging issue, as most agents have been successfully processing PowerShell build steps for hours before this happens.  And it does not appear to be a specific build step issue, as once this happens all PowerShell build steps cease to function.  And it does not appear to be an agent resource issue, as we have no problem connecting remotely to the agent and inspecting disk, CPU, and memory and they are plentiful.

We saw that a flaw in the Durable Task Plugin caused this or something very similar in the past.  So we just upgraded our Durable Task Plugin from v587.v84b_877235b_45 to v595.ve87b_f1318d67, but that did not fix the issue. 

The only mitigation we've found so far is to take the agent offline, abort the build, restart the agent, bring it back online, retry the build.  This also fixes PowerShell hanging when we login and try to run a PowerShell session.  We have hundreds of PowerShell build steps run every day, so this is not a viable short-term mitigation strategy.

---

Originally reported by bboss, imported from: PowerShell build step never starts and hangs indefinitely

• status: Open
• priority: Blocker
• component(s): durable-task-plugin
• resolution: Unresolved
• votes: 0
• watchers: 4
• imported: 2025-12-09

Raw content of original issue

We will see something like the following in the console output:

[build] $ powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass -File C:\Users\user\AppData\Local\Temp\jenkins14472050587877702380.ps1

But the step will never start.
When we login/RDP to the windows agent and try to run the following, a console window opens and hangs:

powershell.exe -NoProfile -NonInteractive

And once this happens on one Jenkins agent, it seems to immediately spread to all other agents - so they all begin hanging on any PowerShell build step (even in the middle of existing ones), no matter how old each agent is.  This does not appear to be an agent imaging issue, as most agents have been successfully processing PowerShell build steps for hours before this happens.  And it does not appear to be a specific build step issue, as once this happens all PowerShell build steps cease to function.  And it does not appear to be an agent resource issue, as we have no problem connecting remotely to the agent and inspecting disk, CPU, and memory and they are plentiful.
We saw that a flaw in the Durable Task Plugin caused this or something very similar in the past.  So we just upgraded our Durable Task Plugin from v587.v84b_877235b_45 to v595.ve87b_f1318d67, but that did not fix the issue. 
The only mitigation we've found so far is to take the agent offline, abort the build, restart the agent, bring it back online, retry the build.  This also fixes PowerShell hanging when we login and try to run a PowerShell session.  We have hundreds of PowerShell build steps run every day, so this is not a viable short-term mitigation strategy.

environment

Jenkins 2.492.3<br/>
Azure VM Agents 1013.v7a_2a_cd831714<br/>
Durable Task Plugin v587.v84b_877235b_45

[jenkins-infra-bot]

bboss:

• Original comment link
• Raw content of original comment:

  Mitigation Strategy Attempt
  
  We're using Azure VM Agents.  We're going to try to change the Retention Strategy from "Azure VM Idle Retention Strategy" with a retention time of 180 minutes, to the "Azure VM Once Retention Strategy" with "Shutdown instead of deleting" checked to see if that helps mitigate the issue (since the issue does not persist after a restart).
  
  We still saw locking occurring with this change, but the recovery was a little easier - just cancel the job and the retention strategy took care of the rest.
  

Mitigation Strategy Attempt

We're using Azure VM Agents.  We're going to try to change the Retention Strategy from "Azure VM Idle Retention Strategy" with a retention time of 180 minutes, to the "Azure VM Once Retention Strategy" with "Shutdown instead of deleting" checked to see if that helps mitigate the issue (since the issue does not persist after a restart).

We still saw locking occurring with this change, but the recovery was a little easier - just cancel the job and the retention strategy took care of the rest.

[jenkins-infra-bot]

bboss:

• Original comment link
• Raw content of original comment:

  Multiple Mitigations Needed to Resolve for Us - Will not work for everyone
  
  We found this was started by the need to minimize disk space usage.  We started to run out of disk space on our agents, so we introduced workspace cleanup operations at the end of each build.  We cannot explain why, but somehow this caused the PowerShell locking described above.  We confirmed that the workspace cleanup was only removing files and folders in a single job's workspace folder.  And we confirmed that what it was removing was not being tried to be archived or used in further or post-build steps.
  
  Mitigation:
  
  	
  Modify Azure VM Agents Retention strategy to "Azure VM Once Retention Strategy" and set it to "Shutdown instead of deleting"
  	
  
  		
  Make sure to use the latest Azure VM Agents and Azure SDK plug-ins.  Earlier versions had issues with bringing agents online in a timely manner.
  	
  	
  	
  Remove all workspace cleanup
  	
  
  		
  Rely on Azure's ephemeral storage to cleanup the disk on shutdown and before it was next used.  We were already using an ephemeral disk for the jenkins workspace so we didn't have to change anything to use this to our advantage.
  	
  	
  	
  Create a recurring job to remove idle agents
  
  
  

Multiple Mitigations Needed to Resolve for Us - Will not work for everyone

We found this was started by the need to minimize disk space usage.  We started to run out of disk space on our agents, so we introduced workspace cleanup operations at the end of each build.  We cannot explain why, but somehow this caused the PowerShell locking described above.  We confirmed that the workspace cleanup was only removing files and folders in a single job's workspace folder.  And we confirmed that what it was removing was not being tried to be archived or used in further or post-build steps.

Mitigation:

1. Modify Azure VM Agents Retention strategy to "Azure VM Once Retention Strategy" and set it to "Shutdown instead of deleting"
   1. Make sure to use the latest Azure VM Agents and Azure SDK plug-ins.  Earlier versions had issues with bringing agents online in a timely manner.
2. Remove all workspace cleanup
   1. Rely on Azure's ephemeral storage to cleanup the disk on shutdown and before it was next used.  We were already using an ephemeral disk for the jenkins workspace so we didn't have to change anything to use this to our advantage.
3. Create a recurring job to remove idle agents