Packer Build Intermittently Erroring

[bapesta786]

Packer Version
1.15.0

Packer Plugin Version
2.1.2

Plugin Builder
vsphere-iso

vCenter Version
8.0.3

I am using Packer along with the vSphere plugin to build VM's in my production environment however I have intermittent errors where a VM build will just fail saying that the session is not authenticated. I have no issues in my test or homelab environment but I am not sure what exactly could be causing this.

I have attached a screenshot for you to see what happens. This could happen at any point in the build. It could be at the start, middle, or end and can happen after any provisioner step.

I'm at a loss with this. Has anyone seen this issue before?