Move Sensu local scripts to standard location at deployment time

[jeanml]

We are getting alerts raised by Sensu shortly after deployments that are caused by how we are handling check scripts ans Sensu client's inability to pick up check definition changes quickly enough.

For example:

1. Deployment 1 defines a check with a script called my-check.sh
2. Deployment agent creates a Sensu check definition as such:

{
  "checks" : {
    "check-name": {
      "command": "/temp/deployments/1/healthchecks/sensu/my-check.sh"
    }
  }
}

3. Check definition is picked up by Sensu client
4. Later, deployment 2 redefines the same check with the same script
5. Deployment agent overwrites the Sensu check definition with:

{
  "checks" : {
    "check-name": {
      "command": "/temp/deployments/2/healthchecks/sensu/my-check.sh"
    }
  }
}

6. Previous directory /temp/deployments/1 is deleted by deployment agent
7. Sensu client takes a long time to pick up this change and results in the check failing due to the fact that /temp/deployments/1/healthchecks/sensu/my-check.sh no longer exists on disk.

Potential solutions:

1. Deployments should automate the silencing of checks until we know for sure that Sensu client has picked up the latest definitions. How we know that? I am not sure.
2. Deployment agent can more local check scripts to the deployment location of an application, e.g. /opt/application_name/healthchecks/sensu/my-script.sh. When redeploying, we would end up overwriting the existing script to ensure that the checks currently defined still work.