Fix race condition in ProcessesSync causing SSH key mismatch

The sync could update an LRP with stale SSH route data from a previous
version when the process version changed between the initial scan and
the batch execution.

Timeline of the race condition:

  Sync                              User Request
  ----                              ------------
  |                                      |
  | fetch diego LRPs                     |
  | (gets {guid}-{vA} with SSH key A)    |
  |                                      |
  | scan CC processes                    |
  | process has version A, matches       |
  | to_update[id] = diego_lrp_A          |
  |                                      |
  |                                      | update process
  |                                      | version changes to B
  |                                      |
  |                                      | desire_app called
  |                                      | creates {guid}-{vB}
  |                                      | with new SSH key B
  |                                      |
  | re-fetch process by id               |
  | (now has version B)                  |
  |                                      |
  | update_app(process_vB, diego_lrp_A)  |
  | process_guid = {guid}-{vB}           |
  | but uses SSH route from diego_lrp_A! |
  |                                      |
  v                                      v

Result: LRP {guid}-{vB} has SSH key A in routes but SSH key B in
run_info (sshd arguments), breaking cf ssh.

Fix: Before executing desire/update, verify the process version still
matches what was recorded during the initial scan. Skip if mismatched -
the next sync cycle will handle it correctly.

Author: philippthun

lib/cloud_controller/diego/processes_sync.rb

@@ -21,15 +21,15 @@ def sync
         @bump_freshness = true
         diego_lrps = bbs_apps_client.fetch_scheduling_infos.index_by { |d| d.desired_lrp_key.process_guid }
         logger.info('fetched-scheduling-infos')
-        to_desire = []
+        to_desire = {}
         to_update = {}
         batched_processes_for_sync do |processes|
           processes.each do |process|
             process_guid = ProcessGuid.from_process(process)
             diego_lrp = diego_lrps.delete(process_guid)
 
             if diego_lrp.nil?
-              to_desire.append(process.id)
+              to_desire[process.id] = process_guid
             elsif process.updated_at.to_f.to_s != diego_lrp.annotation
               to_update[process.id] = diego_lrp
             end
@@ -87,7 +87,10 @@ def process_workpool_exceptions(exceptions)
       def update_lrps(to_update)
         batched_processes(to_update.keys) do |processes|
           processes.each do |process|
-            workpool.submit(process, to_update[process.id]) do |p, l|
+            existing_lrp = to_update[process.id]
+            next if ProcessGuid.from_process(process) != existing_lrp.desired_lrp_key.process_guid
+
+            workpool.submit(process, existing_lrp) do |p, l|
               logger.info('updating-lrp', process_guid: p.guid, app_guid: p.app_guid)
               bbs_apps_client.update_app(p, l)
               logger.info('update-lrp', process_guid: p.guid)
@@ -97,8 +100,11 @@ def update_lrps(to_update)
       end
 
       def desire_lrps(to_desire)
-        batched_processes(to_desire) do |processes|
+        batched_processes(to_desire.keys) do |processes|
           processes.each do |process|
+            desired_process_guid = to_desire[process.id]
+            next if ProcessGuid.from_process(process) != desired_process_guid
+
             workpool.submit(process) do |p|
               logger.info('desiring-lrp', process_guid: p.guid, app_guid: p.app_guid)
               bbs_apps_client.desire_app(p)