Connector upstream-status annotation not re-mirrored after Ready condition flips True

[drewr]

Summary

When a tunnel connector's Ready condition transitions False → True (heartbeat renews the lease), the downstream connector's networking.datumapis.com/upstream-status annotation is not updated. The extension server reads this stale annotation and classifies the connector as offline, causing persistent HTTP 503 responses.

Sequence

1. Connector created — replicator mirrors status into the downstream annotation. At this moment Ready: False (lease not yet renewed).
2. Heartbeat connects — upstream connector becomes Ready: True, connectionDetails populated.
3. Replicator does not re-reconcile — skipUpstreamStatusSync: true suppresses downstream→upstream sync, and there is no watch on upstream status changes to re-trigger the mirror path.
4. Downstream connector's annotation still contains Ready: False with stale connectionDetails.
5. Extension server reads the annotation, sees Ready: False, marks all routes for that connector as offline → HTTP 503.

Evidence

Downstream connector annotation captured while upstream connector was Ready: True:

{
  "conditions": [
    { "type": "Ready", "status": "False", "reason": "ConnectorNotReady",
      "message": "Connector lease has expired. Agent may be offline." }
  ],
  "connectionDetails": { "publicKey": { "id": "378843c806c8c93c5770abaa19bc47e04e9f56977c6e7cc28044a09ef5a1cd23", ... } }
}

Upstream connector at the same moment:

{ "type": "Ready", "status": "True", "reason": "ConnectorReady",
  "message": "The connector is ready to tunnel traffic." }

Extension server log confirms all vhosts for this connector remain offline:

connector_offline_routes:26  clusters_replaced:1

Root Cause

replicationResourceConfig for the Connector type sets:

mirrorStatusToAnnotation: true,
skipUpstreamStatusSync:   true,

The replicator reconciles on spec changes but has no watch/enqueue path for upstream status changes. When Ready flips after the initial replication, nothing re-queues the replicator to update the annotation.

touchDownstreamGatewayAnnotations in the connector controller fires to trigger EG re-translation, but the stale annotation means the extension server still classifies the connector as offline even after re-translation.

Fix

The replicator should watch upstream connector status changes and re-mirror the annotation when status changes. Alternatively, the connector controller should enqueue a replicator reconcile after writing Ready: True.

[drewr]

Workaround implemented in the datumctl plugin

The CLI plugin (connect/connect-lib) works around this by calling a refresh_connection_details() method immediately after the connector's Ready condition becomes True (i.e., after progress monitoring completes). This does a second patch_status of connectionDetails on the upstream connector, which triggers a spec change that causes the replicator to re-reconcile and re-mirror the annotation with the current Ready:True state, which in turn triggers EG re-translation.

Why the desktop app doesn't hit this

The desktop app (app/lib) has HeartbeatAgent continuously re-patching connectionDetails on every lease renewal cycle (~15s). Each re-patch is a spec change on the upstream connector, which keeps the replicator running and the downstream annotation current. So even if the initial mirror captures Ready:False, the next heartbeat cycle triggers a fresh replication with Ready:True. The CLI is a single-shot process and doesn't do continuous re-patching, so it had to add the explicit post-ready refresh.

Is the workaround safe once #209 is fixed?

Yes. After the fix, the replicator will re-mirror automatically on Ready transitions, so the extra patch_status call will trigger a replicator reconcile that finds the annotation already correct and produces no downstream change. It's a harmless no-op and can be removed at that point, but leaving it in does no harm.

Why the workaround is not ideal

The re-patch races with the same window: if the replicator mirrors the annotation between the patch_status call and the moment Ready:True is written upstream, the annotation can still be captured as Ready:False. In practice this window is very small, but it's not zero. The proper fix (replicator watching upstream status changes) closes the race entirely.