Merge pull request #18 from togethercomputer/nikmitra/operations-scripts

Add operations folder with diagnose and remediate scripts

Author: Part-TimeWizard

scripts/operations/CONTEXT.md

@@ -0,0 +1,99 @@
+# Instant Clusters GUID ASSIGNMENT CONTEXT
+This is provides additional context on how GUIDs are assigned to tcloud instant cluster VMs.  Every tenant VM is provided with Infiniband HCA sriov virtual functions.  These HCA VFs are dynamically given GUIDs that are allocated and tracked via the ib-kubernetes daemon.  The daemon also communicates with the Nvidia Unified Fabric Manager (UFM) to register these GUIDs into pkeys for tenant isolation.
+
+# Relevant resources
+
+## configmap: cluster-operator-system/cluster-operator
+PKEY_POOL_START and PKEY_POOL_END - This is the range of tenant PKeys that can be assigned to a tenant.  We should ONLY access PKeys in this range (with the exception of DEFAULT_LIMITED_PARTITION (see ib-kubernetes-config) iff it's set).
+
+## configmap: cluster-operator-system/storage-network-metadata-v2
+linkType - Link type for storage networks.  If this is set to ipOverIb and DEFAULT_LIMITED_PARTITION is set, that pkey should support ipOverIB.
+
+
+## configmap: kube-system/ib-kubernetes-config
+DEFAULT_LIMITED_PARTITION (optional) - When this value is set, GUIDs are also added to this pkey as limited members.
+  - IMPORTANT: This pkey must already exist in UFM.  ib-kubernetes will NOT create it — if it doesn't exist, the limited-partition add will be logged as an error and skipped.
+  - GUIDs are added with membership=limited, index0=false (always).
+
+ENABLE_IP_OVER_IB - Unused, see storage-network-metadata-v2 for if DEFAULT_LIMITED_PARTITION should support ipOverIb, all other pkeys in [PKEY_POOL_START, PKEY_POOL_END] should always support ipOverIb.
+
+GUID_POOL_RANGE_START - The start of the range of GUIDs created by ib-kubernetes.  On startup the daemon syncs all GUIDs in UFM (across ALL pkeys) that fall in this range and marks them allocated to avoid collisions.
+
+GUID_POOL_RANGE_END - The end of the range of GUIDs created by ib-kubernetes.
+
+ENABLE_INDEX0_FOR_PRIMARY_PKEY - Whether or not to set index0=true when adding GUIDs to the primary pkey.  Default: true.
+  - This is a per-GUID membership attribute in UFM (sent as the "index0" field in the POST to /ufmRest/resources/pkeys), NOT a per-pkey attribute.
+  - It does NOT retroactively affect GUIDs that were already added before the setting changed.
+
+DAEMON_PERIODIC_UPDATE - How often (in seconds) the daemon runs its add and delete cycles.  Default: 5.
+
+SM_TIMEZONE - IANA timezone of the UFM server's local clock (e.g. "America/Los_Angeles", "UTC").  ib-kubernetes uses this to correctly parse UFM timestamps when deciding whether a previous pkey modification has completed before issuing the next one.  Default: UTC.
+
+## secret: kube-system/ib-kubernetes-ufm-secret
+UFM_ADDRESS - The address of UFM to make requests against
+
+UFM_HTTP_SCHEMA - Schema to make requests with
+
+UFM_PASSWORD - Password for UFM.  Needed for making requests
+
+UFM_PORT (optional) - Port to make requests on. (Can be blank)
+
+UFM_USERNAME - Username for UFM.  Needed for making requests
+
+## pod annotations: k8s.v1.cni.cncf.io/networks
+The `k8s.v1.cni.cncf.io/networks` annotation holds a JSON array of NetworkSelectionElement objects, one per requested network.  For an ib-sriov network, the relevant element carries:
+  - `infinibandGUID` field (if the NAD spec has `capabilities: {infinibandGUID: true}`): the GUID allocated for this pod+network.
+  - `cni-args.guid` field (otherwise): the GUID allocated for this pod+network.
+  - `cni-args.mellanox.infiniband.app: "configured"`: written ONLY after the GUID has been successfully registered in UFM AND the pod annotation has been persisted to k8s.  A network element that has a GUID but lacks this flag is in an intermediate state — the GUID may or may not be in UFM yet.
+
+The daemon only enqueues a pod for add-processing if the network element does NOT yet have `mellanox.infiniband.app: configured`.  It only enqueues a pod for delete-processing if the network element DOES have that flag.
+
+## pod finalizer: ufm.together.ai/pod-guid-cleanup-protection
+Added to the pod BEFORE the GUID is registered in UFM (as soon as the GUID is allocated in-memory).  Removed AFTER the GUID has been successfully removed from UFM.  While this finalizer exists, k8s will not allow the pod to be fully deleted.
+
+## net-attach-def finalizer: ufm.together.ai/guid-cleanup-protection
+Added to the NetworkAttachmentDefinition(also refered to as net-attach-def or NAD) after GUIDs are successfully added to UFM for that network.  Removed only after all pods using that NAD have had their GUIDs cleaned up from UFM.  Prevents the NAD from being deleted while pods still hold GUIDs in UFM — which matters because the NAD is required during the delete path (the daemon reads the NAD to find the pkey for removal).
+
+## net-attach-def: .spec.config.pkey
+Pkey - The tenant pkey this network is a part of.  This can be linked to the virt-launcher pods by the same namespace.
+
+# GUID lifecycle — add path
+1. Pod is scheduled and has a `k8s.v1.cni.cncf.io/networks` annotation referencing an ib-sriov NAD.
+2. ib-kubernetes allocates a GUID from [GUID_POOL_RANGE_START, GUID_POOL_RANGE_END] and writes it into the pod's annotation element (as `infinibandGUID` or `cni-args.guid` depending on NAD capabilities).
+3. Pod finalizer `ufm.together.ai/pod-guid-cleanup-protection` is added.
+4. GUID is registered in UFM: added to the tenant pkey as a full member, and to DEFAULT_LIMITED_PARTITION as a limited member (if configured).
+5. NAD finalizer `ufm.together.ai/guid-cleanup-protection` is added to the NAD.
+6. `mellanox.infiniband.app: configured` is written to the pod's annotation element.
+
+# GUID lifecycle — delete path
+1. Pod's DeletionTimestamp is set (pod enters Terminating).
+2. ib-kubernetes reads the pod's annotation, finds networks with `mellanox.infiniband.app: configured`, and enqueues them.
+3. For each network, the daemon looks up the NAD to find the pkey.  If the NAD is gone, the entry is dropped from the queue and the pod gets stuck (the NAD finalizer is meant to prevent this).
+4. GUIDs are removed from UFM: from the tenant pkey and from DEFAULT_LIMITED_PARTITION (if configured).
+5. Pod finalizer is removed → k8s can fully delete the pod.
+6. If no other pods are still using the NAD (checked by scanning all pods for that network), the NAD finalizer is removed.
+
+# Assertions
+These are all true statements in a properly functioning system.  They have been broken down into two sections based on the severity of if the assertion is not true.
+
+## Critical
+If these are not true, the system will not work correctly or will degrade into a state in which it will not work correctly.  These need to be resolved immediately.
+- For all pods where an IB network element has `mellanox.infiniband.app: configured` set and the `ufm.together.ai/pod-guid-cleanup-protection` is set, the GUID in that element MUST be present in UFM in the pkey specified by the net-attach-def in the same namespace as a FULL member.
+- If DEFAULT_LIMITED_PARTITION is set, for all pods where an IB network element has `mellanox.infiniband.app: configured` set and the `ufm.together.ai/pod-guid-cleanup-protection` is set, the GUID MUST also be present in UFM in DEFAULT_LIMITED_PARTITION as a LIMITED member.
+- If ENABLE_INDEX0_FOR_PRIMARY_PKEY is true, all GUIDs added to the tenant pkey should have index0=true in their UFM membership record.
+- All partitions from [PKEY_POOL_START, PKEY_POOL_END] must have "ip_over_ib": true in UFM.  If linkType in storage-network-metadata-v2 is ipOverIb and DEFAULT_LIMITED_PARTITION is set, DEFAULT_LIMITED_PARTITION must also have "ip_over_ib": true in UFM.
+    - Modifications to ip_over_ib state can't be done by ib-kubernetes, the remediate script, nor simple UI actions and should be done by substrate SRE.
+
+## Warning
+These are sub-optimal, but do not indicate the system will work incorrectly.  Additionally, they will not cause the system to degrade into a critical state.
+- If a pod is stuck in terminating for >5min and still has the ufm.together.ai/pod-guid-cleanup-protection finalizer, this pod is considered hung and will not clean up properly without manual intervention.
+    - If there are no net-attach-def objects in the same namespace (they have all been cleaned up without the finalizer being respected), the pod is considered hung as there is no chance it will be cleaned up properly without manual intervention.  NOTE: in a healthy system the NAD finalizer (ufm.together.ai/guid-cleanup-protection) should prevent NAD deletion while pods are still using it.
+    - A pod stuck terminating can be considered critical if the full deletion of the pod is required for a critical workflow. i.e. if a tenant cluster is being created, virt-launcher pods are created then deleted, then a cdi-uploader pod is created, then a second virt-launcher pod is created.  If the first pod is hung, the VM will never be created properly.
+- For all pkeys between the range PKEY_POOL_START and PKEY_POOL_END, for all guids between the range GUID_POOL_RANGE_START and GUID_POOL_RANGE_END, the guids must be present in the k8s.v1.cni.cncf.io/networks infiniband-guid annotation of a running pod.
+    - This mapping should be 1 to 1.  GUIDs in UFM but not in k8s are considered warnings, GUIDs in k8s but not in UFM is considered critical.
+
+## Intermediate / edge cases (not immediately actionable)
+These are miscelaneous edge cases that may cause a degraded system, but they are not as easily tracked mapped into assertions.  Deeper log investigation would be required to confirm these are the root cause and the solutions are not as simple and likely will need to be done by a member of the tcloud team.
+- A pod network element may have a GUID but NOT yet have `mellanox.infiniband.app: configured`.  This means the daemon allocated the GUID but has not yet persisted the "configured" flag (the UFM add and/or annotation write is still in flight or failed).  If the pod is deleted in this window, its GUID in UFM may need manual cleanup — ib-kubernetes will not enqueue it for the delete path since the network is not marked as configured.
+- ib-kubernetes syncs the GUID pool on startup from ALL pkeys in UFM (not just tenant pkeys).  Any GUID within [GUID_POOL_RANGE_START, GUID_POOL_RANGE_END] found in any UFM pkey will be marked as allocated in the pool, even if there is no corresponding k8s pod annotation.
+- If SM_TIMEZONE is misconfigured (or not set), the daemon's pkey-modification rate-limiting (canProceedWithPkeyModification) may behave incorrectly: it could skip cycles unnecessarily (if SM time appears ahead of local time) or issue duplicate modifications (if SM time appears behind).