Skip to content

general-diagnostics-nats-module-readiness-and-connectivity-8cdf63e.md

Latest commit

 

History

History
108 lines (59 loc) · 4.43 KB

general-diagnostics-nats-module-readiness-and-connectivity-8cdf63e.md

File metadata and controls

108 lines (59 loc) · 4.43 KB

General Diagnostics: NATS Module Readiness and Connectivity

Symptom

  • The NATS module is not in a Ready state.
  • The Eventing module reports that it cannot connect to the NATS backend.
  • NATS Pods are in a CrashLoopBackOff or Pending state.

Cause

Issues with the NATS module can stem from misconfigurations in the NATS custom resource (CR), problems with the underlying Kubernetes Nodes, or storage issues with Persistent Volume Claims (PVCs).

Solution

1. Check the NATS CR Status

  1. To verify the health of the NATS cluster, check the NATS CR.

    kubectl get nats -n kyma-system

  2. Look for STATE: Ready. If the state is Error or Processing, inspect the CR for detailed error messages.

    kubectl get nats {NATS_CR_NAME} -n kyma-system -o yaml

  3. Review the status.conditions to understand the root cause.

2. Check the NATS Pods

  1. Ensure all NATS Pods are running correctly.

    kubectl get pods -n kyma-system -l nats_cluster=eventing-nats

  2. If any Pods are not in the Running state, use kubectl describe pod and kubectl logs to investigate further.

3. Check the Persistent Volume Claims (PVCs)

  1. If you use file storage, a common issue is a problem with the PVCs.

    kubectl get pvc -n kyma-system -l nats_cluster=eventing-nats

  2. Check the STATUS column. If a PVC is Pending, there may be no available Persistent Volume that satisfies its request. If it is Bound, check if it is full by referring to the NATS Backend Storage Is Full guide.

4. Inspect the NATS JetStream

If the cluster appears healthy, you can inspect the JetStream components directly using the NATS CLI.

  1. Ensure that you have access to the NATS server (see Accessing the NATS Server Using CLI).

  2. Port-forward to a NATS Pod.

    kubectl -n kyma-system port-forward svc/eventing-nats 4222

  3. Verify that the kyma stream exists.

          $ nats stream ls
  ╭────────────────────────────────────────────────────────────────────────────╮
  │                                  Streams                                   │
  ├──────┬─────────────┬─────────────────────┬──────────┬───────┬──────────────┤
  │ Name │ Description │ Created             │ Messages │ Size  │ Last Message │
  ├──────┼─────────────┼─────────────────────┼──────────┼───────┼──────────────┤
  │ sap  │             │ 2022-05-03 00:00:00 │ 0        │ 318 B │ 5.80s        │
  ╰──────┴─────────────┴─────────────────────┴──────────┴───────┴──────────────╯

  4. If the stream exists, check the timestamp of the Last Message that the stream received. A recent timestamp would mean that the event was published correctly.

  5. Check if the consumers were created and have the expected configurations.

    nats consumer info

    To correlate the consumer to the Subscription and the specific event type, check the description field of the consumer.

  6. If the PVC storage is fully consumed and matches the stream size as shown above, the stream can no longer receive messages. Either increase the PVC storage size (see NATS Backend Storage Is Full) or set the MaxBytes property which removes the old messages.