Skip to content

troubleshooting.adoc

Latest commit

 

History

History
27 lines (19 loc) · 1.15 KB

troubleshooting.adoc

File metadata and controls

27 lines (19 loc) · 1.15 KB
Raw

Troubleshooting

Init container format-namenodes fails

When creating fresh HDFS clusters, unexpected pod restarts might corrupt the initial namenode formatting. This leaves the namenode data PVC in a dangling state, where e.g. the ../current/VERSION file is created, but fsimage_xxx files are missing.

After a restart corrupted the namenode formatting, reformatting again fails due to directories and files existing. We do not want to force (override) the formatting process to avoid data loss and other implications.

Running in non-interactive mode, and data appears to exist in Storage Directory root= /stackable/data/namenode; location= null. Not formatting.

Another error message indicating a corrupt formatting state appears in the namenode main container during startup.

java.io.FileNotFoundException: No valid image files found
Warning
 The following fix should only be applied to fresh clusters. For existing clusters please consider support.

  1. Remove the PVC called data-<cluster-name>-namenode-<rolegroup>-0 for a failed namenode 0.

  2. Restart the namenode afterwards.