🐛 [BUG] - 1D Isotropic PREM fails to mesh with higher resolution topography

[lsawade]

Description

I have been playing around with higher-resolution topography and associated smoothing functions. When I use anything higher in resolution than ETOPO4 (e.g. ETOPO2, ETOPO1), something weird happens at the MPI interfaces. Since the error is caught, I have to assume that is expected. Are elements deleted/added depending on stretching?

ERROR:

Details

 Error MPI interface rank:            2
 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=     9644753 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=     9644805 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=     9644809 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=     9644813 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=     9644817 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=    10697381 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=    10697397 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=    10697413 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=    10697429 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           7
   flag=           0 missed iglob=     9644817 interface=           1 is_face_edge T

 warning            9  flag: missed rank=           7
   flag=           0 missed iglob=    10697493 interface=           1 is_face_edge T

 warning            9  flag: missed rank=           2
   flag=           0 missed iglob=    10697493 interface=           6 is_face_edge T

 Error MPI interface rank:            9
   work_test_flag min/max :           0          12
 Error: MPI points remain unrecognized, please check mesh interfaces
 Error detected, aborting MPI... proc            2
   work_test_flag min/max :           0           3
 Error: MPI points remain unrecognized, please check mesh interfaces
 Error detected, aborting MPI... proc            9

---

Added .txt otherwise it wouldn't let me upload.

Par_file.txt
output_mesher.txt
constants.h.in.txt

Affected SPECFEM3D_GLOBE version

COMMIT: f604d7c

Your software and hardware environment

GNU Fortran (GCC) 8.5.0 20210514 (Red Hat 8.5.0-20), openmpi/gcc/4.1.2, cudatoolkit/11.7, --with-cuda=cuda11, 4x Nvidia A100, MPS using 24 MPI slices and 4 GPUs

Reproduction steps

1. Download the newest specfem devel
2. Download etopo2, smooth it, 
3. Use the attached `Par_file`, topography, `constants.h.in`
4. Hopefully reproduce error

Screenshots

Error MPI interface rank:            2
 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=     9644753 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=     9644805 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=     9644809 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=     9644813 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=     9644817 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=    10697381 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=    10697397 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=    10697413 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           6
   flag=           0 missed iglob=    10697429 interface=           3 is_face_edge T

 warning            9  flag: missed rank=           7
   flag=           0 missed iglob=     9644817 interface=           1 is_face_edge T

 warning            9  flag: missed rank=           7
   flag=           0 missed iglob=    10697493 interface=           1 is_face_edge T

 warning            9  flag: missed rank=           2
   flag=           0 missed iglob=    10697493 interface=           6 is_face_edge T

 Error MPI interface rank:            9
   work_test_flag min/max :           0          12
 Error: MPI points remain unrecognized, please check mesh interfaces
 Error detected, aborting MPI... proc            2
   work_test_flag min/max :           0           3
 Error: MPI points remain unrecognized, please check mesh interfaces
 Error detected, aborting MPI... proc            9

Logs

output_mesher.txt

OS

Linux