bug? `test_bodo_nan` in `tests/integration/test_reads.py` hangs locally

[kevinjqliu]

Apache Iceberg version

None

Please describe the bug 🐞

Related to #2167

When im running integration tests locally on my laptop make integration
it always get stuck on

tests/integration/test_reads.py::test_bodo_nan[session_catalog_hive] 

not really sure what the problem is. im using python 3.12.11

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[kevinjqliu]

@ehsantn would you have any ideas?

[ehsantn]

@kevinjqliu Thanks for opening the issue. Looking at it right now.

[ehsantn]

@kevinjqliu I can't reproduce the hang on my local Mac (Arm). I'm on latest main (fb78904714e47e8ca1d281bc4a4796347faa3310) and Python 3.12. Could you share more about your setup? Has anyone else seen this?

[IsaacWarren]

I tried to reproduce this on my arm mac as well and was unable to as well. I did have a hang on another test, tests/integration/test_inspect_table.py::test_inspect_partitions_partitioned, it had a user warning:

UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but 
has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation.

[geruh]

Hey @ehsantn and @IsaacWarren! I tried this out on both 3.11 and 3.12 and it also hangs for me. Maybe we have some misconfiguration or a block somewhere? Also, I tried this on my other computer and saw no issue...

I'm running this: poetry run pytest tests/integration/test_reads.py::test_bodo_nan -m integration -vv -s -x

output:

platform darwin -- Python 3.11.12, pytest-7.4.4, pluggy-1.6.0 -- xxx
cachedir: .pytest_cache
rootdir: xxx
configfile: pyproject.toml
plugins: checkdocs-2.13.0, mock-3.14.1, lazy-fixture-0.6.3, requests-mock-1.12.1
collected 2 items                                                                   

tests/integration/test_reads.py::test_bodo_nan[session_catalog_hive] Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
0   libpmpi.12.dylib                    0x0000000119854cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x00000001197b5b18 MPIR_Assert_fail + 48

[kevinjqliu]

yep i can reproduce that too, thanks!

python 3.12, MBP, with orbstack

➜  iceberg-python git:(main) poetry run pytest tests/integration/test_reads.py::test_bodo_nan -m integration -vv -s -x


====================================================================== test session starts ======================================================================
platform darwin -- Python 3.12.11, pytest-7.4.4, pluggy-1.6.0 -- /Users/kevinliu/Library/Caches/pypoetry/virtualenvs/pyiceberg-Is5Rt7Ah-py3.12/bin/python
cachedir: .pytest_cache
rootdir: /Users/kevinliu/repos/iceberg-python
configfile: pyproject.toml
plugins: checkdocs-2.13.0, mock-3.14.1, lazy-fixture-0.6.3, requests-mock-1.12.1
collected 2 items                                                                                                                                               

tests/integration/test_reads.py::test_bodo_nan[session_catalog_hive] Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
Assertion failed in file /mpich-4.1.3/src/mpid/ch4/src/ch4_init.c at line 565: MPIR_Process.comm_parent != NULL
0   libpmpi.12.dylib                    0x0000000126df4cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x0000000126d55b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x0000000126d89654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x0000000126d1be94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x0000000126d1ba28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x0000000122206eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x00000001468d372c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x0000000146889c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x00000001035bd3c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x00000001035bcb3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x00000001034ca980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x0000000103582e24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x000000010347d72c object_vacall + 312
13  libpython3.12.dylib                 0x000000010347d558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x00000001035ba680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x0000000103580ccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x000000010357dc04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x000000010357b4f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x00000001034caa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x0000000103582e24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x000000010347d72c object_vacall + 312
21  libpython3.12.dylib                 0x000000010347d558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x00000001035ba680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x000000010357a348 builtin___import__ + 276
24  libpython3.12.dylib                 0x00000001034caa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x0000000103582e24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x000000010347d72c object_vacall + 312
27  libpython3.12.dylib                 0x000000010347d558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x00000001035ba680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x000000010357a348 builtin___import__ + 276
30  libpython3.12.dylib                 0x00000001034caa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x000000010347bf74 PyObject_Vectorcall + 88
Abort(1) on node 1: Internal error
0   libpmpi.12.dylib                    0x0000000146368cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x00000001462c9b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x00000001462fd654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x000000014628fe94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x000000014628fa28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x0000000133b86eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x000000015322b72c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x00000001531e1c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x0000000104db13c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x0000000104db0b3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x0000000104cbe980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x0000000104d76e24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x0000000104c7172c object_vacall + 312
13  libpython3.12.dylib                 0x0000000104c71558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x0000000104dae680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x0000000104d74ccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x0000000104d71c04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x0000000104d6f4f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x0000000104cbea4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x0000000104d76e24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x0000000104c7172c object_vacall + 312
21  libpython3.12.dylib                 0x0000000104c71558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x0000000104dae680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x0000000104d6e348 builtin___import__ + 276
24  libpython3.12.dylib                 0x0000000104cbea4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x0000000104d76e24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x0000000104c7172c object_vacall + 312
27  libpython3.12.dylib                 0x0000000104c71558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x0000000104dae680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x0000000104d6e348 builtin___import__ + 276
30  libpython3.12.dylib                 0x0000000104cbea4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x0000000104c6ff74 PyObject_Vectorcall + 88
Abort(1) on node 2: Internal error
0   libpmpi.12.dylib                    0x0000000120368cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x00000001202c9b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x00000001202fd654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x000000012028fe94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x000000012028fa28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x0000000111806eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x000000016b22b72c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x000000016b1e1c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x000000010126d3c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x000000010126cb3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x000000010117a980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x0000000101232e24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x000000010112d72c object_vacall + 312
13  libpython3.12.dylib                 0x000000010112d558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x000000010126a680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x0000000101230ccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x000000010122dc04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x000000010122b4f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x000000010117aa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x0000000101232e24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x000000010112d72c object_vacall + 312
21  libpython3.12.dylib                 0x000000010112d558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x000000010126a680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x000000010122a348 builtin___import__ + 276
24  libpython3.12.dylib                 0x000000010117aa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x0000000101232e24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x000000010112d72c object_vacall + 312
27  libpython3.12.dylib                 0x000000010112d558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x000000010126a680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x000000010122a348 builtin___import__ + 276
30  libpython3.12.dylib                 0x000000010117aa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x000000010112bf74 PyObject_Vectorcall + 88
Abort(1) on node 4: Internal error
0   libpmpi.12.dylib                    0x0000000125980cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x00000001258e1b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x0000000125915654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x00000001258a7e94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x00000001258a7a28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x0000000124206eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x00000001311d372c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x0000000131189c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x00000001012713c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x0000000101270b3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x000000010117e980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x0000000101236e24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x000000010113172c object_vacall + 312
13  libpython3.12.dylib                 0x0000000101131558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x000000010126e680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x0000000101234ccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x0000000101231c04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x000000010122f4f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x000000010117ea4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x0000000101236e24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x000000010113172c object_vacall + 312
21  libpython3.12.dylib                 0x0000000101131558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x000000010126e680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x000000010122e348 builtin___import__ + 276
24  libpython3.12.dylib                 0x000000010117ea4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x0000000101236e24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x000000010113172c object_vacall + 312
27  libpython3.12.dylib                 0x0000000101131558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x000000010126e680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x000000010122e348 builtin___import__ + 276
30  libpython3.12.dylib                 0x000000010117ea4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x000000010112ff74 PyObject_Vectorcall + 88
Abort(1) on node 5: Internal error
0   libpmpi.12.dylib                    0x0000000116df4cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x0000000116d55b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x0000000116d89654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x0000000116d1be94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x0000000116d1ba28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x0000000114686eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x00000001358d372c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x0000000135889c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x00000001054253c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x0000000105424b3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x0000000105332980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x00000001053eae24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x00000001052e572c object_vacall + 312
13  libpython3.12.dylib                 0x00000001052e5558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x0000000105422680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x00000001053e8ccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x00000001053e5c04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x00000001053e34f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x0000000105332a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x00000001053eae24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x00000001052e572c object_vacall + 312
21  libpython3.12.dylib                 0x00000001052e5558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x0000000105422680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x00000001053e2348 builtin___import__ + 276
24  libpython3.12.dylib                 0x0000000105332a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x00000001053eae24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x00000001052e572c object_vacall + 312
27  libpython3.12.dylib                 0x00000001052e5558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x0000000105422680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x00000001053e2348 builtin___import__ + 276
30  libpython3.12.dylib                 0x0000000105332a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x00000001052e3f74 PyObject_Vectorcall + 88
Abort(1) on node 6: Internal error
0   libpmpi.12.dylib                    0x000000011b5f4cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x000000011b555b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x000000011b589654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x000000011b51be94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x000000011b51ba28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x000000010f486eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x000000011da2b72c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x000000011d9e1c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x0000000104ca93c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x0000000104ca8b3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x0000000104bb6980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x0000000104c6ee24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x0000000104b6972c object_vacall + 312
13  libpython3.12.dylib                 0x0000000104b69558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x0000000104ca6680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x0000000104c6cccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x0000000104c69c04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x0000000104c674f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x0000000104bb6a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x0000000104c6ee24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x0000000104b6972c object_vacall + 312
21  libpython3.12.dylib                 0x0000000104b69558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x0000000104ca6680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x0000000104c66348 builtin___import__ + 276
24  libpython3.12.dylib                 0x0000000104bb6a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x0000000104c6ee24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x0000000104b6972c object_vacall + 312
27  libpython3.12.dylib                 0x0000000104b69558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x0000000104ca6680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x0000000104c66348 builtin___import__ + 276
30  libpython3.12.dylib                 0x0000000104bb6a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x0000000104b67f74 PyObject_Vectorcall + 88
Abort(1) on node 3: Internal error
0   libpmpi.12.dylib                    0x0000000125b68cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x0000000125ac9b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x0000000125afd654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x0000000125a8fe94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x0000000125a8fa28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x0000000107d06eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x00000001310d372c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x0000000131089c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x00000001012393c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x0000000101238b3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x0000000101146980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x00000001011fee24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x00000001010f972c object_vacall + 312
13  libpython3.12.dylib                 0x00000001010f9558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x0000000101236680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x00000001011fcccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x00000001011f9c04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x00000001011f74f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x0000000101146a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x00000001011fee24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x00000001010f972c object_vacall + 312
21  libpython3.12.dylib                 0x00000001010f9558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x0000000101236680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x00000001011f6348 builtin___import__ + 276
24  libpython3.12.dylib                 0x0000000101146a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x00000001011fee24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x00000001010f972c object_vacall + 312
27  libpython3.12.dylib                 0x00000001010f9558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x0000000101236680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x00000001011f6348 builtin___import__ + 276
30  libpython3.12.dylib                 0x0000000101146a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x00000001010f7f74 PyObject_Vectorcall + 88
0   libpmpi.12.dylib                    0x00000001475f4cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x0000000147555b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x0000000147589654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x000000014751be94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x000000014751ba28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x0000000142606eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x000000016aa2b72c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x000000016a9e1c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x0000000103c053c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x0000000103c04b3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x0000000103b12980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x0000000103bcae24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x0000000103ac572c object_vacall + 312
13  libpython3.12.dylib                 0x0000000103ac5558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x0000000103c02680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x0000000103bc8ccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x0000000103bc5c04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x0000000103bc34f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x0000000103b12a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x0000000103bcae24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x0000000103ac572c object_vacall + 312
21  libpython3.12.dylib                 0x0000000103ac5558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x0000000103c02680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x0000000103bc2348 builtin___import__ + 276
24  libpython3.12.dylib                 0x0000000103b12a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x0000000103bcae24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x0000000103ac572c object_vacall + 312
27  libpython3.12.dylib                 0x0000000103ac5558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x0000000103c02680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x0000000103bc2348 builtin___import__ + 276
30  libpython3.12.dylib                 0x0000000103b12a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x0000000103ac3f74 PyObject_Vectorcall + 88
Abort(1) on node 10: Internal error
Abort(1) on node 12: Internal error
0   libpmpi.12.dylib                    0x000000010ecf4cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x000000010ec55b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x000000010ec89654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x000000010ec1be94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x000000010ec1ba28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x000000010cc86eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x000000012b22b72c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x000000012b1e1c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x0000000102cfd3c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x0000000102cfcb3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x0000000102c0a980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x0000000102cc2e24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x0000000102bbd72c object_vacall + 312
13  libpython3.12.dylib                 0x0000000102bbd558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x0000000102cfa680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x0000000102cc0ccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x0000000102cbdc04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x0000000102cbb4f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x0000000102c0aa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x0000000102cc2e24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x0000000102bbd72c object_vacall + 312
21  libpython3.12.dylib                 0x0000000102bbd558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x0000000102cfa680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x0000000102cba348 builtin___import__ + 276
24  libpython3.12.dylib                 0x0000000102c0aa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x0000000102cc2e24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x0000000102bbd72c object_vacall + 312
27  libpython3.12.dylib                 0x0000000102bbd558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x0000000102cfa680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x0000000102cba348 builtin___import__ + 276
30  libpython3.12.dylib                 0x0000000102c0aa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x0000000102bbbf74 PyObject_Vectorcall + 88
Abort(1) on node 9: Internal error
0   libpmpi.12.dylib                    0x000000012a680cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x000000012a5e1b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x000000012a615654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x000000012a5a7e94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x000000012a5a7a28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x0000000128f06eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x00000001390d372c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x0000000139089c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x00000001055bd3c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x00000001055bcb3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x00000001054ca980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x0000000105582e24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x000000010547d72c object_vacall + 312
13  libpython3.12.dylib                 0x000000010547d558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x00000001055ba680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x0000000105580ccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x000000010557dc04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x000000010557b4f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x00000001054caa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x0000000105582e24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x000000010547d72c object_vacall + 312
21  libpython3.12.dylib                 0x000000010547d558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x00000001055ba680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x000000010557a348 builtin___import__ + 276
24  libpython3.12.dylib                 0x00000001054caa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x0000000105582e24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x000000010547d72c object_vacall + 312
27  libpython3.12.dylib                 0x000000010547d558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x00000001055ba680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x000000010557a348 builtin___import__ + 276
30  libpython3.12.dylib                 0x00000001054caa4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x000000010547bf74 PyObject_Vectorcall + 88
Abort(1) on node 8: Internal error
0   libpmpi.12.dylib                    0x0000000130368cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x00000001302c9b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x00000001302fd654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x000000013028fe94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x000000013028fa28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x0000000111c86eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x00000001360d372c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x0000000136089c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x0000000102f193c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x0000000102f18b3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x0000000102e26980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x0000000102edee24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x0000000102dd972c object_vacall + 312
13  libpython3.12.dylib                 0x0000000102dd9558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x0000000102f16680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x0000000102edcccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x0000000102ed9c04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x0000000102ed74f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x0000000102e26a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x0000000102edee24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x0000000102dd972c object_vacall + 312
21  libpython3.12.dylib                 0x0000000102dd9558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x0000000102f16680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x0000000102ed6348 builtin___import__ + 276
24  libpython3.12.dylib                 0x0000000102e26a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x0000000102edee24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x0000000102dd972c object_vacall + 312
27  libpython3.12.dylib                 0x0000000102dd9558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x0000000102f16680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x0000000102ed6348 builtin___import__ + 276
30  libpython3.12.dylib                 0x0000000102e26a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x0000000102dd7f74 PyObject_Vectorcall + 88
Abort(1) on node 13: Internal error
0   libpmpi.12.dylib                    0x000000010de80cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x000000010dde1b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x000000010de15654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x000000010dda7e94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x000000010dda7a28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x000000010c706eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x000000011a8d372c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x000000011a889c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x00000001017593c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x0000000101758b3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x0000000101666980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x000000010171ee24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x000000010161972c object_vacall + 312
13  libpython3.12.dylib                 0x0000000101619558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x0000000101756680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x000000010171cccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x0000000101719c04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x00000001017174f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x0000000101666a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x000000010171ee24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x000000010161972c object_vacall + 312
21  libpython3.12.dylib                 0x0000000101619558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x0000000101756680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x0000000101716348 builtin___import__ + 276
24  libpython3.12.dylib                 0x0000000101666a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x000000010171ee24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x000000010161972c object_vacall + 312
0   libpmpi.12.dylib                    0x0000000124fbccc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x0000000124f1db18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x0000000124f51654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x0000000124ee3e94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x0000000124ee3a28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x0000000117006eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x00000001340d372c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x0000000134089c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x0000000102cc53c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x0000000102cc4b3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x0000000102bd2980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x0000000102c8ae24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x0000000102b8572c object_vacall + 312
13  libpython3.12.dylib                 0x0000000102b85558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x0000000102cc2680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x0000000102c88ccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x0000000102c85c04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x0000000102c834f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x0000000102bd2a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x0000000102c8ae24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x0000000102b8572c object_vacall + 312
21  libpython3.12.dylib                 0x0000000102b85558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x0000000102cc2680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x0000000102c82348 builtin___import__ + 276
24  libpython3.12.dylib                 0x0000000102bd2a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x0000000102c8ae24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x0000000102b8572c object_vacall + 312
27  libpython3.12.dylib                 0x0000000102b85558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x0000000102cc2680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x0000000102c82348 builtin___import__ + 276
30  libpython3.12.dylib                 0x0000000102bd2a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x0000000102b83f74 PyObject_Vectorcall + 88
Abort(1) on node 7: Internal error
27  libpython3.12.dylib                 0x0000000101619558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x0000000101756680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x0000000101716348 builtin___import__ + 276
30  libpython3.12.dylib                 0x0000000101666a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x0000000101617f74 PyObject_Vectorcall + 88
Abort(1) on node 11: Internal error
0   libpmpi.12.dylib                    0x00000001435f4cc4 MPL_backtrace_show + 52
1   libpmpi.12.dylib                    0x0000000143555b18 MPIR_Assert_fail + 48
2   libpmpi.12.dylib                    0x0000000143589654 MPID_InitCompleted + 240
3   libpmpi.12.dylib                    0x000000014351be94 MPII_Init_thread + 1076
4   libpmpi.12.dylib                    0x000000014351ba28 MPIR_Init_impl + 180
5   libmpi.12.dylib                     0x0000000141286eac MPI_Init + 100
6   ext.cpython-312-darwin.so           0x0000000145a2b72c PyInit_hdist + 96
7   ext.cpython-312-darwin.so           0x00000001459e1c50 PyInit_ext + 56
8   libpython3.12.dylib                 0x00000001055953c4 _PyImport_LoadDynamicModuleWithSpec + 712
9   libpython3.12.dylib                 0x0000000105594b3c _imp_create_dynamic + 380
10  libpython3.12.dylib                 0x00000001054a2980 cfunction_vectorcall_FASTCALL + 280
11  libpython3.12.dylib                 0x000000010555ae24 _PyEval_EvalFrameDefault + 20424
12  libpython3.12.dylib                 0x000000010545572c object_vacall + 312
13  libpython3.12.dylib                 0x0000000105455558 PyObject_CallMethodObjArgs + 104
14  libpython3.12.dylib                 0x0000000105592680 PyImport_ImportModuleLevelObject + 1972
15  libpython3.12.dylib                 0x0000000105558ccc _PyEval_EvalFrameDefault + 11888
16  libpython3.12.dylib                 0x0000000105555c04 PyEval_EvalCode + 252
17  libpython3.12.dylib                 0x00000001055534f0 builtin_exec + 1156
18  libpython3.12.dylib                 0x00000001054a2a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
19  libpython3.12.dylib                 0x000000010555ae24 _PyEval_EvalFrameDefault + 20424
20  libpython3.12.dylib                 0x000000010545572c object_vacall + 312
21  libpython3.12.dylib                 0x0000000105455558 PyObject_CallMethodObjArgs + 104
22  libpython3.12.dylib                 0x0000000105592680 PyImport_ImportModuleLevelObject + 1972
23  libpython3.12.dylib                 0x0000000105552348 builtin___import__ + 276
24  libpython3.12.dylib                 0x00000001054a2a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
25  libpython3.12.dylib                 0x000000010555ae24 _PyEval_EvalFrameDefault + 20424
26  libpython3.12.dylib                 0x000000010545572c object_vacall + 312
27  libpython3.12.dylib                 0x0000000105455558 PyObject_CallMethodObjArgs + 104
28  libpython3.12.dylib                 0x0000000105592680 PyImport_ImportModuleLevelObject + 1972
29  libpython3.12.dylib                 0x0000000105552348 builtin___import__ + 276
30  libpython3.12.dylib                 0x00000001054a2a4c cfunction_vectorcall_FASTCALL_KEYWORDS + 172
31  libpython3.12.dylib                 0x0000000105453f74 PyObject_Vectorcall + 88
Abort(1) on node 0: Internal error

Tried to ctrl-c

ç^C[mpiexec@Kevins-Work-MacBook-Pro.local] Sending Ctrl-C to processes as requested
[mpiexec@Kevins-Work-MacBook-Pro.local] Press Ctrl-C again to force abort
[mpiexec@Kevins-Work-MacBook-Pro.local] HYDU_sock_write (/mpich-4.1.3/src/pm/hydra/lib/utils/sock.c:250): write error (Bad file descriptor)
[mpiexec@Kevins-Work-MacBook-Pro.local] send_hdr_downstream (/mpich-4.1.3/src/pm/hydra/mpiexec/pmiserv_cb.c:28): sock write error
[mpiexec@Kevins-Work-MacBook-Pro.local] HYD_pmcd_pmiserv_send_signal (/mpich-4.1.3/src/pm/hydra/mpiexec/pmiserv_cb.c:222): unable to write data to proxy
[mpiexec@Kevins-Work-MacBook-Pro.local] ui_cmd_cb (/mpich-4.1.3/src/pm/hydra/mpiexec/pmiserv_pmci.c:61): unable to send signal downstream
[mpiexec@Kevins-Work-MacBook-Pro.local] HYDT_dmxu_poll_wait_for_event (/mpich-4.1.3/src/pm/hydra/lib/tools/demux/demux_poll.c:76): callback returned error status
[mpiexec@Kevins-Work-MacBook-Pro.local] HYD_pmci_wait_for_completion (/mpich-4.1.3/src/pm/hydra/mpiexec/pmiserv_pmci.c:181): error waiting for event
[mpiexec@Kevins-Work-MacBook-Pro.local] main (/mpich-4.1.3/src/pm/hydra/mpiexec/mpiexec.c:260): process manager error waiting for completion

[ehsantn]

Thanks for the details @geruh and @kevinjqliu. I will try to reproduce on other machines. Looks a bit like network configuration issue (e.g. reverse hostname lookup). Does any other MPI use work (e.g. some simple mpi4py code)? Does some hellow work Bodo work?

Maybe this would help? https://stackoverflow.com/questions/23112515/mpich2-gethostbyname-failed

[kevinjqliu]

Looks a bit like network configuration issue (e.g. reverse hostname lookup).

im not seeing anything related to network from the error log above.

this might become a release blocker if we cannot verify and run integration tests

[ehsantn]

The MPI init errors can be obscure but the reverse hostname lookup issue with /etc/hosts is a common one. This error looks like a networking issue when spawned processes with MPI_Comm_Spawn try to connect back to the parent process.

Does setting BODO_DATAFRAME_LIBRARY_RUN_PARALLEL=0 eliminate the issue?

I'm trying to find a computer that reproduces. Installing OrbStack now too. What Linux distro/container do you use with OrbStack?

[kevinjqliu]

What Linux distro/container do you use with OrbStack?
im just running make test-integration-setup

Does setting BODO_DATAFRAME_LIBRARY_RUN_PARALLEL=0 eliminate the issue?

it passed, but also errored with a stacktrace

➜  iceberg-python git:(main) BODO_DATAFRAME_LIBRARY_RUN_PARALLEL=0 poetry run pytest tests/integration/test_reads.py::test_bodo_nan -m integration -vv -s -x


====================================================================== test session starts ======================================================================
platform darwin -- Python 3.12.11, pytest-7.4.4, pluggy-1.6.0 -- /Users/kevinliu/Library/Caches/pypoetry/virtualenvs/pyiceberg-Is5Rt7Ah-py3.12/bin/python
cachedir: .pytest_cache
rootdir: /Users/kevinliu/repos/iceberg-python
configfile: pyproject.toml
plugins: checkdocs-2.13.0, mock-3.14.1, lazy-fixture-0.6.3, requests-mock-1.12.1
collected 2 items                                                                                                                                               

tests/integration/test_reads.py::test_bodo_nan[session_catalog_hive] PASSED
tests/integration/test_reads.py::test_bodo_nan[session_catalog] PASSED

====================================================================== 2 passed in 14.70s =======================================================================
Abort(868841743): Fatal error in internal_Finalize: Other MPI error, error stack:
internal_Finalize(50)...........: MPI_Finalize failed
MPII_Finalize(400)..............: 
MPID_Finalize(652)..............: 
MPIDI_OFI_mpi_finalize_hook(830): 
flush_send_queue(780)...........: 
(unknown)(): Other MPI error
/Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/filesystem/s3fs.cc:3461:  arrow::fs::FinalizeS3 was not called even though S3 was initialized.  This could lead to a segmentation fault at exit
[1]    40811 segmentation fault  BODO_DATAFRAME_LIBRARY_RUN_PARALLEL=0 poetry run pytest  -m integration -vv -
➜  iceberg-python git:(main) 

[ehsantn]

I finally found a Mac laptop that reproduces! Here is a workaround to unblock while working on a fix:

BODO_DATAFRAME_LIBRARY_RUN_PARALLEL=0 FI_PROVIDER=tcp poetry run pytest tests/integration/test_reads.py::test_bodo_nan -m integration -vv -s -x

[kevinjqliu]

thank you! works like a charm. can we add those env vars to the test context?

iceberg-python/tests/integration/test_reads.py

Line 345 in b15937a

def test_bodo_nan(catalog: Catalog) -> None:

➜  iceberg-python git:(main) BODO_DATAFRAME_LIBRARY_RUN_PARALLEL=0 FI_PROVIDER=tcp poetry run pytest tests/integration/test_reads.py::test_bodo_nan -m integration -vv -s -x


====================================================================== test session starts ======================================================================
platform darwin -- Python 3.12.11, pytest-7.4.4, pluggy-1.6.0 -- /Users/kevinliu/Library/Caches/pypoetry/virtualenvs/pyiceberg-Is5Rt7Ah-py3.12/bin/python
cachedir: .pytest_cache
rootdir: /Users/kevinliu/repos/iceberg-python
configfile: pyproject.toml
plugins: checkdocs-2.13.0, mock-3.14.1, lazy-fixture-0.6.3, requests-mock-1.12.1
collected 2 items                                                                                                                                               

tests/integration/test_reads.py::test_bodo_nan[session_catalog_hive] PASSED
tests/integration/test_reads.py::test_bodo_nan[session_catalog] PASSED

======================================================================= 2 passed in 7.70s =======================================================================

[ehsantn]

Glad it worked! I'll open a PR ASAP. Will also work on a general fix in the Bodo package.