Share knowledge on profiling & debugging compiled code

[lagru]

I'd be curious to talk about and share how other people profile and debug their compiled code.

I've yet to figure out how to do that effectively for scikit-image (using meson and a lot of Cython code). I'm hopeful that print statements and temporarily porting algorithms to Python are not the only answers. 😅

If it looks like it would be useful, I'd be happy to help getting started with a small guide / tutorial on this.

[lesteve]

I have never tried but the best reference I know about this is https://willayd.com/fundamental-python-debugging-part-3-cython-extensions.html.

I also heard that some people are comfortable debugging and reading the generated C code by Cython. Full disclosure: I am not one of this people 😅.

[lesteve]

Another use case that came up, how do you genrate the HTML annotations cython -a through meson.

[lagru]

Reporting back with some findings.

I managed to get scikit-image to build with the --gdb and -g3 flags set that are mentioned in that blog post.

Diff

diff --git a/src/skimage/meson.build b/src/skimage/meson.build
index bb2891e31..fbbc73100 100644
--- a/src/skimage/meson.build
+++ b/src/skimage/meson.build
@@ -2,7 +2,7 @@
 is_windows = host_machine.system() == 'windows'
 is_mingw = is_windows and cc.get_id() == 'gcc'
 
-cython_c_args = []
+cython_c_args = ['-g3']
 if is_windows
   # For mingw-w64, link statically against the UCRT.
   gcc_link_args = ['-lucrt', '-static']
@@ -128,7 +128,7 @@ py3.install_sources(
 cython_cli = find_program('_build_utils/cythoner.py')
 cy = meson.get_compiler('cython')
 
-cython_args = ['@INPUT@', '@OUTPUT@']
+cython_args = ['@INPUT@', '@OUTPUT@', '--gdb']
 if cy.version().version_compare('>=3.1.0')
   cython_args += ['-Xfreethreading_compatible=True']
 endif

I even managed to break at the start of a Cython function.

Details

spin build
spin run -- cygdb --build-dir build/

(gdb) cy break _local_maxima
Function "__pyx_pf_7skimage_10morphology_11_extrema_cy__local_maxima" not defined.
Breakpoint 1 (__pyx_pf_7skimage_10morphology_11_extrema_cy__local_maxima) pending.
(gdb) cy run

>>> import numpy as np
>>> from skimage.morphology import local_maxima
>>> local_maxima(np.ones((3, 3)))

Thread 1 "python3" hit Breakpoint 1, __pyx_pf_7skimage_10morphology_11_extrema_cy__local_maxima (
    __pyx_self=<optimized out>, __pyx_v_defaults=(), __pyx_v__fused_sigindex=, __pyx_v_signatures=, 
    __pyx_v_args=(<numpy.ndarray at remote 0x7ffff6ffa790>, <numpy.ndarray at remote 0x7fffc9295290>, <numpy.ndarray at remote 0x7fffc9295230>), __pyx_v_kwargs=None)
    at src/skimage/morphology/_extrema_cy.cpython-313-x86_64-linux-gnu.so.p/_extrema_cy.c:21938
21938     __Pyx_INCREF(__pyx_v_kwargs);
44    def _local_maxima(dtype_t[::1] image not None,
(gdb) cy list
    39        unsigned char QUEUED_CANDIDATE = 1
    40        # None of the above is true
    41        unsigned char NOT_MAXIMUM = 0
    42    
    43    
>   44    def _local_maxima(dtype_t[::1] image not None,
    45                      unsigned char[::1] flags,
    46                      Py_ssize_t[::1] neighbor_offsets not None):
    47        """Detect local maxima in n-dimensional array.
    48

However, if I try to step with cy step things start to fail and the process gets stuck at 100% CPU usage. Stepping in C code with step seems to work fine. I haven't been able to figure out more so far.