Memory Leak Issue with Custom Series

[RDennis1]

Version of Dear PyGui

Version: 1.9.0
Operating System: Windows 10

My Issue/Question

While using a custom series in dearpygui, as explained in the official documentation, a memory leak has been observed.
Specifically, upon creation of a viewport with the custom series, the main process begins to steadily consume increasing amounts of memory.

Although the memory leak remains relatively small when following the demo application's example, it's been noticed that the leak becomes substantially larger when larger array arguments are passed to the x_data and y_data parameters. The memory leak seems to scale up proportionally with the size of the input data, which can potentially lead to significant performance issues and memory overflow when dealing with larger datasets.

To Reproduce

Steps to reproduce the behavior:

Copy and paste the following code into a python file and run it:

import dearpygui.dearpygui as dpg

dpg.create_context()
dpg.create_viewport()
dpg.setup_dearpygui()

x_data = [0.0, 1.0, 2.0, 4.0, 5.0]
y_data = [0.0, 10.0, 20.0, 40.0, 50.0]

def callback(sender, app_data):

    _helper_data = app_data[0]
    transformed_x = app_data[1]
    transformed_y = app_data[2]
    #transformed_y1 = app_data[3] # for channel = 3
    #transformed_y2 = app_data[4] # for channel = 4
    #transformed_y3 = app_data[5] # for channel = 5
    mouse_x_plot_space = _helper_data["MouseX_PlotSpace"]   # not used in this example
    mouse_y_plot_space = _helper_data["MouseY_PlotSpace"]   # not used in this example
    mouse_x_pixel_space = _helper_data["MouseX_PixelSpace"]
    mouse_y_pixel_space = _helper_data["MouseY_PixelSpace"]
    dpg.delete_item(sender, children_only=True, slot=2)
    dpg.push_container_stack(sender)
    dpg.configure_item("demo_custom_series", tooltip=False)
    for i in range(0, len(transformed_x)):
        dpg.draw_text((transformed_x[i]+15, transformed_y[i]-15), str(i), size=20)
        dpg.draw_circle((transformed_x[i], transformed_y[i]), 15, fill=(50+i*5, 50+i*50, 0, 255))
        if mouse_x_pixel_space < transformed_x[i]+15 and mouse_x_pixel_space > transformed_x[i]-15 and mouse_y_pixel_space > transformed_y[i]-15 and mouse_y_pixel_space < transformed_y[i]+15:
            dpg.draw_circle((transformed_x[i], transformed_y[i]), 30)
            dpg.configure_item("demo_custom_series", tooltip=True)
            dpg.set_value("custom_series_text", "Current Point: " + str(i))
    dpg.pop_container_stack()

with dpg.window(label="Tutorial") as win:
    dpg.add_text("Hover an item for a custom tooltip!")
    with dpg.plot(label="Custom Series", height=400, width=-1):
        dpg.add_plot_legend()
        xaxis = dpg.add_plot_axis(dpg.mvXAxis)
        with dpg.plot_axis(dpg.mvYAxis):
            with dpg.custom_series(x_data, y_data, 2, label="Custom Series", callback=callback, tag="demo_custom_series"):
                dpg.add_text("Current Point: ", tag="custom_series_text")
            dpg.fit_axis_data(dpg.top_container_stack())

dpg.set_primary_window(win, True)
dpg.show_viewport()
dpg.start_dearpygui()
dpg.destroy_context()

Open the task manager and observe the memory usage of the python process. It should steadily increase over time.

Expected behavior

The memory usage should remain constant after the initial creation of the custom series, and not increase over time.

Screenshots/Video

Example 1 (Demo application's example)

1. Memory usage of the python process after the initial creation of the custom series:

2. Memory usage of the python process after ~1 minute:

Example 2 (Larger input data)

Increasing the size of the input data to 100 elements for both x_data and y_data, the memory usage of the python process after the initial creation of the custom series is:

x_data = list(range(0, 100))
y_data = list(range(0, 1000, 10))

1. Memory usage of the python process after the initial creation of the custom series:

2. Memory usage of the python process after ~1 minute:

Standalone, minimal, complete and verifiable example

See above.

If you wish to test the memory leak with larger input data, you can use the following code:

Note: This leaks ~4MB a second.

import dearpygui.dearpygui as dpg

dpg.create_context()
dpg.create_viewport()
dpg.setup_dearpygui()

x_data = list(range(0, 1000))
y_data = list(range(0, 10000, 10))


def callback(sender, app_data):

    _helper_data = app_data[0]
    transformed_x = app_data[1]
    transformed_y = app_data[2]
    # transformed_y1 = app_data[3] # for channel = 3
    # transformed_y2 = app_data[4] # for channel = 4
    # transformed_y3 = app_data[5] # for channel = 5
    mouse_x_plot_space = _helper_data["MouseX_PlotSpace"]  # not used in this example
    mouse_y_plot_space = _helper_data["MouseY_PlotSpace"]  # not used in this example
    mouse_x_pixel_space = _helper_data["MouseX_PixelSpace"]
    mouse_y_pixel_space = _helper_data["MouseY_PixelSpace"]
    dpg.delete_item(sender, children_only=True, slot=2)
    dpg.push_container_stack(sender)
    dpg.configure_item("demo_custom_series", tooltip=False)
    for i in range(0, len(transformed_x)):
        dpg.draw_text((transformed_x[i] + 15, transformed_y[i] - 15), str(i), size=20)
        dpg.draw_circle((transformed_x[i], transformed_y[i]), 15, fill=(50 + i * 5, 50 + i * 50, 0, 255))
        if (
            mouse_x_pixel_space < transformed_x[i] + 15
            and mouse_x_pixel_space > transformed_x[i] - 15
            and mouse_y_pixel_space > transformed_y[i] - 15
            and mouse_y_pixel_space < transformed_y[i] + 15
        ):
            dpg.draw_circle((transformed_x[i], transformed_y[i]), 30)
            dpg.configure_item("demo_custom_series", tooltip=True)
            dpg.set_value("custom_series_text", "Current Point: " + str(i))
    dpg.pop_container_stack()


with dpg.window(label="Tutorial") as win:
    dpg.add_text("Hover an item for a custom tooltip!")
    with dpg.plot(label="Custom Series", height=400, width=-1):
        dpg.add_plot_legend()
        xaxis = dpg.add_plot_axis(dpg.mvXAxis)
        with dpg.plot_axis(dpg.mvYAxis):
            with dpg.custom_series(
                x_data, y_data, 2, label="Custom Series", callback=callback, tag="demo_custom_series"
            ):
                dpg.add_text("Current Point: ", tag="custom_series_text")
            dpg.fit_axis_data(dpg.top_container_stack())

dpg.set_primary_window(win, True)
dpg.show_viewport()
dpg.start_dearpygui()
dpg.destroy_context()

Additional context

I have only tested this on Windows 10 with Python 3.9.7 and DearPyGui 1.9.0.

I ran the above code using Pympler's SummaryTracker and it seems that thousands of float's are being created every second, which is likely the cause of the memory leak.

from pympler import tracker

tr = tracker.SummaryTracker()

# ... Code from above

tr.print_diff()

types	# objects	total size
float	2562004	58.64 MB
list	6810	20.06 MB
str	4242	297.61 KB
dict	1284	290.62 KB
tuple	1279	109.90 KB
int	2661	72.76 KB

[v-ein]

Most probably it's leaking the callback arguments, where transformed_x and transformed_y represent those lots of floats. There's a known issue with reference counting in callbacks and handlers, see #2036 for more info. I hope to push a fix one day; unfortunately I'm too busy at the moment.

[Atlamillias]

I recently came across this issue myself. Until the root cause(s) are fixed, I recommend using my approach to the problem on an as-needed basis using the Python C-API;

import ctypes


Py_DECREF = ctypes.pythonapi.Py_DecRef
Py_DECREF.argtypes = (ctypes.py_object,)
Py_DECREF.restype  = None

On my PC, your script leaked about twice as fast (roughly ~8MB/s). Calling the above Py_DECREF function on both transformed_x/y at the end of the callback stems the leak to ~1 MB every minute or so. However, calling it on app_data instead corks it.

[v-ein]

A heads up - #2580 fixes the leak of callback arguments that I mentioned. However, custom_series still leaks something in rather small portions, something like a dozen Kb per second, as opposed to megabytes per second without #2580. I'm keeping this ticket open in a hope that we'll eventually hunt down the remaining leak.