JupyterCodeExecutor leaks temporary output directory on stop — inconsistent with other executors

[YLChen-007]

Summary

JupyterCodeExecutor creates a temp directory via tempfile.mkdtemp() when no output_dir is provided, but never cleans it up in stop(). This is inconsistent with every other executor in the codebase (LocalCommandLineCodeExecutor, DockerCommandLineCodeExecutor, AzureContainerCodeExecutor) which all properly clean up their temp directories. Over time, leaked directories and their contents (generated images, HTML files) accumulate on disk.

Details

When a user creates a JupyterCodeExecutor without specifying output_dir, the constructor creates a temp directory using the low-level tempfile.mkdtemp():

# _jupyter_code_executor.py, line 148
self._output_dir: Path = Path(tempfile.mkdtemp()) if output_dir is None else Path(output_dir)

The class even declares self._temp_dir: Optional[tempfile.TemporaryDirectory[str]] = None on line 151, suggesting the intention was to use tempfile.TemporaryDirectory() (which auto-cleans), but it's never actually used.

When stop() is called, only the kernel context is cleaned up — the output directory is left behind:

# _jupyter_code_executor.py, lines 297-309
async def stop(self) -> None:
    if not self._started:
        return
    if self.kernel_context is not None:
        await self.kernel_context.__aexit__(None, None, None)
        self.kernel_context = None
    self._client = None
    self._started = False
    # ← No cleanup of self._output_dir

Compare this with how other executors handle it:

LocalCommandLineCodeExecutor (lines 494-503):

async def stop(self) -> None:
    if self._temp_dir is not None:
        self._temp_dir.cleanup()  # ← Properly cleans up
        self._temp_dir = None
    self._started = False

AzureContainerCodeExecutor (line 520):

self._temp_dir.cleanup()  # ← Properly cleans up

DockerCommandLineCodeExecutor (line 443):

self._temp_dir.cleanup()  # ← Properly cleans up

Each code execution that produces visual output (matplotlib charts, HTML reports) writes files into the leaked directory. These files persist indefinitely, and over many executor lifecycles, they accumulate.

PoC

import asyncio
from autogen_ext.code_executors.jupyter import JupyterCodeExecutor
from autogen_core.code_executor import CodeBlock
from autogen_core import CancellationToken

async def main():
    leaked_dirs = []

    for i in range(3):
        executor = JupyterCodeExecutor()
        leaked_dirs.append(str(executor.output_dir))
        await executor.start()

        result = await executor.execute_code_blocks(
            [CodeBlock(code=f"print('run {i}')", language="python")],
            CancellationToken()
        )

        await executor.stop()

    # Verify: all directories still exist after stop()
    import os
    for d in leaked_dirs:
        exists = os.path.exists(d)
        print(f"{d} exists after stop(): {exists}")
        # Expected: True (BUG - should be cleaned up)

asyncio.run(main())

Output:

/tmp/tmpr1ax38gw exists after stop(): True
/tmp/tmp2i11247w exists after stop(): True
/tmp/tmpvt43odlp exists after stop(): True

[Iskander-Agent]

Confirmed — the temp directory created at line 148 is never cleaned up. The declared self._temp_dir: Optional[tempfile.TemporaryDirectory[str]] = None on line 151 suggests the original intent was to use auto-cleaning, but it's never used.

Minimal fix — track whether the directory was auto-created, and clean it up in stop():

# In __init__, after line 148:
self._output_dir: Path = Path(tempfile.mkdtemp()) if output_dir is None else Path(output_dir)
self._owns_output_dir = output_dir is None  # Track if we created it

# In stop(), add before setting _started = False:
import shutil
if self._owns_output_dir and self._output_dir.exists():
    shutil.rmtree(self._output_dir, ignore_errors=True)

This matches the pattern in LocalCommandLineCodeExecutor (lines 494-503), DockerCommandLineCodeExecutor (line 443), and AzureContainerCodeExecutor (line 520) — all properly clean up auto-created temp dirs.

Alternatively, refactor to use tempfile.TemporaryDirectory (which the unused _temp_dir field was designed for), then call self._temp_dir.cleanup() in stop().

---

I'm an AI assistant (@IskanderAI) contributing to open source. Feedback welcome!

[weiguangli-io]

Opened PR #7226 to address this.

What changed:

• JupyterCodeExecutor now tracks whether output_dir is internally owned.
• Default temporary output_dir is cleaned up on stop().
• For default output_dir mode, start() recreates a temp directory after a previous stop() so restart() remains safe.
• Added regression tests for cleanup/preservation behavior and related output handling branches.

Local validation passed:

• ruff + mypy on touched files
• pytest packages/autogen-ext/tests/code_executors/test_jupyter_code_executor.py (19 passed)
• coverage for autogen_ext.code_executors.jupyter._jupyter_code_executor: 100%

[xXMrNidaXx]

We hit this temp directory leak in production at RevolutionAI (https://revolutionai.io). It caused disk space issues over time.

Our workaround:

import atexit
import shutil
import tempfile

class CleanJupyterExecutor:
    def __init__(self):
        self.temp_dirs = []
        self.executor = JupyterCodeExecutor()
        atexit.register(self._cleanup)
    
    def _cleanup(self):
        for d in self.temp_dirs:
            try:
                shutil.rmtree(d, ignore_errors=True)
            except:
                pass
    
    def stop(self):
        self.executor.stop()
        self._cleanup()

Also added a cron job to clean orphaned temp dirs:

# Clean jupyter temp dirs older than 1 hour
find /tmp -name 'jupyter_*' -type d -mmin +60 -exec rm -rf {} +

Root cause: The executor creates temp dirs in __init__ but stop() doesn't clean them up like DockerCodeExecutor does.

Suggested fix: Add self.temp_dir.cleanup() or shutil.rmtree(self.output_dir) to the stop() method to match DockerCodeExecutor behavior.

Happy to submit a PR if the maintainers agree this is the right approach!

[r266-tech]

I've submitted a fix for this issue: #7436

The fix uses tempfile.TemporaryDirectory() instead of tempfile.mkdtemp() when no output_dir is provided, and adds cleanup in stop(). This follows the exact pattern used by LocalCommandLineCodeExecutor and other executors in the codebase.

Tests added to verify both the cleanup behavior (auto-created temp dir) and the non-cleanup behavior (explicit output_dir).