feat(embed): support baking PythonCall into a juliacall system image

When PythonCall is compiled into a juliacall system image, its __init__
runs during jl_init_with_image, before juliacall's bootstrap defines
Main.__PythonCall_libptr. Embedding was therefore mis-detected as
non-embedded and failed with "'juliacall' module already exists".

Add an opt-in embedded preference / JULIA_PYTHONCALL_EMBEDDED (via the
same getpref mechanism as exe/lib) that forces the embedded path and
obtains libpython from the lib preference / JULIA_PYTHONCALL_LIB (already
loaded in the host process). Unset, behaviour is unchanged. Docs and
CHANGELOG updated.

Author: ncudlenco

CHANGELOG.md

@@ -1,6 +1,9 @@
 # Changelog
 
 ## Unreleased
+* Support baking `PythonCall` into a juliacall system image via the new opt-in
+  `embedded` preference / `JULIA_PYTHONCALL_EMBEDDED` option, removing the
+  `using PythonCall` cost from cold start. No behaviour change unless opted in.
 * Added option `lib` to JuliaCall. Setting this will skip the discovery subprocess.
 * Bug fixes.
 

docs/src/juliacall.md

@@ -115,6 +115,26 @@ systems that may be readonly. Note that the project set in
 `PYTHON_JULIACALL_PROJECT` *must* already have PythonCall.jl installed and it
 *must* match the JuliaCall version, otherwise loading Julia will fail.
 
+### Baking PythonCall into a system image
+
+For the fastest possible startup you can compile `PythonCall` itself (alongside
+your own packages) into a system image with
+[PackageCompiler.jl](https://github.com/JuliaLang/PackageCompiler.jl), so that
+the `using PythonCall` performed at startup is a memory-map rather than a load.
+
+When `PythonCall` is baked into the system image its `__init__` runs *during*
+`jl_init_with_image`, before juliacall's bootstrap has defined the
+`Main.__PythonCall_libptr` global it normally uses to detect that it is
+embedded. To support this, set the `embedded` preference (or the
+`JULIA_PYTHONCALL_EMBEDDED=yes` environment variable) together with the `lib`
+preference / `JULIA_PYTHONCALL_LIB` pointing at the running interpreter's
+libpython. With `embedded` set, PythonCall takes the embedded path even without
+the global and opens libpython by path (it is already loaded in the process, so
+this is just a handle). The default is `no`, leaving normal behaviour
+unchanged. Use this together with `PYTHON_JULIACALL_SYSIMAGE` (below), and
+`PYTHON_JULIACALL_EXE` / `PYTHON_JULIACALL_PROJECT` so juliacall resolves the
+baked environment directly.
+
 ## [Configuration](@id julia-config)
 
 Some features of the Julia process, such as the optimization level or number of threads, may

src/C/context.jl

@@ -105,11 +105,36 @@ on_main_thread
 
 function init_context()
 
-    CTX.is_embedded = hasproperty(Base.Main, :__PythonCall_libptr)
+    # Normally PythonCall is embedded when Python (via juliacall) defines the
+    # global `Main.__PythonCall_libptr`, set by juliacall's bootstrap *after*
+    # `jl_init_with_image`. If PythonCall is baked into a juliacall system
+    # image, its `__init__` runs *during* `jl_init_with_image` — before that
+    # global exists — yet we are still embedded (Python is the running host).
+    # The opt-in `embedded` preference / `JULIA_PYTHONCALL_EMBEDDED` forces the
+    # embedded path in that case; libpython is obtained by path since it is
+    # already loaded in this process. Unset, behaviour is unchanged.
+    has_libptr = hasproperty(Base.Main, :__PythonCall_libptr)
+    CTX.is_embedded = has_libptr || Utils.getpref_embedded()
 
     if CTX.is_embedded
-        # In this case, getting a handle to libpython is easy
-        CTX.lib_ptr = Base.Main.__PythonCall_libptr::Ptr{Cvoid}
+        if has_libptr
+            # In this case, getting a handle to libpython is easy
+            CTX.lib_ptr = Base.Main.__PythonCall_libptr::Ptr{Cvoid}
+        else
+            # Baked into a sysimage: open libpython by path (the `lib`
+            # preference / JULIA_PYTHONCALL_LIB). dlopen of an
+            # already-loaded library just returns a handle to it.
+            lib_path = something(Utils.getpref_lib(), Some(nothing))
+            lib_path === nothing && error(
+                "JULIA_PYTHONCALL_EMBEDDED is set but libpython is unknown; " *
+                "set the `lib` preference or JULIA_PYTHONCALL_LIB to its path.",
+            )
+            lib_ptr = dlopen_e(lib_path, CTX.dlopen_flags)
+            lib_ptr == C_NULL &&
+                error("Python library $(repr(lib_path)) could not be opened.")
+            CTX.lib_path = lib_path
+            CTX.lib_ptr = lib_ptr
+        end
         init_pointers()
         # Check Python is initialized
         Py_IsInitialized() == 0 && error("Python is not already initialized.")

src/Utils/Utils.jl

@@ -17,6 +17,7 @@ checkpref(::Type{String}, x::AbstractString) = convert(String, x)
 getpref_exe() = getpref(String, "exe", "JULIA_PYTHONCALL_EXE", "")
 getpref_lib() = getpref(String, "lib", "JULIA_PYTHONCALL_LIB", nothing)
 getpref_pickle() = getpref(String, "pickle", "JULIA_PYTHONCALL_PICKLE", "pickle")
+getpref_embedded() = getpref(String, "embedded", "JULIA_PYTHONCALL_EMBEDDED", "no") == "yes"
 
 function explode_union(T)
     @nospecialize T