feat(juliacall): supply libjulia path/bindir to skip the discovery subprocess#2
Merged
Merged
Conversation
ncudlenco
force-pushed
the
feat/juliacall-skip-libjulia-discovery
branch
from
f7c8ad4 to
6e2b9c1
Compare
…very subprocess init() starts a short-lived Julia process solely to print libjulia's path and Sys.BINDIR. In pre-built containers / system images these are static and known ahead of time; allow supplying them via the libpath / default_bindir options (PYTHON_JULIACALL_LIBPATH / PYTHON_JULIACALL_DEFAULT_BINDIR) to skip the extra process. Behaviour is unchanged unless both are set. Docs and CHANGELOG updated.
ncudlenco
force-pushed
the
feat/juliacall-skip-libjulia-discovery
branch
from
6e2b9c1 to
df51ae6
Compare
Member
Author
|
Merged into |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds opt-in
libpath/default_bindirjuliacall options so the path to libjulia andSys.BINDIRcan be supplied directly, skipping the short-lived Julia processinit()otherwise spawns at startup just to discover them. No behaviour change unless both are set.Motivation
In short-lived Python processes that embed Julia via juliacall (serverless / autoscaled containers), every fresh process pays
import juliacallstartup with no long-lived server to amortise it. During that startupinit()launches a whole separate Julia process —julia -O0 --compile=min -e 'print(Libdl.dlpath("libjulia"), Sys.BINDIR)'— solely to discover two strings. That is an entire extra Julia interpreter boot on the critical path of every cold start. In a pre-built container or system image those paths are static and known at build time, so the subprocess is pure repeated waste.Change
Two new juliacall options (same
-X/PYTHON_JULIACALL_*mechanism as the rest):libpath→PYTHON_JULIACALL_LIBPATHdefault_bindir→PYTHON_JULIACALL_DEFAULT_BINDIRWhen both are set the discovery subprocess is skipped and the supplied values used; otherwise behaviour is exactly as before.
docs/src/juliacall.mdconfig table andCHANGELOG.mdupdated.Backward compatibility
No behaviour change unless both options are set. Values are validated with the existing
path_option(..., check_exists=True)helper, so a bad path fails fast with the standard error message.Testing
The new options only take effect when explicitly set, so default behaviour is unchanged and the existing test suites are unaffected.
Independent of, and complementary to, the system-image PR (JuliaPy#773): together they remove the two biggest fixed costs from a cold juliacall start — loading
PythonCall, and this extra Julia process.Related issues
Relates to JuliaPy#762 ("Improve juliacall startup time?") — this removes a whole extra Julia process from every cold start. Adjacent to JuliaPy#619 (offline /
BINDIRstartup); note JuliaPy#619's specific complaint (juliapkg still attempting a network install) is a different concern this PR does not by itself resolve.