A Python module for round-tripping data between Python and Julia

[jpivarski]

How are the dependencies managed by package managers? How do PyJulia and PyCall.jl do it? Maybe only conda can do it automatically?

How is memory managed between Python and Julia? Can they zero-copy NumPy arrays to and from Julia Vectors? What if a buffer is own by Python, referenced in Julia, and then that reference is referenced in Python? And then when it's released in Python, do the garbage collectors collect it?

What kind of garbage collector does Julia have? Mark-and-sweep (with or without Python's reference-count shortcut)? Generational (which would involve a lot of copying, after all)?

We should probably give the user an option to copy buffers at the Python-Julia boundary, so that they can break references, at the cost of a memcpy (often not a big deal).

[Moelf]

Julia GC is generational mark-and-sweep except that it doesn't move data around (precisely because of interpolation with other languages, they don't want to break pointers Julia gave to C or whatever).

---

PyCall.jl and PythonCall.jl have different "designs" regarding no-copy conversion rules, but the meat of no-copy getting Numpy array in the end always rely on unsafe_wrap on pointer pointing to data owned by Python:

• https://github.com/JuliaPy/PyCall.jl/blob/2a9f07725b1bce8c07ac5437d66701dd9579a221/src/pyarray.jl#L326
• https://github.com/cjdoris/PythonCall.jl/blob/a803e2e30408cb9b0c5ed23d1a1344a13bd63af8/src/concrete/numpy.jl#L3-L9 (this is for a Numpy scalar)
• https://github.com/cjdoris/PythonCall.jl/blob/a803e2e30408cb9b0c5ed23d1a1344a13bd63af8/src/Py.jl#L40-L43

One thing though, by default, PythonCall.jl (and julicall) does no-copy in both directions; by contrast, PyCall.jl (and pyjulia) by default is only non-copy in Julia array -> Python array direction. (the first link in bulletin is the special function you need for non-copy Numpy -> Julia array)

[jpivarski]

@ianna, I don't plan to make more (significant) changes to the library before the JuliaHEP workshop, so it should be stable enough to develop the Python bridges anytime you're ready.

[PallHaraldsson]

by default, PythonCall.jl (and julicall) does no-copy in both directions; by contrast, PyCall.jl (and pyjulia) by default is only non-copy in Julia array -> Python array direction.

I just learned of this Python/C++ package, and now this Julia-only implementation. Both seem great, but I think PythonCall.jl should be documented in the readme. It's unclear to me how the other Python interop projects are used, I don't see that in the source code. Is it meant that they can be used? Or are somehow, by the help of some dependency?

PythonCall.jl/Juliacall should be enough, doesn't rule out use of the other, even together, so I think you may want to suggest just the new/better replacement. If you're unsure, then you could document both options. Do you think the older packages are somehow preferred?

[Moelf]

One reason one may prefer PyCall.jl is that somehow I think it's a tad less buggy and we have pydiffeq and pysr as example to follow

[ianna]

I've been testing it with PyCall/PyJulia combination and it seems that these packages prefer separate environments. I will check the PythonCall.jl/Juliacall combination - thanks for the suggestion @PallHaraldsson !

[PallHaraldsson]

Was or is buggy? At least yes, pydiffeq used [py]julia (then additionally required jill), and now in early September migrated to JuliaCall:

SciML/diffeqpy@1d1ab4b

maybe because I proposed it. So I hope it's ok. [I'm not sure why it still references/uses? jill, it seems redundant by now.] I've not used JuliaCall or PythonCall.jl much, just some testing, actually musefl, while I did use PyCall.jl much at the time in my work.

I think PySR still uses it but see here talk on it also migrating from it: MilesCranmer/PySR#363 (comment)