Dreams

[hinerm]

A core goal is to continue advancing Fiji + Python integration.

The Script Editor where you can simply use python from Fiji seems like the most user-friendly direction for integration. I think a general opting-in to a "Python integration mode" is quite reasonable.

The fundamental things I think we still need:

• Automatically open images in shared memory when in Python mode
• VERY SIMPLE environment management. Ideally this would be totally invisible to the user when running existing scripts and plugins.

[ctrueden]

For scripting in Python, I want:

1. #@repository and #@dependency support for Python packages a la scijava-grab but with understanding of Python repos and dependencies.
2. #@script support for declaring script metadata, including which script language plugin/mode should be used—e.g. Jython, GraalPy, Python in-process (i.e. this repository), or Appose-driven interprocess Python (probably another new repo, but possibly a new feature of this one).
3. Much better code completion, so that scripting Python in the Script Editor is not so incredibly painful compared to other options.

(1) may belong in its own component, maybe scijava-grab-python?
(2) is part of scijava-common

For automatically opening images into shared memory:

• That is something that would be handled by Fiji's image opening mechanism. So probably part of SCIFIO—either the current one, or a new incarnation of it.
• Of course, we don't need to open images into shared memory unless we are using interprocess scripting, i.e. Appose script mode, which we do not have at all yet (although Appose has laid the foundation—e.g. Icy3 has already prototyped it). But we do need to open images off-heap for in-process Python mode, which can be done either by allocating the memory in Python (e.g. just creating a numpy array of size X), or via ByteBuffer.allocateDirect on the Java side. It's also possible as of Java 22's FFM subsystem to share out on-heap memory segments (see this example from @elevans), but it's only intended to be safe/stable for short-duration native function calls.
• So another idea would be to have a smart "Share" command/feature where—whenever an image is requested to be passed to a native function—the user is asked if it is OK to make "share" the image by replacing it with a copy. It would gain a symbol in the title bar similar to virtual's (V), but instead indicate that the image is in shared memory (for use with Appose-based scripts), or off-heap memory (for use with in-process/JPype-based Python scripts).
• Generalizing this idea, one can imagine Fiji being smart about the backing storage container of each image: on-heap, off-heap, shared memory, and other/fully virtual where the memory layout is unknown (and indeed there may not even be sample values in memory, e.g. FunctionRandomAccessible). And when invoking a command that requires an image as input, Fiji could know which container(s) are acceptable for passing to that command. This might even be possible to express solely via SciJava Ops with type converters.

For environment management, if you are talking about a GUI, that should go into a new appose-gui library in the apposed org. I am hoping to use @stefanhahmann's Python Environment Manager UI prototype as a starting point.