pyca/cryptography: cffi build and __file__

[alexzorin]

Hello,

Very cool project - it's what I wish Oracle's Graal could do for Python.

I'm wondering whether you could give a hint for using pyca/cryptography. I'm actually trying to see how hard it would be to build Certbot, but this is the minimal repro:

[[packaging_rule]]
type = "pip-install-simple"
package = "cryptography==2.7"

[[embedded_python_run]]
mode = "eval"
code = "from cryptography.hazmat.backends import default_backend; default_backend()"

Runtime error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cryptography.hazmat.backends", line 15, in default_backend
  File "cryptography.hazmat.backends.openssl", line 7, in <module>
  File "cryptography.hazmat.backends.openssl.backend", line 18, in <module>
  File "cryptography.x509", line 8, in <module>
  File "cryptography.x509.base", line 16, in <module>
  File "cryptography.x509.extensions", line 19, in <module>
  File "cryptography.hazmat.primitives.constant_time", line 11, in <module>
ModuleNotFoundError: No module named 'cryptography.hazmat.bindings._constant_time'

Looking at how pyca/cryptography seems to build this module, I believe this is a variation of the __file__ problem described in the docs.

I've not had much success with applying the advice from the docs. I think this might be because of the use of __file__ as part of the cffi build process rather than something happening at runtime. I'm not that familiar with the Python runtime in general 🤷‍♂️.

If this is too complex or impossible at the moment, that's okay too! Thanks.

[seanjensengrey]

I'd like to take a look at the specifics of how the file system encapsulation is done, but if we interpose open/read/etc we could reroute some calls for resolving files to an inproc function and read only segment in the executable. Stuff like late loading of resources (even non-python resources) would then just work. Other calls that are outside of a certain root would be passed to libc/syscall and continue to function as expected.

[indygreg]

I'm pretty certain the cryptography package uses C extensions. As things are currently implemented, C extensions may be built during PyOxidizer packaging. But there is no guarantee that will work, those binaries will be portable, or that the extension files will be recognized as such and installed in the proper location.

I think what may happen in a lot of cases is that the C extensions get built (as .so files on Linux) and these .so files are detected as Python resource files and embedded into the binary as such.

Try looking in the build output logs for constant_time.

[alexzorin]

With your advice, I was able to get a little bit further (I think, maybe it's an unhelpful detour) by relocating the built extension files (.so) from lib/cryptography.hazmat.bindings/ to lib/cryptography/hazmat/bindings/.

This leaves me with a linked symbol error for PyObject_CallMethod. I think this is because Python extensions on Linux do not explicitly depend on libpython and the linker would normally locate the symbol globally?

ImportError: /home/alex/devel/certbot-oxidized/build/apps/certbot-oxidized/lib/cryptography/hazmat/bindings/_constant_time.abi3.so: undefined symbol: PyObject_CallMethod

OTOH maybe extensions are meant to be loaded in an entirely different way ...

[indygreg]

PyOxidizer does not yet support loading custom C extensions. It may just work in some scenarios. But I expect that not to be true in many environments. The way binaries are compiled is completely different from how Python normally does things. You almost certainly need to compile C extensions in a very specific manner to have them work with PyOxidizer binaries. And the default compilation settings used by traditional python setup.py ... will not work.

My tentative plan for this is to hack up distutils when invoking setup.py such that C compiler invocations are crafted in such a way as to be compatible with PyOxidizer. I'm pretty sure what will end up happening is setup.py will communicate a set of object files to PyOxidizer, which will then link those object files into the libpython that is dynamically produced as part of creating binaries. This is a bit complicated, I know. But it should work.

There's a separate avenue to explore on whether we could support alternate build modes where we do use a separate libpython in PyOxidizer and existing Python C extensions just work. I'm not opposed to this idea. But one of my goals with PyOxidizer was to enable single file executables. I just haven't been that interested in the "simple" solution of supporting multi-file Python applications :)

[jayvdb]

@alexzorin @bsiegel @cvanlabe , #181 should fix the extension problem with cryprography and other packages which use ext_package.

It seems to not be very common any more. https://github.com/pyside/pyside-setup/blob/master/setup.py also uses it.

Note that there is another problem with cryptography #170

I am testing with the setup.py deleted in a fork

packaging_rules.append(PipInstallSimple("git+https://github.com/jayvdb/cryptography#egg=cryptography"))

[alexzorin]

Thank you for your work on this @jayvdb! #181 + removing pyproject.toml did the trick.

[jayvdb]

#199 avoids pyproject.toml problem. Just need to get it working on Windows :/