Discovery: How will distribution of this package work?

[MarquessV]

The build process for this library is relatively complex and likely sensitive to different platforms. How can we distribute this crate in a way that it works on each platform? How can we ease compilation as much as possible for downstream users? For our use case specifically, qcs-sdk-rust compiles Python wheels for every major platform, how can we make sure that still works as needed?

[MarquessV]

One wrinkle that has come up so far in #3:

When pulling the library into qcs-sdk-rust, there are issues with memory limits:

ensure_space: failed to allocate 1048576 bytes at 0x50000000
(hint: Try "ulimit -a"; maybe you should increase memory limits.)

This occurs even after maxing out all limits.

ulimit unlimited
ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             65532
-c: core file size (blocks)         unlimited
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       5568
-n: file descriptors                256

Some cursory google search hints towards this having something to do with sbcl, but I haven't found a fix yet. It's also curious that this doesn't happen inside of the crate. For example, if we run the bell_state example in this crate, everything works fine, but an equivalent example in qcs-sdk-rust blows up.

EDIT: This fixes the issue within the libquil-sys repo itself, but doesn't prevent the error from happening in packages that depend on libquil-sys.

[MarquessV]

I've tried using git submodules to make it possible to build the crate with just sbcl and quicklisp installed but with no real luck. There doesn't seem to be a good API for triggering the builds of any of the dependent packages from within Rust. Additionally, quicklisp also reaches out to the internet to download other dependencies while building quilc, which doesn't feel good to have as part of a build step.

[MarquessV]

Using libquilc-sys as a dynamically linked library in a Rust crate

Three things need to happen for libquilc-sys to work in a dependent crate properly:

1. The libquilc dynamic library (ie. libquilc.dylib) needs to be in some known library search path so the Rust compiler can pick up on it. There are a handful of default search paths, so we can document some common ones as options. (As an aside: Rust and Python don't use all the same search paths for dynamic libraries).

2. We need to figure out a better way to load the core file from a known location. Currently, dependents need to copy the core file to the root of their project so libquilc-sys can find it.

3. For the crate to work on Intel macs, we need to add the following linker argument to build.rs (see quil-lang/sbcl-librarian for why this is a thing):

    if cfg!(target_os = "macos") {
        println!("cargo:rustc-link-arg=-pagezero_size 0x100000");
    }

We set this in the build script for libquilc-sys as well, but it doesn't propagate down to dependents of the crate.

Using libquilc-sys as a dynamically linked library in a Rust crate with Python bindings

All the same steps taken above apply, but (3) poses a bigger issue in this context. Because -pagezero_size is a linker arg, it needs to be applied to an executable during a link step. The bindings generated by pyo3 is a Python library, so upon building them, you get this error:

  = note: ld: -pagezero_size option can only be used when linking a main executable
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

If you omit the linker arg, the library crashes when trying to load the core file, so there is no working around this. The only fix would be to build Python from source with -pagezero_size 0x100000 which obviously is a ton to ask of users (on top of an already exceedingly complex build process).

The sbcl-librarian README strongly states that this linker flag "MUST" be used for it to work so I don't think this is something we'll easily be able to workaround ourselves. We'll have to followup with the quil-lang group and see if they have any thoughts or advice.

[Shadow53]

Rust and Python don't use all the same search paths for dynamic libraries

Specifically, Rust will search $HOME/lib (at least on mac), while Python restricts itself to /usr/local/lib

For the crate to work on Intel macs, we need to add the following linker argument to build.rs... but it doesn't propagate down to dependents of the crate.

I want to call extra attention to this: every executable binary crate consuming libquil-sys, no matter how transitively, has to have a build.rs with this. If we could get everything else working, we'd probably offer a function to make this a one-liner, similar to what PyO3 does.

Because -pagezero_size is a linker arg, it needs to be applied to an executable during a link step...

Quick correction on this: pagezero_size specifically can only be applied to executables. AFAIK, most linker args can be applied to libraries. The reason for this is a little beyond my understanding, but I will try to explain.

PAGEZERO is a section of an executable that, AFAICT, defines the range of pointers to consider invalid/null. On Intel Macs, this is defined as 4GB, effectively disallowing 32-bit pointers on the 64-bit system. The rationale is that it helps prevent bad things from happening if a program accidentally truncates a 64-bit pointer to 32-bit.

sbcl-librarian, from what I understand, depends on this address space for the embedding lisp interpreter, which is why trying to use the quilc library without this flag will fail. As @MarquessV mentions, though, the only way to make this work with Python is to compile the Python executable with this linker flag.

---

From my understanding, this is very much a problem with Intel MacOS. Maybe M1s wouldn't have an issue. Sounds like Linux shouldn't have an issue either. So it may be possible that we can get this working on Linux, which would at least help avoid the need for sidecar docker images. We would have to figure out how to support MacOS in that case, though.

[MarquessV]

We were pointed towards this branch of sbcl that has some patches that workaround the need for pagezero_size. Unfortunately it's based on 2.1.10, which is just short of the 2.2.0 version that we know works. On my machine at least, this version of SBCL doesn't build.