Graal Native Image Support

[aherlihy]

Hi, does the DuckDB JDBC driver support use from a graal native image?

I tried with v1.1.1 using graalvm-jdk-17.0.12+8.1 using SBT NativeImagePlugin, but it fails with:

Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load library: <...>/build/debug/libduckdb_java.so_osx_universal
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibraryAbsolute(NativeLibrarySupport.java:100)
	at java.base@17.0.12/java.lang.ClassLoader.loadLibrary(ClassLoader.java:57)
	at java.base@17.0.12/java.lang.Runtime.load0(Runtime.java:755)
	at java.base@17.0.12/java.lang.System.load(System.java:1957)
	at org.duckdb.DuckDBNative.<clinit>(DuckDBNative.java:52)
	at org.duckdb.DuckDBConnection.newConnection(DuckDBConnection.java:51)
	at org.duckdb.DuckDBDriver.connect(DuckDBDriver.java:41)
	at java.sql@17.0.12/java.sql.DriverManager.getConnection(DriverManager.java:681)
	at java.sql@17.0.12/java.sql.DriverManager.getConnection(DriverManager.java:252)
        ...

on Ubuntu 22.04 LTS (and equivalently build/debug/libduckdb_java.so_osx_universal on MacOS 15.3.1). It works fine from a regular JAR. The build.sbt native image settings are:

 .settings(
    nativeImageInstalled := true,
    nativeImageGraalHome := {
      val executable = "native-image"
      val paths = System.getenv("PATH").split(Pattern.quote(pathSeparator))
      paths.map(Paths.get(_)).find(p => Files.isExecutable(p.resolve(executable))) match {
        case Some(binDir) =>
          binDir.getParent
        case None =>
          throw new MessageOnlyException(s"Could not find binary `$executable` in PATH.")
      }
    },
    nativeImageCommand := nativeImageCommand.value ++ Seq(
      "--no-fallback",
      "--report-unsupported-elements-at-runtime"
    ),

    libraryDependencies ++= Seq(
       ...,
      "org.duckdb" % "duckdb_jdbc" % "1.1.1",
    )

I tried extracting the library files from the DuckDB jar and linking them manually but it just segfaulted, so I would be very appreciative of any help. Thank you!

[staticlibs]

Hi! AOT is unlikely to work in the current version.

Short term, the idea is to make native library loading more flexible - support loading it from FS without unpacking from JAR, on a default or specified path. When this is working, the Java part, that is pretty thin and does not use reflection, should be easy to compile with Graal. And library loading should be flexible enough to work with Graal.

Long term, the idea (only an idea at this point) to have DuckDB engine (without Java-specific bits) to be packaged as a shared lib that exposes stable C-API. And to have JDBC impl as Java-only (again, without reflection) that uses Foreign Function & Memory API to work with the engine shared lib over C-API. This way the Java part (that will have more logic than now) can be AOT-compiled as an ordinary Java application, assuming Graal will have the full support for FFM.

PS: I will comment here when library loading enhancements are done.

[xiaobo]

any progress? thx

[davclark]

To make the context a little more explicit on the above mention, this would unlock embedding in babashka, which is a fast-starting clojure(ish) interpreter that provides a lightweight dag syntax, pipelining, and a dynamic lispy DSL for SQL.

[borkdude]

@davclark Thanks for mentioning babashka. I think I might explore the golang route for building a pod for duckdb since in golang you can statically embed the library in an easier way. Similar work has been done here for sqlite3: https://github.com/babashka/pod-babashka-go-sqlite3

[FreifeldRoyi]

Wow, that would be awesome if you could support Graal 🙏🏻
Is it in the upcoming roadmap?

[staticlibs]

@FreifeldRoyi

The library loading improvements went in in #450. Current Graal-related activity is in #452. Graal support is currently not scheduled for 1.5, though if someone knows how to make AOT-compiled driver to run at least hello-world examples - I can include such changes promptly.

[PavelNikolaichev]

@FreifeldRoyi

The library loading improvements went in in #450. Current Graal-related activity is in #452. Graal support is currently not scheduled for 1.5, though if someone knows how to make AOT-compiled driver to run at least hello-world examples - I can include such changes promptly.

A few days ago I've tried creating JNI config for quarkus through trial and error. Maybe I am missing something, but I've set build args to init duckdb at runtime and added a dll from jar file so that native image was able to link it. It did work - I was able to init duckdb, fetch data from the parquet and query it. If you want, I can make an example repo to show what I've done. But maybe I just misunderstand something.