Support select_artifacts.jl (new in Julia 1.6)

[colinxs]

I stumbled across some changes in 1.6 and later versions of Pkg.jl that might break julia2nix or at least limit its function. Prior to 1.6, everything in the Artifacts.toml was statically defined. Now there is an optional file located at project_root/.pkg/select_artifacts that can generate an arbitrary Artifacts.toml. The description is here and relevant code here.

[colinxs]

Also thought you'd find this interesting: JuliaLang/Pkg.jl#2515

[thomasjm]

Thanks, interesting stuff. I know there's an effort going on to get 1.6 merged into Nixpkgs master, once that's working seems like a good time to investigate this.

[balodja]

There is julia 1.6 in nixpkgs for some time already. I have tried to use julia2nix with it and failed. Should I paste a separate issue?

[thomasjm]

Thanks for the reminder, let's just use this issue to track it.

[djacu]

I am getting some strange errors when trying to run julia2nix on a depot created with Julia 1.6.

Creating the depot.

[bakerdn@tweag-laptop:~/tweag/jupyterWith/depot]$ nix run nixpkgs#julia_16-bin
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.6 (2022-03-28)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.6) pkg> activate .
  Activating new environment at `~/tweag/jupyterWith/depot/Project.toml`

(depot) pkg> add IJulia
  Installing known registries into `~/.julia`
       Added registry `General` to `~/.julia/registries/General`
   Resolving package versions...
    Updating `~/tweag/jupyterWith/depot/Project.toml`
  [7073ff75] + IJulia v1.23.3
    Updating `~/tweag/jupyterWith/depot/Manifest.toml`
  [8f4d0f93] + Conda v1.7.0
  [7073ff75] + IJulia v1.23.3

Running julia2nix on the depot.

[bakerdn@tweag-laptop:~/tweag/jupyterWith/depot]$ ~/tweag/julia2nix/julia2nix 
nix-pre-commit-hooks: updating /home/bakerdn/tweag/jupyterWith/depot repo
pre-commit uninstalled
pre-commit installed at .git/hooks/pre-commit
nix-pre-commit-hooks: hooks up to date
Cloning https://github.com/JuliaRegistries/General.git to /tmp/nix-shell.2ChYH5/nix-shell.37R8ZN/tmpvvassuli/registry
Got registry revision c12e5d509ef3f49c5866d19d03c7d455758e98c6
Got registry sha256 0k5cara24b8fl8xzzf83qlybfcl6p41frysgfl8jwbidrkz9s82h
Failed to nix-prefetch-git for package Base64 (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Dates (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package FileWatching (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package InteractiveUtils (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package ArgTools (url = https://github.com/JuliaIO/ArgTools.jl.git, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Downloads (url = https://github.com/JuliaLang/Downloads.jl.git, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Artifacts (url = https://github.com/JuliaPackaging/Artifacts.jl.git, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Logging (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Markdown (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package LibCURL_jll (url = https://github.com/JuliaBinaryWrappers/LibCURL_jll.jl.git, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package LibGit2 (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package LibSSH2_jll (url = https://github.com/JuliaBinaryWrappers/LibSSH2_jll.jl.git, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Libdl (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Mmap (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package MozillaCACerts_jll (url = https://github.com/JuliaBinaryWrappers/MozillaCACerts_jll.jl.git, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package NetworkOptions (url = https://github.com/JuliaLang/NetworkOptions.jl.git, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Pkg (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Printf (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package REPL (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Random (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package SHA (url = https://github.com/JuliaCrypto/SHA.jl.git, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Serialization (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Sockets (url = None, githash = None). Hopefully it's built-in?
error: Package ‘julia-1.5.4’ in /nix/store/hnwfkyq66f7favb9rp2xb1ijrql9lvcs-nixos-21.11/nixos/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
Failed to nix-prefetch-git for package Tar (url = https://github.com/JuliaIO/Tar.jl.git, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Test (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package UUIDs (url = None, githash = None). Hopefully it's built-in?
Failed to nix-prefetch-git for package Unicode (url = None, githash = None). Hopefully it's built-in?
error: Package ‘julia-1.5.4’ in /nix/store/hnwfkyq66f7favb9rp2xb1ijrql9lvcs-nixos-21.11/nixos/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
error: Package ‘julia-1.5.4’ in /nix/store/hnwfkyq66f7favb9rp2xb1ijrql9lvcs-nixos-21.11/nixos/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
error: Package ‘julia-1.5.4’ in /nix/store/hnwfkyq66f7favb9rp2xb1ijrql9lvcs-nixos-21.11/nixos/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
Failed to nix-prefetch-git for package p7zip_jll (url = https://github.com/JuliaBinaryWrappers/p7zip_jll.jl.git, githash = None). Hopefully it's built-in?
error: Package ‘julia-1.5.4’ in /nix/store/hnwfkyq66f7favb9rp2xb1ijrql9lvcs-nixos-21.11/nixos/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
error: Package ‘julia-1.5.4’ in /nix/store/hnwfkyq66f7favb9rp2xb1ijrql9lvcs-nixos-21.11/nixos/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
error: Package ‘julia-1.5.4’ in /nix/store/hnwfkyq66f7favb9rp2xb1ijrql9lvcs-nixos-21.11/nixos/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
error: Package ‘julia-1.5.4’ in /nix/store/hnwfkyq66f7favb9rp2xb1ijrql9lvcs-nixos-21.11/nixos/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
error: Package ‘julia-1.5.4’ in /nix/store/hnwfkyq66f7favb9rp2xb1ijrql9lvcs-nixos-21.11/nixos/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
error: Package ‘julia-1.5.4’ in /nix/store/hnwfkyq66f7favb9rp2xb1ijrql9lvcs-nixos-21.11/nixos/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
error: Package ‘julia-1.5.4’ in /nix/store/hnwfkyq66f7favb9rp2xb1ijrql9lvcs-nixos-21.11/nixos/pkgs/development/compilers/julia/1.5.nix:138 is marked as broken, refusing to evaluate.

       a) To temporarily allow broken packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_BROKEN=1

        Note: For `nix shell`, `nix build`, `nix develop` or any other Nix 2.4+
        (Flake) command, `--impure` must be passed in order to read this
        environment variable.

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowBroken = true; }
       in configuration.nix to override this.

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowBroken = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/bakerdn/tweag/julia2nix/gather_packages.py", line 85, in process_item
    artifacts = subprocess.check_output([script_dir.joinpath("extract_artifacts.jl"), src]).decode()
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '[PosixPath('/home/bakerdn/tweag/julia2nix/extract_artifacts.jl'), 'fetchgit { url = "https://github.com/stevengj/SoftGlobalScope.jl.git"; rev = "986ec2b6162ccb95de5892ed17832f95badf770c"; sha256 = "1n3l0al1vw5jpb4k9a29a71666cdb617nmiqg34wcmyfzrxpvv39"; }']' returned non-zero exit status 1.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/bakerdn/tweag/julia2nix/gather_packages.py", line 107, in <module>
    formatted_dicts = p.map(process_item, manifest.keys())
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
subprocess.CalledProcessError: Command '[PosixPath('/home/bakerdn/tweag/julia2nix/extract_artifacts.jl'), 'fetchgit { url = "https://github.com/stevengj/SoftGlobalScope.jl.git"; rev = "986ec2b6162ccb95de5892ed17832f95badf770c"; sha256 = "1n3l0al1vw5jpb4k9a29a71666cdb617nmiqg34wcmyfzrxpvv39"; }']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/bakerdn/tweag/julia2nix/julia2nix", line 45, in <module>
    main()
  File "/home/bakerdn/tweag/julia2nix/julia2nix", line 40, in main
    packages_nix = subprocess.check_output([script_dir.joinpath("gather_packages.py"), environment_dir, repo_url, str(workers)]).decode()
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/nix/store/nvxp3xmlrxj9sw66dk7l0grz9m4889jn-python3-3.9.12/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '[PosixPath('/home/bakerdn/tweag/julia2nix/gather_packages.py'), PosixPath('.'), 'https://github.com/JuliaRegistries/General.git', '8']' returned non-zero exit status 1.

[thomasjm]

@djacu it looks like you're trying to build with Julia 1.5, which is broken in Nixpkgs. I'd recommend you try the latest version, which should make it easier to get this right.

@colinxs: Julia 1.6 should work fine in general now (and is even tested in CI!). But I think I'll leave this open to track the select_artifacts.jl work. There is approximately 1 more line to fill in:

julia2nix/extract_artifacts.jl

Line 25 in b5b02b0

# TODO: add these extra artifacts to artifact_dict so they get processed below

. Do you happen to know of a package that uses select_artifacts.jl?