Guidance on using private packages and registries #42

jademackay · 2022-10-30T19:00:27Z

Hello,

This might be more of a question about using Nix itself.

My Project /Manifest includes a package QuxPrivate that resides in a private repository. Running

nix run github:JuliaCN/Julia2Nix.jl#packages.x86_64-linux.julia2nix

results in error:

  Activating project at `/nix/store/z1x616mawlfjsljxsmbl16nc7jrhwwnq-incl`
[ Info: Fetching registries...
[ Info: Fetching packages...
Failed to run cmd:===============>                   ]  17/33
`nix-prefetch '    { nixpkgs ? <nixpkgs> }:
    let pkgs = import nixpkgs { };
    in with pkgs; pkgs.fetchgit
' --hash-algo sha256 --output raw --name package-QuxPrivate --rev 2bf158ea22deb09c0da06783d51ad6651f83f9b0 --url [email protected]:dejakaymac/QuxPrivate.jl.git`
Error:

The fetcher will be called as follows:
> fetchgit {
>   name = "package-QuxPrivate";
>   rev = "2bf158ea22deb09c0da06783d51ad6651f83f9b0";
>   sha256 = "sha256:0000000000000000000000000000000000000000000000000000";
>   url = "[email protected]:dejakaymac/QuxPrivate.jl.git";
> }

exporting [email protected]:dejakaymac/QuxPrivate.jl.git (rev 2bf158ea22deb09c0da06783d51ad6651f83f9b0) into /nix/store/ci5gpc6736l0674h163p7cvng1f1b0x9-package-QuxPrivate
Initialized empty Git repository in /nix/store/ci5gpc6736l0674h163p7cvng1f1b0x9-package-QuxPrivate/.git/
error: cannot run ssh: No such file or directory
fatal: unable to fork
error: cannot run ssh: No such file or directory
fatal: unable to fork
error: cannot run ssh: No such file or directory
fatal: unable to fork
Unable to checkout 2bf158ea22deb09c0da06783d51ad6651f83f9b0 from [email protected]:dejakaymac/QuxPrivate.jl.git.
error: builder for '/nix/store/gz6vli90d63pcv8x029lw4nd5wdp5nq4-package-QuxPrivate.drv' failed with exit code 1;
       last 9 log lines:
       > exporting [email protected]:dejakaymac/QuxPrivate.jl.git (rev 2bf158ea22deb09c0da06783d51ad6651f83f9b0) into /nix/store/ci5gpc6736l0674h163p7cvng1f1b0x9-package-QuxPrivate
       > Initialized empty Git repository in /nix/store/ci5gpc6736l0674h163p7cvng1f1b0x9-package-QuxPrivate/.git/
       > error: cannot run ssh: No such file or directory
       > fatal: unable to fork
       > error: cannot run ssh: No such file or directory
       > fatal: unable to fork
       > error: cannot run ssh: No such file or directory
       > fatal: unable to fork
       > Unable to checkout 2bf158ea22deb09c0da06783d51ad6651f83f9b0 from [email protected]:dejakaymac/QuxPrivate.jl.git.
       For full logs, run 'nix log /nix/store/gz6vli90d63pcv8x029lw4nd5wdp5nq4-package-QuxPrivate.drv'.


┌ Error: Fetcher failed: nix-prefetch pkgs.fetchgit --hash-algo sha256 --output raw --name package-QuxPrivate --rev 2bf158ea22deb09c0da06783d51ad6651f83f9b0 --url [email protected]:dejakaymac/QuxPrivate.jl.git
│ failed process: Process(`nix-prefetch '    { nixpkgs ? <nixpkgs> }:
│     let pkgs = import nixpkgs { };
│     in with pkgs; pkgs.fetchgit
│ ' --hash-algo sha256 --output raw --name package-QuxPrivate --rev 2bf158ea22deb09c0da06783d51ad6651f83f9b0 --url [email protected]:dejakaymac/QuxPrivate.jl.git`, ProcessExited(1)) [1]
│
│ Stacktrace:
│  [1] select_fetcher(fetchers::Vector{Julia2Nix.JuNix.Fetcher}, opts::Julia2Nix.JuNix.Options)
│    @ Julia2Nix.JuNix /nix/store/z1x616mawlfjsljxsmbl16nc7jrhwwnq-incl/src/JuNix/fetch.jl:219
│  [2] macro expansion
│    @ /nix/store/z1x616mawlfjsljxsmbl16nc7jrhwwnq-incl/src/JuNix/fetch.jl:170 [inlined]
│  [3] (::Julia2Nix.JuNix.var"#12#14"{Julia2Nix.JuNix.Options, Channel{Tuple{Julia2Nix.JuNix.PackageInfo, Union{Nothing, Julia2Nix.JuNix.Fetcher}}}, Channel{Tuple{Julia2Nix.JuNix.PackageInfo, Vector{Julia2Nix.JuNix.Fetcher}}}})()
│    @ Julia2Nix.JuNix ./task.jl:484
└ @ Julia2Nix.JuNix /nix/store/z1x616mawlfjsljxsmbl16nc7jrhwwnq-incl/src/JuNix/fetch.jl:220

I think I need to communicate my credentials to julia2nix but am not sure how to do so.

The text was updated successfully, but these errors were encountered:

GTrunSec · 2022-10-31T02:32:45Z

Looks like we should use nix-fetcher-url and nix-fetcher-git instead. \CC @colinxs

nix shell nixpkgs#nix-prefetch-git --command nix-prefetch-git --url [email protected]:dejakaymac/QuxPrivate.jl.git

hash require:

nix-prefetch-git --url [email protected]:GTrunSec/haskell-roadmap.git --quiet | jq ".sha256"

GTrunSec · 2022-10-31T02:37:38Z

@jademackay Since I am a beginner of julia lang, I don't know if you are willing to migrate nix-prefecturl to nix-prefetch-url/git.

jademackay · 2022-10-31T23:37:50Z

Thanks @GTrunSec , I'll have a go.

jademackay · 2023-01-09T02:55:25Z

Hi @GTrunSec , having made your suggested changes I can now generate a julia2nix.toml successfully. However, I am having difficulty actually using it. I have a project using the Julia2Nix flake template:

julia2nix-pkgs
├── flake.lock
├── flake.nix
├── julia2nix.toml
├── Manifest.toml
└── Project.toml

The flake.nix is basically unchanged and has:

...
in {
   packages = {
     # make sure you have generated the julia2nix.toml
     default = project;
   };
   devShells.default = pkgs.devshell.mkShell {
     imports = [
       # you can keep either one of them devshellProfiles.packages or julia-wrapped
       # inputs.julia2nix.${pkgs.system}.julia2nix.devshellProfiles.packages

       # add dev-tools in your devshell
       inputs.julia2nix.${pkgs.system}.julia2nix.devshellProfiles.dev

       # add nightly julia
       # inputs.julia2nix.${pkgs.system}.julia2nix.devshellProfiles.nightly
     ];
     commands = [
       {
         # package = julia-wrapped;
         package = project;
         help = julia2nix.packages.${pkgs.system}.julia_18-bin.meta.description;
       }
     ];
   };
 })
...

Running nix develop fails with:

warning: input 'flake-utils' has an override for a non-existent input 'nixpkgs'
error: hash '"1csa0yzkwnkj4aa141g9cl6ajk4brxz4bg1zlh84rzhfjx1v7r1g"' has wrong length for hash type 'sha256'

Any thoughts welcome.

GTrunSec · 2023-01-09T06:47:48Z

looks like you got an error hash format. Could you share your julia2nixtoml file with me? For now, we have a better choice to fetch the sources by https://github.com/nix-community/nurl. Could you try it as well? thanks

jademackay · 2023-01-09T09:39:09Z

Ah I see, my julia2nix.toml is bad, thanks. Sure, I will take a look at nurl.

jademackay · 2023-01-09T20:44:32Z

OK, I have adjusted my Julia2Nix.jl fetch_sha256 so that it produces a correct julia2nix.toml. The the project flake (above) now fails with:

error: builder for '/nix/store/m87jqn6lrvji2c4i537m7ay22nrdzwhm-packages-QuxPrivate-Stkot.drv' failed with exit code 1;
       last 9 log lines:
       > exporting [email protected]:dejakaymac/QuxPrivate.jl.git (rev 2bf158ea22deb09c0da06783d51ad6651f83f9b0) into /nix/store/inlyzrfjpvlwr7pkma1xvvam174739y6-packages-QuxPrivate-Stkot
       > Initialized empty Git repository in /nix/store/inlyzrfjpvlwr7pkma1xvvam174739y6-packages-QuxPrivate-Stkot/.git/
       > error: cannot run ssh: No such file or directory
       > fatal: unable to fork
       > error: cannot run ssh: No such file or directory
       > fatal: unable to fork
       > error: cannot run ssh: No such file or directory
       > fatal: unable to fork
       > Unable to checkout 2bf158ea22deb09c0da06783d51ad6651f83f9b0 from [email protected]:dejakaymac/QuxPrivate.jl.git.
       For full logs, run 'nix log /nix/store/m87jqn6lrvji2c4i537m7ay22nrdzwhm-packages-QuxPrivate-Stkot.drv'.
error: 1 dependencies of derivation '/nix/store/58l3wbcma8sb4zkjgbmx020xmpmdmcfv-package-QuxPrivate.drv' failed to build
error: 1 dependencies of derivation '/nix/store/b2jmln207vpvlkx099p6hcc1b6j4x8sm-julia-depot.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qivl4xab1lyz72apawmsv3xwc5i6kq2k-julia-baz.drv' failed to build
error: 1 dependencies of derivation '/nix/store/4pvfg2ajnyidn8rkv0f3a8agamzv1x0f-devshell-dir.drv' failed to build
error: 1 dependencies of derivation '/nix/store/7a2jp0cjars13k3fzd15m25k76q0h8lq-devshell-env.drv' failed to build

The julia2nix.toml is:

[depot.x86_64-linux.fetchzip.registry-General]
name = "registries/General"
sha256 = "sha256-hPxVGsYtYf89xyF1LzbF3EjyIGNuSp6slRhlh9AwSsw="
stripRoot = false
url = "https://pkg.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/2262acf6a53c83300b5d2fd2d6d03ed7bed1673f#registry.tar.gz"

[depot.x86_64-linux.fetchgit.package-QuxPrivate]
name = "packages/QuxPrivate/Stkot"
sha256 = "0vcr9z1hqjc30hibi1gr4v2jfmrjkyf1618qyg12bq9cp190i5f5"
rev = "2bf158ea22deb09c0da06783d51ad6651f83f9b0"
url = "[email protected]:dejakaymac/QuxPrivate.jl.git"

[depot.x86_64-linux.fetchgit.package-Example]
name = "packages/Example/aqsx3"
sha256 = "1vrwpk57jv6rqyfgl9m23fg1x0q6xny4fs90hsk4d2048wax9v96"
rev = "46e44e869b4d90b96bd8ed1fdcf32244fddfb6cc"
url = "https://github.com/JuliaLang/Example.jl.git"

GTrunSec · 2023-01-09T21:38:16Z

You should add SSH_AUTH_SOCK env to nix-daemon.
or

SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) nix build .#<>

# do we need the arg of -I ? I'm not sure.
SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) nix build .#<> -I $(gpgconf --list-dirs agent-ssh-socket)

https://discourse.nixos.org/t/how-to-set-up-a-system-wide-ssh-agent-that-would-work-on-all-terminals/14156

jademackay · 2023-01-09T21:56:50Z

Hmmm

 SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) nix build .\#packages.x86_64-linux.default -I $(gpgconf --list-dirs agent-ssh-socket)

Results in same error as above.

GTrunSec · 2023-01-09T21:59:39Z

let me test it.

jademackay · 2023-01-09T22:41:45Z

Cool. I note that using public urls of the type [email protected]:owner/Package.jl works fine, suggesting the issue is, as you imply, due to authorisation.

GTrunSec · 2023-01-09T23:44:12Z

You must modify the fetch type if the URL is private.

<pkgs.fetchgit> -> <builtins.fetchGit >
can be sure this works for me.

builtins.fetchGit {                                                                                          
+   1 │ │ "url" = "[email protected]:GTrunSec/my-learning-rust.git";
+   2 │ │ "rev" = "641521fb246853409377dd8a9a45c8f124698ead";
+   3 │ };

GTrunSec mentioned this issue Aug 4, 2023

Issues with build #52

Closed

GTrunSec mentioned this issue Nov 1, 2023

Refactor discussion #62

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Guidance on using private packages and registries #42

Guidance on using private packages and registries #42

jademackay commented Oct 30, 2022

GTrunSec commented Oct 31, 2022 •

edited

Loading

GTrunSec commented Oct 31, 2022

jademackay commented Oct 31, 2022 •

edited

Loading

jademackay commented Jan 9, 2023 •

edited

Loading

GTrunSec commented Jan 9, 2023

jademackay commented Jan 9, 2023

jademackay commented Jan 9, 2023

GTrunSec commented Jan 9, 2023 •

edited

Loading

jademackay commented Jan 9, 2023

GTrunSec commented Jan 9, 2023

jademackay commented Jan 9, 2023 •

edited

Loading

GTrunSec commented Jan 9, 2023

Guidance on using private packages and registries #42

Guidance on using private packages and registries #42

Comments

jademackay commented Oct 30, 2022

GTrunSec commented Oct 31, 2022 • edited Loading

GTrunSec commented Oct 31, 2022

jademackay commented Oct 31, 2022 • edited Loading

jademackay commented Jan 9, 2023 • edited Loading

GTrunSec commented Jan 9, 2023

jademackay commented Jan 9, 2023

jademackay commented Jan 9, 2023

GTrunSec commented Jan 9, 2023 • edited Loading

jademackay commented Jan 9, 2023

GTrunSec commented Jan 9, 2023

jademackay commented Jan 9, 2023 • edited Loading

GTrunSec commented Jan 9, 2023

GTrunSec commented Oct 31, 2022 •

edited

Loading

jademackay commented Oct 31, 2022 •

edited

Loading

jademackay commented Jan 9, 2023 •

edited

Loading

GTrunSec commented Jan 9, 2023 •

edited

Loading

jademackay commented Jan 9, 2023 •

edited

Loading