support package dependencies via paths relative to the build root

[TUSF]

zig build should be able to fetch from a relative path such as:

    .url = "../myproject/",

or a file URI like:

    .url = "file:///C:/Code/myproject/"

I think this would mostly be useful for when operating on local forks of code that aren't published anywhere else, without needing to explicitly vendor those dependencies into the same folder. (Although perhaps this might make vendored dependencies also easier to work with using the build system?)

[nektro]

this is a rabbit hole of un-reproducibile builds and an anti-pattern

I think this would mostly be useful for when operating on local forks of code that aren't published anywhere else

this should be possible once the git, hg, svn, etc backends are supported

[MoAlyousef]

Could you elaborate on how this is an antipattern?

This is supported by CMake (FetchContent and ExternalProject_Add) and Cargo (path) for example.

[andrewrk]

There is no need to debate whether the file:/// scheme will work; it is planned. The discussion is about how it should work. Some potential topics for discussion:

• How to handle relative paths (relative to build.zig probably)
• Should environment variables be expanded? If so, what syntax for this?
• Potential footguns, and ideas to guide the user away from triggering them.

As for implementation, here is where it parses the URI and then is hard-coded to use the http client:

zig/src/Package.zig

Lines 334 to 355 in 37424fd

	const uri = try std.Uri.parse(url);
	const rand_int = std.crypto.random.int(u64);
	const tmp_dir_sub_path = "tmp" ++ s ++ hex64(rand_int);
	const actual_hash = a: {
	var tmp_directory: Compilation.Directory = d: {
	const path = try global_cache_directory.join(gpa, &.{tmp_dir_sub_path});
	errdefer gpa.free(path);
	const iterable_dir = try global_cache_directory.handle.makeOpenPathIterable(tmp_dir_sub_path, .{});
	errdefer iterable_dir.close();
	break :d .{
	.path = path,
	.handle = iterable_dir.dir,
	};
	};
	defer tmp_directory.closeAndFree(gpa);
	var req = try http_client.request(uri, .{}, .{});
	defer req.deinit();

Instead, it should check the scheme of the URI after parsing it and decide what to do.

[AdamGoertz]

If no one else is working on this, I'd like to take a stab at it. I think there's plenty of room to get started on this without having to answer all questions about relative paths and environment variables.

The main trade-off I see with the questions that Andrew has raised is how much we prioritize making it easy to use file:/// URIs versus making it portable and easy to depend on. A user who is writing their own build.zig.zon can easily choose the a URI that works for their system, but dependencies buried several layers deep in the dependency tree are the main concern.

My take on the questions Andrew raised:

• Relative paths are relative to build.zig
• I would say 'no' to expanding environment variables, simply because they aren't portable across operating systems. If we did choose to support this, we'd need a syntax for expanding them that can't appear in a valid URI. Probably something like {VAR}, since afaik {} can't appear in a URI and it's a pretty common choice of syntax.

[mitchellh]

I'm just sharing my use case here: monorepos. I sometimes will use a monorepo for my projects, particularly when I'm still "baking" a package I don't feel is ready to fully extract. I cleanly separate my packages out into a separate dir like pkg with its own dedicated build.zig so that its ready to be extracted one day if I want to.

It'd be nice to represent these in the build.zig.zon file somehow. Right now I just import their build.zig directly.

Here is the current folder layout of a project I'm working on. The folders in pkg/ are extractable standalone Zig packages but I just haven't extracted them yet. They may depend on each other (adjacent folders).

.
├── pkg
│   ├── fontconfig
│   ├── freetype
│   ├── harfbuzz
│   ├── <etc...>
├── src
│   ├── <main app zig files>
└── vendor
    ├── <git submodules>

[praschke]

absolute paths do not make sense across machines, let alone across operating systems. additionally, file:// only supports absolute paths; relative paths are not URIs.

1. why is this being implemented as .url = "../myproject/", and not .path = "../myproject/",? at least for relative paths?

2. what good reason is there for absolute paths to be supported?

3. .url = "file:///etc/shadow",

this can easily become a supply chain attack, targeted or shotgun, and you cannot make a list of every user's secret files. please do not tell me that packages will have to have zig in them, or the package manager won't be a replacement for git submodules. #14488

[tauoverpi]

It would be best if paths were all relative to build.zig used in the zig build invocation without the ability to reference directories above it. This can later be enforced for build.zig aswell with #14286

[jiacai2050]

It would be best if paths were all relative to build.zig used in the zig build invocation without the ability to reference directories above it.

This sounds counter-intuitive, the path should be relative to its package root, not where zig build is invoked.

This is how go mod's replace works

• https://go.dev/doc/modules/gomod-ref#replace

[AdamGoertz]

It would be best if paths were all relative to build.zig used in the zig build invocation without the ability to reference directories above it.

This sounds counter-intuitive, the path should be relative to its package root, not where zig build is invoked.

This is how go mod's replace works

• https://go.dev/doc/modules/gomod-ref#replace

I agree with jiacai2050 that relative paths should be relative to the package root, which is currently how it is implemented in my open PR.

As for whether or not absolute paths should be allowed, I’m torn. I agree with the sentiment that in most cases it’s a bad practice to use absolute paths, but I’m not convinced that’s sufficient justification to disallow them.

I’m curious if anyone has a scenario in mind where absolute paths would be especially useful. I haven’t yet come up with a compelling use-case myself (just vague notions), but that doesn’t mean no such use-case exists.

[jiacai2050]

As for whether or not absolute paths should be allowed, I’m torn.

Thanks for your hard work, I think we need to wait maintainer's idea.

I’m curious if anyone has a scenario in mind where absolute paths would be especially useful.

I haven't seen absolute path used in go project, IMO if you are uncertain with this, you can leave this as TODO.

[rsepassi]

Excited for this to get in, I have a similar use case to mitchellh: a monorepo with multiple subpackages. wrt supply chain security and general sanity, seems like it'd be nice to control file:// deps in some way when building, e.g. disallowing them for a given dependency (tree), or only allowing relative paths that do not escape the root of the depended-on package (e.g. in the case of depending on a package within a monorepo), etc. Maybe the same control mechanism could be extended to URIs as well, e.g. {allow,block}-listing domains.

[ianprime0509]

Since #14603 introduces another way to declare the location of a dependency which is explicitly file-based (the path field, as opposed to the existing url field), perhaps it would make sense to relax the hash field requirement for path-based dependencies? That way, projects using path dependencies for a monorepo structure wouldn't need to update the hash value for every change in a sub-package, but it would remain required for url dependencies where it's important to verify the integrity of downloads and cache to avoid unnecessary network usage.

To clarify, by "relax", I mean to make the hash field optional for dependencies where the location is specified by path rather than url. A hash could still be provided, in which case it would be validated as it does in the current implementation.

[AdamGoertz]

I agree that there definitely needs to be some changes made for improving the ergonomics of using hashes for monorepo style projects. For example, given the current implementation of #14603 and the way in which modules are hashed, making a single change to a dependency that lives in a subdirectory of another the top-level project requires updating hashes 3 times: once for the outer project, once for the nested dependency, and again for the outer project.

This particular problem would be mitigated if we only hash files that belong to the module (there’s a TODO comment to this effect in Package.zig), but personally I don’t know how to go about doing this, and it’s tangential (though related) to this issue specifically.

An option that doesn’t involve relaxing the hash requirement would be to incorporate this use case into #14288, and add some tooling for automatically updating hashes so the user isn’t forced to do it themselves on every change. This would still likely require fixing the issue outlined above somehow.

Regarding the suggestion to relax the hash requirement for path dependencies, personally I think it’s reasonable, and would absolutely improve the user experience (at least up to the point where they encounter a versioning issue that a strict hash requirement would have revealed). I’d like some input from the people in charge of language/package manager design before proceeding down that route, however. @andrewrk @Vexu

[andrewrk]

File URI issue extracted into a separate issue: #17364