Global binary links/shims

[pitaj]

Is your feature request related to a problem?

xpm creates a different global directory for every version of an xpack. This makes running and upgrading globally installed binaries difficult.

Describe the solution you'd like

An explicit global install (xpm install --global abc) should link/shim the various binaries within the xpack into the global xPacks/.bin

Describe alternatives you've considered

One alternative is to require an additional flag to add the global link/shim.

Additional context and comments

Currently I have to change my PATH every time I upgrade arm-gcc, for instance.

I'm on Windows.

[ilg-ul]

Please read the documentation on how to use xpm to install the toolchain for each project:

• https://xpack.github.io/blog/2021/11/12/arm-none-eabi-gcc-v10-3-1-2-2-released/

[ilg-ul]

xpm creates a different global directory for every version of an xpack

xPacks allow, by design, to install multiple versions of a package and to lock each project to a specific version.

This makes running and upgrading globally installed binaries difficult.

Installing globally was the initial method, for use by Eclipse, but it is now deprecated in favour of installing locally, for each project.

link/shim the various binaries within the xpack into the global xPacks/.bin

This is exactly what installing locally does, it adds links/shims from <project>/xpacks/.bin to the global store.

Even more, you can define separate dependencies for each build configuration, for example you can build a configuration with the Arm toolchain, and another with the RISC-V toolchain. For this to work, xpm adds links/shims to each build/<configuration>/xpacks/.bin folder.

You can see a functional example in the hello-world project template.

[pitaj]

I understand that you don't want to support installing globally. Do you plan on deprecating and eventually removing --global?

[ilg-ul]

I do support installing globally, and I don't think --global should be removed, it is still used by Eclipse, which is aware of the global store location and can manage the paths itself.

But, since dependencies are defined per project or per build configuration, the recommended method is to install locally, which, in addition to installing in the global store, adds the links/shims. Plus that if you use xpm run to start actions, <project>/xpacks/.bin is automatically added in front of the existing path.

I can add links/shims to a global folder, but you loose the ability to control which versions are used. Is this really what you want?

Can you help me understand what are the arguments in favour of using global (un-versioned, in a folder that you must manually add to the path) vs project local (versioned, in a folder automatically added to the path) dependencies?

[pitaj]

One example in favor of global dependencies: if you want, as a user, to have access to the commands in windows-build-tools in your terminal.

Personally, I agree that local dependencies are better. In fact you convinced me to switch to local dependencies for my own project here. But it seems weird to support explicitly installing xPacks globally, but to provide no easy way to use those global packs.

Should eclipse not move towards using local xPacks?

[ilg-ul]

access to the commands ... in your terminal

I initially considered this, but I have a big problem with adding these commands to a global path, since they can be picked by various build tools by accident, and result in non-reproducible builds.

This can be solved by adding an optional --link flag, to add the links/shims only upon request, but I think that by default these links should not be created.

What happens on Windows if you install a global npm package, do you get the shims? If so, in which folder? (npm was the inspiration for xpm and preferably the behaviour should be similar).

BTW, the initial design also included a way to install in system locations (--system), but it was not implemented, one of the reasons being the complication introduced by these links.

Should Eclipse not move towards using local xPacks?

Probably it should, but interest for it is declining anyway, and I'm migrating everything to Visual Studio Code, which already is able to use xPacks.

[pitaj]

What happens on Windows if you install a global npm package, do you get the shims? If so, in which folder? (npm was the inspiration for xpm and preferably the behaviour should be similar).

Global modules are installed under %appdata%/npm/node_modules and shims are added to %appdata%/npm which is added to the PATH. This enables the recommended upgrade path for npm: npm install -g npm@latest

[ilg-ul]

Aha. So there is a precedent. In this case we'll do something similar. I have to think on the details, since each platform is slightly different.

[ilg-ul]

How about xpm install --global --links-into-folder <folder-path> ... to create the links/shims in the given folder?

[pitaj]

I think if you want to support linking multiple different versions of an xpack to different places, that would be acceptable. I'd probably call it just --link or --link-into though.

But a simpler and more npm-like approach would be to use a single directory like %appdata%/xPacks/.bin. You could support both approaches I guess. If no directory is supplied, use xpacks/.bin by default.

[ilg-ul]

My main concern is how to prevent global installs interfere with other builds, in other words (unless explicitly instructed by the user), there should be no links in system locations.

A path like %appdata%/xPacks/.bin is safe on Windows, since there is no weird CMake script to search for anything there, but similar locations should be identified on macOS and GNU/Linux too.

But things are not that simple, if a safe path like %appdata%/xPacks/.bin is used, you'll probably add it to the user PATH, and then everything you install globally will be available for everybody, so it is no longer safe.

On my Mac I never add anything to the user PATH, and I always install in custom folders. Then I have shell aliases which I use to temporarily extend the PATH of the current shell with the desired folders, and in this way I have total control on various build environments.

I know that on Windows the PATH management is a nightmare, with lots of folders; in such environments installing multiple versions of the same tool is a gamble, depending on path order.

Thus my first idea was to do not add links by default. Those who want the links should ask for them explicitly, and should take responsibility for the PATH.

Another detail is that while there are a lot of general purpose npm packages, that can be installed in a system location, and they are usually not a danger for other builds, the xpm binary packages are all tools related to builds, so they present a higher risk.

This will require some thinking to sort things out...

[ilg-ul]

The time has come for some more work on xpm, and I'm considering this feature too.

How about --add-links <folder-path>?