Integrating pacquet to pnpm's existing codebase

[KSXGitHub]

Rewriting all pnpm functionalities from scratch in pacquet is a huge task, so I think we should integrate with pnpm. Let pacquet handle tasks sensitive to performance and correctness, and pnpm the rest. It is also important to test pacquet against pnpm's tests and real world use cases. This would hopefully allow us to eventually convert the codebase to Rust (if we want to) without creating too many regressions or changing too many existing quirks.

How will we going to approach this? Should it be a separate process that communicate via IPC? Should it be a child process? Should it be a Node.js addon? Should it be a WASM library? Or should pacquet spawn pnpm as a child process instead?

@anonrig @zkochan

[steveklabnik]

Chiming in with my two cents, even though I know nothing about pnpm :D

I think it's important to get clarity on what the objective is here, specifically. The one brought up is

It is important to test pacquet against pnpm's tests and real world use cases.

This is a decent starting point, but like, is the goal here correctness of pacquet on its own? is the goal to eventually have paquet be an actual part of the pnpm "upstream" release?

This matters because I'd make different tradeoffs in each of these cases. for example:

Should it be a Node.js addon?

These are both decent options, but like all FFI, has some amount of overhead at the boundary. And I know perf matters to this project, generally. This style approach tends to work best if you can minimize the number of times you call back and forth between the two. So like, "we do argument parsing in js and then invoke pacquet to do all the work," that kind of thing could work well. But once you start to call back and forth, it may be more trouble than it's worth.

This would also require users to have the rust toolchain installed, which matters a lot more if the intent is to have this in upstream.

Should it be a WASM library?

This has the same shape as the node addon with the additional issue that you want to be making network and filesystem calls, and therefore are in the realm of wasi. you could structure this so that users don't need a rust toolchain involved, which is a pro.

Should it be a separate process that communicate via IPC?

This strategy is decent, but has an even bigger "back and forth across the gap" overhead.

One that's not mentioned would be something like what many CLIs do: cli foo invokes a program named cli-foo. this is sort of like the IPC strategy, except there's no, well, inter-proces communication, haha. This would require more of each command to be implemented before it works, but would have the least overhead overall.

[KSXGitHub]

This is a decent starting point, but like, is the goal here correctness of pacquet on its own? is the goal to eventually have paquet be an actual part of the pnpm "upstream" release?

Thank you for your question. I have updated the issue description.

[anonrig]

I think we need to decouple pnpm tests and make them language agnostic. Later we can have the same test of integration tests run on both of the CLIs.

[KSXGitHub]

@anonrig I don't think it is either necessary or possible to go as far as "language agnostic", just make the e2e tests independent from pnpm is fine. By "independent", I mean the tests shouldn't import functions as test subjects, instead, it should spawn pnpm or pacquet as a subprocess.

[zkochan]

We have a registry-mock for e2e tests which can be for sure used in the pacquet tests.

We have some e2e tests that test the pnpm CLI directly. Those could be shared potentially. But I don't think they cover a lot of the functionality. Most of the tests are package level. Like in the @pnpm/core package for instance.

In my opinion the best way would be to start with writing a custom tarball fetcher for pnpm. A tarball fetcher is relatively easy to implement. I have already worked on a PoC, see the related issue: pnpm/pnpm#6663. If we can create a custom fetcher that will make pnpm faster (it won't be easy as I worked on some big performance improvements recently), then this part of pacquet will be tested by the hundreds of tests in the pnpm/pnpm repository. So basically we can have some code production ready and used by thousand of our existing users relatively soon.