feat: `pixi project export conda` to export project to conda environment.yml's #1427

abkfenris · 2024-05-22T11:54:54Z

Adds an export command group, and a subcommand for exporting in Conda environment.yml files.

Currently it just exports the direct dependency names, but I could see adding flags to include all locked dependencies. It also currently does not export the versions, but that could also be a flag to select if it should export the specs as in the manifest, or as locked.

Similarly I could see export having subcommands for conda-lock and pip requirements, if not possibly some other formats.

❯ pixi export conda
name: pixi-default-osx-arm64
channels:
- conda-forge
dependencies:
- pre-commit
- rust
- openssl
- pkg-config
- git
- cffconvert
- tbump

❯ pixi export conda
name: xpublish-dev-default-osx-arm64
channels:
- conda-forge
dependencies:
- xarray
- nox
- netcdf4
- pytest
- uv
- jupyterlab
- ipython
- cf_xarray
- nodejs
- pydap
- intake-xarray
- rasterio
- asciitree
- pooch
- pytest-xprocess
- kerchunk
- scipy
- pip:
  - xpublish
  - xpublish_edr
  - xpublish_opendap
  - xpublish_intake_provider
  - xpublish_intake

works on #800

Adds an export command group, and a subcommand for exporting in Conda environment.yml files. Currently it just exports the direct dependency names, but I could see adding flags to include all locked dependencies. It also currently does not export the versions, but that could also be a flag to select if it should export the specs as in the manifest, or as locked. works on prefix-dev#800

ruben-arts · 2024-05-22T12:08:47Z

Hi @abkfenris, thanks for adding this!

I would like to include the full matchspec before we merge it. The full inclusion of the lockfile would be even more awesome but I agree this could be an iteration! Great work. Let me know when we can take a stab at reviewing this!

abkfenris · 2024-05-22T12:11:45Z

@ruben-arts cool! I wanted to get it out early, to make sure that an export group with subcommands made sense to others before trying to wrap my head around parsing the matchspecs.

ruben-arts · 2024-05-22T12:44:31Z

Oh that reminds me, would it make sense to hide it behind the pixi project command so the keep the highlevel CLI real-estate as clean as we can?
I could see this as a feature most often used by automated CI or when people search for this functionality, less of a daily run command.

abkfenris · 2024-05-22T13:00:29Z

Ya, pixi project export would make sense. Let me give that a shot.

ruben-arts · 2024-05-22T13:44:55Z

src/utils/conda_environment_file.rs

 }

-#[derive(Deserialize, Debug, Clone)]
+#[derive(Serialize, Deserialize, Debug, Clone)]
 #[serde(untagged)]
 pub enum CondaEnvDep {


I think if you implement a impl From<MatchSpec> for CondaEnvDep it should be pretty easy to add them.

You already have the PacakgeName and NamelessMatchSpec in the manifest.

So it should be easy to create a MatchSpec and use the .to_string() to get the actual string. e.g.:

let spec = MatchSpec::from_nameless(nameless_spec, Some(package_name));

I'm able to quickly get some of the specs to render ok with CondaEnvDep::Conda(format!("{}{}", name.as_source(), spec)), but a handful (*) aren't looking right.

For pixi's pixi.toml

name: pixi-default-osx-arm64 channels: - conda-forge dependencies: - pre-commit~=3.3.0 - rust~=1.77.0 - openssl3.* - pkg-config0.29.* - git2.42.0.* - cffconvert>=2.0.0,<2.1 - tbump>=6.9.0,<6.10

I haven't tried writing an impl yet.

It needs a space. But this is indeed basically it.

Without an impl that gets me closer CondaEnvDep::Conda(MatchSpec::from_nameless(spec, Some(name)).to_string()):

name: pixi-default-osx-arm64 channels: - conda-forge dependencies: - pre-commit ~=3.3.0 - rust ~=1.77.0 - openssl 3.* - pkg-config 0.29.* - git 2.42.0.* - cffconvert >=2.0.0,<2.1 - tbump >=6.9.0,<6.10

Wait, if there is a space and no leading operator, does that get interpreted as =?

Yes I believe so

abkfenris · 2024-05-22T16:22:38Z

I've got an initial implementation working with some PyPI dependency specs now, but it gets weird around editable requirements, but can those even be rendered in environment.yml?

name: xpublish-dev-default-osx-arm64
channels:
- conda-forge
dependencies:
- xarray >=2024.2.0,<2024.3
- nox >=2024.3.2,<2024.4
- netcdf4 >=1.6.5,<1.7
- pytest >=8.1.1,<8.2
- uv >=0.1.25,<0.2
- jupyterlab >=4.1.6,<4.2
- ipython >=8.22.2,<8.23
- cf_xarray >=0.9.0,<0.10
- nodejs >=20.12.2,<20.13
- pydap >=3.4.0,<3.5
- intake-xarray >=0.7.0,<0.8
- rasterio >=1.3.10,<1.4
- asciitree >=0.3.3,<0.4
- pooch >=1.8.1,<1.9
- pytest-xprocess >=1.0.1,<1.1
- kerchunk >=0.2.5,<0.3
- scipy >=1.13.0,<1.14
- pip:
  - 'xpublish = EditableRequirement { url: VerbatimUrl { url: Url { scheme: "file", cannot_be_a_base: false, username: "", password: None, host: None, port: None, path: "/Users/akerney/GMRI/xpublish-dev/xpublish", query: None, fragment: None }, given: Some("./xpublish") }, extras: [], path: "/Users/akerney/GMRI/xpublish-dev/xpublish" }'
  - 'xpublish-edr = EditableRequirement { url: VerbatimUrl { url: Url { scheme: "file", cannot_be_a_base: false, username: "", password: None, host: None, port: None, path: "/Users/akerney/GMRI/xpublish-dev/xpublish-edr", query: None, fragment: None }, given: Some("./xpublish-edr") }, extras: [], path: "/Users/akerney/GMRI/xpublish-dev/xpublish-edr" }'
  - 'xpublish-opendap = EditableRequirement { url: VerbatimUrl { url: Url { scheme: "file", cannot_be_a_base: false, username: "", password: None, host: None, port: None, path: "/Users/akerney/GMRI/xpublish-dev/xpublish-opendap", query: None, fragment: None }, given: Some("./xpublish-opendap") }, extras: [], path: "/Users/akerney/GMRI/xpublish-dev/xpublish-opendap" }'
  - 'xpublish-intake-provider = EditableRequirement { url: VerbatimUrl { url: Url { scheme: "file", cannot_be_a_base: false, username: "", password: None, host: None, port: None, path: "/Users/akerney/GMRI/xpublish-dev/xpublish-intake-provider", query: None, fragment: None }, given: Some("./xpublish-intake-provider") }, extras: [], path: "/Users/akerney/GMRI/xpublish-dev/xpublish-intake-provider" }'
  - 'xpublish-intake = EditableRequirement { url: VerbatimUrl { url: Url { scheme: "file", cannot_be_a_base: false, username: "", password: None, host: None, port: None, path: "/Users/akerney/GMRI/xpublish-dev/xpublish-intake", query: None, fragment: None }, given: Some("./xpublish-intake") }, extras: [], path: "/Users/akerney/GMRI/xpublish-dev/xpublish-intake" }'
  - tqdm~=4.66.4

ruben-arts · 2024-05-23T06:57:09Z

What I understood is that environments.yml s are just designed to execute pip. So an editable install look something like this:

dependencies:
- pip:
  - ./path/to/package -e
  - black ==1.2.3

abkfenris · 2024-05-23T12:31:19Z

I dug in and matched on the editables, and specifically formatted them. I haven't gotten to test yet against git+ and other forms of pip dependencies.

name: xpublish-dev-default-osx-arm64
channels:
- conda-forge
dependencies:
- xarray >=2024.2.0,<2024.3
- nox >=2024.3.2,<2024.4
- netcdf4 >=1.6.5,<1.7
- pytest >=8.1.1,<8.2
- uv >=0.1.25,<0.2
- jupyterlab >=4.1.6,<4.2
- ipython >=8.22.2,<8.23
- cf_xarray >=0.9.0,<0.10
- nodejs >=20.12.2,<20.13
- pydap >=3.4.0,<3.5
- intake-xarray >=0.7.0,<0.8
- rasterio >=1.3.10,<1.4
- asciitree >=0.3.3,<0.4
- pooch >=1.8.1,<1.9
- pytest-xprocess >=1.0.1,<1.1
- kerchunk >=0.2.5,<0.3
- scipy >=1.13.0,<1.14
- pip:
  - -e ./xpublish
  - -e ./xpublish-edr
  - -e ./xpublish-opendap
  - -e ./xpublish-intake-provider
  - -e ./xpublish-intake
  - tqdm~=4.66.4

vigneshmanick · 2024-05-24T06:02:05Z

Just stumbled upon this PR, awesome feature.

I have a question regarding the pip section, will the extra-index-url dependencies also be output? The example for this can be found at one of the tests at conda https://github.com/conda/conda/blob/main/tests/env/support/advanced-pip/environment.yml

abkfenris · 2024-05-24T12:16:02Z

@vigneshmanick I haven't tried with extra-index-urls or other configs like that yet, but that is a good reference to make sure I'm able to output all the different possibilities. It looks like it should be pretty easy to support them.

I'll try to import that and some of the other test environments from that repo and see if I can get them to round trip reasonably.

ruben-arts

I really like this PR more and more! I already gave it a review to hopefully get it in soon!

Some requirements for merging are:

documentation in reference/cli.md (possible adding it to a tutorial or an seperate "how to go back and forth between pixi and conda" would be a nice bonus)
A roundtrip test, possibly on pixi itself as that is a moving project so we can keep testing it.
A static test that starts with a full feature manifest, e.g. (different channels, different platforms, dependencies, pypi-dependencies, including -e)

ruben-arts · 2024-05-27T13:05:54Z

src/cli/project/export/conda.rs

+#[clap(arg_required_else_help = false)]
+pub struct Args {
+ /// The platform to list packages for. Defaults to the current platform.
+ #[arg(long)]


Suggested change

#[arg(long)]

#[arg(short, long)]

To align with other cli's.

ruben-arts · 2024-05-27T13:06:57Z

src/cli/project/export/conda.rs

+ #[arg(long, default_value = "manifest", value_enum)]
+ pub version_spec: VersionSpec,


Suggested change

#[arg(long, default_value = "manifest", value_enum)]

pub version_spec: VersionSpec,

#[arg(long, default="false")]

pub lock: bool,

What do you think about this? to keep it simple.

I explained more of my thinking in https://github.com/prefix-dev/pixi/pull/1427/files#r1616283963 but I think there are really 3 different ways that versions are specified in environment.ymls, so the arg needs to be able to handle all three.

How about shortening version_spec to versions?pixi project export conda --versions locked I think that may convey the intent better than version_spec to more users.

ruben-arts · 2024-05-27T13:08:02Z

src/cli/project/export/conda.rs

+ let channels = environment
+ .channels()
+ .into_iter()
+ .map(|channel| channel.name().to_string())


Suggested change

.map(|channel| channel.name().to_string())

.map(|channel| {

default_channel_config().canonical_name(channel.base_url()).to_string()

})

Otherwise the following channel would be exported as conda-forge:

channels = ["https://fast.prefix.dev/conda-forge"]

ruben-arts · 2024-05-27T13:09:51Z

src/cli/project/export/mod.rs

+
+#[derive(Debug, Parser)]
+pub enum Command {
+ #[clap(alias = "c")]


Suggested change

#[clap(alias = "c")]

#[clap(visible_alias = "c")]

ruben-arts · 2024-05-27T13:10:31Z

src/cli/project/export/mod.rs

+mod conda;
+
+#[derive(Debug, Parser)]
+pub enum Command {


Good that you prepared for future options!

Not gonna promise that I'm gonna be the one to write all of them though!

I dare you 😝

ruben-arts · 2024-05-27T13:11:51Z

src/cli/project/export/conda.rs

+ .into_specs()
+ .map(|(name, spec)| match args.version_spec {
+ VersionSpec::Manifest => {
+ CondaEnvDep::Conda(MatchSpec::from_nameless(spec, Some(name)).to_string())


Why would a user not want to add the given match spec?

I'm thinking about the range of ways that folks currently use environment.yml. I see a lot of environment.ymls passed around without any versions specified, and smaller numbers with just the direct dependencies constrained, and lesser again that have the fully resolved dependencies.

I'm trying to provide an option for those non version folks, by giving them an option (as well as to loosen constraints for various testing scenarios), while nudging them towards at least including the direct dependency match specs as default.

At the same time being able to support export the exact dependencies resolved, which is currently a separate code path, but I'd like to clean that up.

I see what you mean, however this might not be the right place to introduce this loosing if dependencies logic. For instance there is another Issue talking about this here: #639.

If this would be a pixi wide solution which export would benefit from. I'd personally like to start with a match-spec or locked environment export mechanism. And later we could possibly add the --pin style here as well.

What do you think of that?

abkfenris · 2024-05-27T17:47:17Z

@ruben-arts Thanks for taking a look, and the help so far. I may have some time to tinker this week, but not a ton so no promises on a timeline.

documentation in reference/cli.md (possible adding it to a tutorial or an seperate "how to go back and forth between pixi and conda" would be a nice bonus)

I was already thinking that the docs might need some Coming from conda/pip/poetry... pages. More how-to guide than tutorial.

A roundtrip test, possibly on pixi itself as that is a moving project so we can keep testing it.

A static test that starts with a full feature manifest, e.g. (different channels, different platforms, dependencies, pypi-dependencies, including -e)

Tests are gonna take me a bit to figure out. Especially since I've managed to break something locally that is causing test failures on a clean branch 🤦

ruben-arts · 2024-05-28T07:56:54Z

Feel free to ask questions about development issues here or in discord while developing!

bollwyvl · 2024-06-09T17:14:25Z

I haven't dug into the changes here much, but as the example shows with name, it's pretty clear anything related to existing conda tools are basically not going to work without being really explicit about the matrix platforms, etc.

Even for the env.yml output, filenames are probably the better way to communicate the platform on which something will run, with the stdout option being a special case.

For example, conda-lock --filename-template allows for decent, predictable automated file outputs, but usually only operates on a single, logical output.

So perhaps something like:

pixi project export conda \
  --subdir=ALL \                       # maybe allow fnmatch wildcards
  --environment=ALL \                  # "
  --filename-template="{env}-{subdir}.yml"

One could even already plan for --format=yml vs --format=explicit... or start referencing yml as CEP81 vs CEP79, or treat those as separate commands, but with the same args.

For that delicious @EXPLICIT format, which would take advantage of pixi solve-group, etc. and work with constructor, I never liked the # <pip requirement> from conda-lock (which doesn't work for constructor, anyhow), so perhaps planning for a strategy that generated {env}-{subdir}-pip-requirements.txt would be more sane. But I basically throw up my hands at editable installs and all the other crazy tricks one can hide in there.

abkfenris · 2024-06-11T14:06:56Z

Right now I'm only trying to tackle (and failing to get over the finish line due to lack of time) a minimal but adaptable way to export environment.yml files. To that, I envision that conda-lock would get a separate pixi project export sub-command for it's file format, similar to pip or other exporters might get.

If --name isn't specified, the command will template the filename based on the current environment and platform, both of which are also specifiable via the command line, so it's possible to generate a matrix that way.

srilman · 2024-06-18T16:35:22Z

@abkfenris Do you need any help getting this ready to review? Really looking forward to this!

abkfenris · 2024-06-18T19:14:39Z

@abkfenris Do you need any help getting this ready to review? Really looking forward to this!

Sorry, @srilman I really haven't gotten any more time to hack on this in the last few weeks, and I'm not sure I'm getting enough time to really dive into it in the next bit either.

Do you know your way around setting up tests for Pixi? If you could mock up some tests, even if they are failing with some of the environment.ymls suggested and/or a round trip test like was suggested here, I might be able to find some time to focus on debugging the places where there are failures or differences.

I could also use a hand with docs, but that might be worth waiting on to make sure there aren't any major changes suggested from debugging tests.

abkfenris changed the title ~~Export conda environment.yml~~ feat: Export conda environment.yml command May 22, 2024

Move export commands to project command group

ffb776b

ruben-arts reviewed May 22, 2024

View reviewed changes

abkfenris added 2 commits May 22, 2024 10:56

Format conda dependencies with matchspecs from manifest

6243e53

Initial pass at PyPI dependency specs

4b10624

Match editable pypi packages and format those specifically

763bcd6

Add the ability to export the locked environments

47a9c69

ruben-arts changed the title ~~feat: Export conda environment.yml command~~ feat: pixi project export conda to export project to conda environment.yml's May 27, 2024

ruben-arts reviewed May 27, 2024

View reviewed changes

stanmart mentioned this pull request Jun 1, 2024

Switch to pixi Quantco/glum#802

Open

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: `pixi project export conda` to export project to conda environment.yml's #1427

feat: `pixi project export conda` to export project to conda environment.yml's #1427

abkfenris commented May 22, 2024

ruben-arts commented May 22, 2024

abkfenris commented May 22, 2024

ruben-arts commented May 22, 2024

abkfenris commented May 22, 2024

ruben-arts May 22, 2024

abkfenris May 22, 2024

ruben-arts May 22, 2024

abkfenris May 22, 2024

abkfenris May 22, 2024

ruben-arts May 22, 2024

abkfenris commented May 22, 2024

ruben-arts commented May 23, 2024 •

edited

abkfenris commented May 23, 2024

vigneshmanick commented May 24, 2024

abkfenris commented May 24, 2024

ruben-arts left a comment

ruben-arts May 27, 2024

ruben-arts May 27, 2024

abkfenris May 27, 2024

ruben-arts May 27, 2024

ruben-arts May 27, 2024

ruben-arts May 27, 2024

abkfenris May 27, 2024 •

edited

ruben-arts May 28, 2024

ruben-arts May 27, 2024

abkfenris May 27, 2024

ruben-arts May 28, 2024

abkfenris commented May 27, 2024

ruben-arts commented May 28, 2024

bollwyvl commented Jun 9, 2024

abkfenris commented Jun 11, 2024

srilman commented Jun 18, 2024

abkfenris commented Jun 18, 2024

		#[arg(long, default_value = "manifest", value_enum)]
		pub version_spec: VersionSpec,

- .map(|channel| channel.name().to_string())
+ .map(|channel| {
+ default_channel_config().canonical_name(channel.base_url()).to_string()
+ })

feat: pixi project export conda to export project to conda environment.yml's #1427

Are you sure you want to change the base?

feat: pixi project export conda to export project to conda environment.yml's #1427

Conversation

abkfenris commented May 22, 2024

ruben-arts commented May 22, 2024

abkfenris commented May 22, 2024

ruben-arts commented May 22, 2024

abkfenris commented May 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abkfenris commented May 22, 2024

ruben-arts commented May 23, 2024 • edited

abkfenris commented May 23, 2024

vigneshmanick commented May 24, 2024

abkfenris commented May 24, 2024

ruben-arts left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abkfenris May 27, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abkfenris commented May 27, 2024

ruben-arts commented May 28, 2024

bollwyvl commented Jun 9, 2024

abkfenris commented Jun 11, 2024

srilman commented Jun 18, 2024

abkfenris commented Jun 18, 2024

feat: `pixi project export conda` to export project to conda environment.yml's #1427

feat: `pixi project export conda` to export project to conda environment.yml's #1427

ruben-arts commented May 23, 2024 •

edited

abkfenris May 27, 2024 •

edited