From d63f82e8f053044e47cc0d08d00e6f9b2c1cd329 Mon Sep 17 00:00:00 2001 From: Kian-Tat Lim Date: Wed, 14 Jul 2021 10:53:22 -0700 Subject: [PATCH 1/5] Add a page about conda and environments. --- stack/conda.rst | 67 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 stack/conda.rst diff --git a/stack/conda.rst b/stack/conda.rst new file mode 100644 index 00000000..7107630c --- /dev/null +++ b/stack/conda.rst @@ -0,0 +1,67 @@ +=================== +Conda and rubin-env +=================== + +The Science Pipelines stack uses ``conda`` as its primary package manager for obtaining third-party Python and C++ dependencies. +In particular, the ``conda-forge`` distribution channel is used, as it provides a wide selection of community-maintained but rigorously verified packages. + +.. _rubin-env-metapackage: + +``rubin-env`` +------------- + +The list of required dependencies is maintained in a ``conda-forge`` metapackage named ``rubin-env``. +Use of a metapackage rather than an environment specification allows for simple installation and environment creation by end users. + +``rubin-env`` is intended to be the minimal set of dependencies required to build and execute all packages of ``lsst_distrib`` in production. +It does not necessarily include dependencies useful for developers to modify existing packages or add new packages. +It also does not necessarily include dependencies useful for science users to analyze, visualize, or otherwise work with raw Science Pipelines inputs or output data products. + +In the future, an expanded ``rubin-env-plus`` may be created that would include additional dependencies for convenience and to ensure that compatible versions of all packages can be determined. +``rubin-env`` would then become a subset of this environment. + +Requests for additions to ``rubin-env`` should be made via RFC. +The DM-CCB must approve such requests. + +.. _rubin-env-versioning: + +Versioning ``rubin-env`` +------------------------ + +``rubin-env`` has a standard semantic version number with major, minor, and patch components, in addition to a conda-internal build number. +These components are updated as follows: + +* Build: increment when adding a ``<`` pin for an existing dependency. +* Patch: increment when adding a ``>`` pin for an existing dependency that is compatible with old code (no major version update) or when adding a ``<`` pin that is earlier than the previous pin. +* Minor: increment when adding a new dependency. +* Major: increment when removing a dependency, or when removing a pin, or when changing a dependency major version + +.. _conda-shared-stack: + +Shared stack +------------ + +A shared Science Pipelines stack is maintained in the ``/software/lsstsw`` directory on the LDF development cluster. +This stack is accessible to all login, development/head/submit, and batch nodes. +It is maintained by a ``cron`` job (running under user ``lsstsw``) that executes a script from ``lsst-dm/shared-stack``. +This script automatically installs new weekly releases from source. +It also augments the standard ``rubin-env`` conda environment with additional packages useful for developers (but not needed in production). +Requests for additions to this list should be made via RFC. + +.. _conda-rsp-notebooks: + +Rubin Science Platform notebooks +-------------------------------- + +Rubin Science Platform notebooks use a container that is built from each release. +As part of the container build process, additional packages useful in the notebook environment are added to the conda environment. +The list of packages is maintained in ``lsst-sqre/nublado2``; requests for additions to this list should be made via RFC. + +.. _conda-exact-environments: + +"Exact" environments +-------------------- + +The conda environments used by Jenkins to build nightly, weekly, and official releases are preserved as ``conda list --explicit`` outputs in ``eups.lsst.codes``. +These can be used in place of the ``rubin-env`` metapackage to exactly reproduce a build for consistency in production or for debugging. +Both ``newinstall`` and ``lsstsw`` allow specification of an eups tag to retrieve the exact environment used when that tag was published. From bab43da0324f7a401aeec4de24dfa6a5e685fc77 Mon Sep 17 00:00:00 2001 From: Kian-Tat Lim Date: Fri, 16 Jul 2021 22:52:26 -0700 Subject: [PATCH 2/5] Point to new conda page and add to toctree. --- index.rst | 2 ++ stack/index.rst | 1 + 2 files changed, 3 insertions(+) diff --git a/index.rst b/index.rst index fae6756d..790b8420 100644 --- a/index.rst +++ b/index.rst @@ -322,6 +322,7 @@ ReStructuredText stack/license-and-copyright stack/packaging-third-party-eups-dependencies stack/renaming-a-package + stack/conda .. _part-dm-stack: @@ -381,6 +382,7 @@ Packaging. - :doc:`stack/license-and-copyright` - :doc:`stack/packaging-third-party-eups-dependencies` - :doc:`stack/renaming-a-package` +- :doc:`stack/conda` .. GIT SECTION =============================================================== diff --git a/stack/index.rst b/stack/index.rst index b3a27f05..885a1b35 100644 --- a/stack/index.rst +++ b/stack/index.rst @@ -24,3 +24,4 @@ DM Stack guides - :doc:`moving-to-github-actions` - :doc:`packaging-third-party-eups-dependencies` - :doc:`renaming-a-package` +- :doc:`conda` From 716e7d5d0bb4ab1a42896768e76a268239d57085 Mon Sep 17 00:00:00 2001 From: Kian-Tat Lim Date: Fri, 16 Jul 2021 22:52:57 -0700 Subject: [PATCH 3/5] Update platform description of conda. --- stack/platforms.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/stack/platforms.rst b/stack/platforms.rst index 860668db..768a176f 100644 --- a/stack/platforms.rst +++ b/stack/platforms.rst @@ -36,10 +36,11 @@ Environment =========== On each of these platforms, we rely on `Conda`_ and `conda-forge`_ to provide our build- and run-time dependencies, including compilers and the Python interpreter. -The Conda environment for Science Pipelines is available in the `scipipe_conda_env`_ repository. -This defines the environment within which our code must execute: all code must be compatible with the contents of this environment when it is written; compatibility with older or newer environments is not guaranteed. -The Conda environment is periodically updated to pull in newer versions of third party packages. -To request such an update, or to request that a new package be added to the environment, please :doc:`file an RFC `. +The Conda environment for Science Pipelines is generated by :doc:`the rubin-env metapackage `. +This defines the environment within which our code must execute: all code must be compatible with the current version of ``rubin-env``. +Compatibility with different versions is determined by the ``rubin-env`` semantic version number. +The Conda environment is periodically updated to maintain compatibility, to utilize new features of third-party packages, or to add new dependencies. +To request that a new package be added to the environment, please :doc:`file an RFC `. .. _Conda: https://conda.io .. _conda-forge: https://conda-forge.org/ From 184e35564170019a8d7f871f153b8ba536070780 Mon Sep 17 00:00:00 2001 From: Kian-Tat Lim Date: Fri, 16 Jul 2021 22:53:23 -0700 Subject: [PATCH 4/5] Indicate most dependencies are from conda. --- stack/packaging-third-party-eups-dependencies.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/stack/packaging-third-party-eups-dependencies.rst b/stack/packaging-third-party-eups-dependencies.rst index 2a3936a2..f4bafa1e 100644 --- a/stack/packaging-third-party-eups-dependencies.rst +++ b/stack/packaging-third-party-eups-dependencies.rst @@ -4,6 +4,11 @@ Distributing Third-Party Packages with EUPS This page documents how to make a third-party software package install-able using the :command:`eups distrib install` command. +Generally, most dependencies :doc:`should be added via conda ` using the ``rubin-env`` metapackage. +The primary reason to add an eups tarball-and-patch (TaP) package as described here is if we require a fix that has not been released in conda-forge (or if the package has no conda-forge recipe at all, and we do not want to create one). + +Other packages that are under active development by Rubin developers may be maintained as forked repositories and distributed as normal eups packages. + Getting Approval ================ From 5915963ab964c617a1a1a61ba35255fa797df27d Mon Sep 17 00:00:00 2001 From: Kian-Tat Lim Date: Mon, 19 Jul 2021 13:40:39 -0700 Subject: [PATCH 5/5] Add compatibility note as suggested in review. --- stack/conda.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stack/conda.rst b/stack/conda.rst index 7107630c..7a73cfae 100644 --- a/stack/conda.rst +++ b/stack/conda.rst @@ -31,7 +31,7 @@ Versioning ``rubin-env`` ``rubin-env`` has a standard semantic version number with major, minor, and patch components, in addition to a conda-internal build number. These components are updated as follows: -* Build: increment when adding a ``<`` pin for an existing dependency. +* Build: increment when adding a ``<`` pin for an existing dependency, as this preserves compatibility with previous builds of the environment. * Patch: increment when adding a ``>`` pin for an existing dependency that is compatible with old code (no major version update) or when adding a ``<`` pin that is earlier than the previous pin. * Minor: increment when adding a new dependency. * Major: increment when removing a dependency, or when removing a pin, or when changing a dependency major version