Merge pull request #489 from lsst-dm/u/ktl/add-conda-page

Add a page about conda and rubin-env and update related documents.

index.rst

@@ -322,6 +322,7 @@ ReStructuredText
  stack/license-and-copyright
  stack/packaging-third-party-eups-dependencies
  stack/renaming-a-package
+ stack/conda
 
 .. _part-dm-stack:
 
@@ -381,6 +382,7 @@ Packaging.
 - :doc:`stack/license-and-copyright`
 - :doc:`stack/packaging-third-party-eups-dependencies`
 - :doc:`stack/renaming-a-package`
+- :doc:`stack/conda`
 
 .. GIT SECTION ===============================================================
 

stack/conda.rst

@@ -0,0 +1,67 @@
+===================
+Conda and rubin-env
+===================
+
+The Science Pipelines stack uses ``conda`` as its primary package manager for obtaining third-party Python and C++ dependencies.
+In particular, the ``conda-forge`` distribution channel is used, as it provides a wide selection of community-maintained but rigorously verified packages.
+
+.. _rubin-env-metapackage:
+
+``rubin-env``
+-------------
+
+The list of required dependencies is maintained in a ``conda-forge`` metapackage named ``rubin-env``.
+Use of a metapackage rather than an environment specification allows for simple installation and environment creation by end users.
+
+``rubin-env`` is intended to be the minimal set of dependencies required to build and execute all packages of ``lsst_distrib`` in production.
+It does not necessarily include dependencies useful for developers to modify existing packages or add new packages.
+It also does not necessarily include dependencies useful for science users to analyze, visualize, or otherwise work with raw Science Pipelines inputs or output data products.
+
+In the future, an expanded ``rubin-env-plus`` may be created that would include additional dependencies for convenience and to ensure that compatible versions of all packages can be determined.
+``rubin-env`` would then become a subset of this environment.
+
+Requests for additions to ``rubin-env`` should be made via RFC.
+The DM-CCB must approve such requests.
+
+.. _rubin-env-versioning:
+
+Versioning ``rubin-env``
+------------------------
+
+``rubin-env`` has a standard semantic version number with major, minor, and patch components, in addition to a conda-internal build number.
+These components are updated as follows:
+
+* Build: increment when adding a ``<`` pin for an existing dependency, as this preserves compatibility with previous builds of the environment.
+* Patch: increment when adding a ``>`` pin for an existing dependency that is compatible with old code (no major version update) or when adding a ``<`` pin that is earlier than the previous pin.
+* Minor: increment when adding a new dependency.
+* Major: increment when removing a dependency, or when removing a pin, or when changing a dependency major version
+
+.. _conda-shared-stack:
+
+Shared stack
+------------
+
+A shared Science Pipelines stack is maintained in the ``/software/lsstsw`` directory on the LDF development cluster.
+This stack is accessible to all login, development/head/submit, and batch nodes.
+It is maintained by a ``cron`` job (running under user ``lsstsw``) that executes a script from ``lsst-dm/shared-stack``.
+This script automatically installs new weekly releases from source.
+It also augments the standard ``rubin-env`` conda environment with additional packages useful for developers (but not needed in production).
+Requests for additions to this list should be made via RFC.
+
+.. _conda-rsp-notebooks:
+
+Rubin Science Platform notebooks
+--------------------------------
+
+Rubin Science Platform notebooks use a container that is built from each release.
+As part of the container build process, additional packages useful in the notebook environment are added to the conda environment.
+The list of packages is maintained in ``lsst-sqre/nublado2``; requests for additions to this list should be made via RFC.
+
+.. _conda-exact-environments:
+
+"Exact" environments
+--------------------
+
+The conda environments used by Jenkins to build nightly, weekly, and official releases are preserved as ``conda list --explicit`` outputs in ``eups.lsst.codes``.
+These can be used in place of the ``rubin-env`` metapackage to exactly reproduce a build for consistency in production or for debugging.
+Both ``newinstall`` and ``lsstsw`` allow specification of an eups tag to retrieve the exact environment used when that tag was published.

stack/index.rst

@@ -24,3 +24,4 @@ DM Stack guides
 - :doc:`moving-to-github-actions`
 - :doc:`packaging-third-party-eups-dependencies`
 - :doc:`renaming-a-package`
+- :doc:`conda`

stack/packaging-third-party-eups-dependencies.rst

@@ -4,6 +4,11 @@ Distributing Third-Party Packages with EUPS
 
 This page documents how to make a third-party software package install-able using the :command:`eups distrib install` command.
 
+Generally, most dependencies :doc:`should be added via conda </stack/conda>` using the ``rubin-env`` metapackage.
+The primary reason to add an eups tarball-and-patch (TaP) package as described here is if we require a fix that has not been released in conda-forge (or if the package has no conda-forge recipe at all, and we do not want to create one).
+
+Other packages that are under active development by Rubin developers may be maintained as forked repositories and distributed as normal eups packages.
+
 Getting Approval
 ================
 

stack/platforms.rst

@@ -36,10 +36,11 @@ Environment
 ===========
 
 On each of these platforms, we rely on `Conda`_ and `conda-forge`_ to provide our build- and run-time dependencies, including compilers and the Python interpreter.
-The Conda environment for Science Pipelines is available in the `scipipe_conda_env`_ repository.
-This defines the environment within which our code must execute: all code must be compatible with the contents of this environment when it is written; compatibility with older or newer environments is not guaranteed.
-The Conda environment is periodically updated to pull in newer versions of third party packages.
-To request such an update, or to request that a new package be added to the environment, please :doc:`file an RFC </communications/rfc>`.
+The Conda environment for Science Pipelines is generated by :doc:`the rubin-env metapackage </stack/conda>`.
+This defines the environment within which our code must execute: all code must be compatible with the current version of ``rubin-env``.
+Compatibility with different versions is determined by the ``rubin-env`` semantic version number.
+The Conda environment is periodically updated to maintain compatibility, to utilize new features of third-party packages, or to add new dependencies.
+To request that a new package be added to the environment, please :doc:`file an RFC </communications/rfc>`.
 
 .. _Conda: https://conda.io
 .. _conda-forge: https://conda-forge.org/