change internal dependency management (#108)

This PR changes how internal dependencies are handled. - List internal dependencies as regular dependencies. - Modify the install commands to install them all at once (or in the right order) - Fix issues with 'direct links' and filepaths to list the internal dependencies by list them as regular dependencies. cf #91 * initial setup * dump * test * dump * backup * scheme implemented * versioning in place * first version * slight change in versioning: main is kept at last released version * make build version strings sorted along date-time * update docs * list internal dependencies as regular dependencies instead of using direct links with filepaths also exclude test/ folder from setuptools packages * install all packages during CI actions to avoid issues with internal dependencies * fix missing traling slashes for packages to point to locations * update installation instructions and install script * update changelog
airo-ugent · Jan 12, 2024 · f166f02 · f166f02
1 parent 1718e06
commit f166f02
Show file tree

Hide file tree

Showing 13 changed files with 84 additions and 81 deletions.
diff --git a/.github/workflows/mypy.yaml b/.github/workflows/mypy.yaml
@@ -24,6 +24,6 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         pip install mypy
-        pip install ${{matrix.package}}/[external]
+        pip install airo-typing/ airo-spatial-algebra/ airo-camera-toolkit/ airo-robots/ airo-teleop/ airo-dataset-tools/
     - name: Run type checker
       run: mypy ${{matrix.package}}/
diff --git a/.github/workflows/pytest.yaml b/.github/workflows/pytest.yaml
@@ -27,7 +27,7 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         pip install pytest
-        pip install ${{matrix.package}}/[external]
+        pip install airo-typing/ airo-spatial-algebra/ airo-camera-toolkit/ airo-robots/ airo-teleop/ airo-dataset-tools/
     - name: Run Tests
       run: pytest ${{matrix.package}}/
     - name: Run Notebooks

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,21 @@ All notable changes for the packages in the airo-mono repo are documented here.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 This project uses a [CalVer](https://calver.org/) versioning scheme with monthly releases, see [here](versioning.md)
 
+## Unreleased
+
+### Breaking changes
+ - internal dependencies are now listed as regular dependencies in the `setup.py` file to overcome issues and make the installation process less complicated. This implies you need to install packages according to their dependencies and can no longer use the `external` tag as in `pip install airo-typing[external]`.
+ see [issue #91](https://github.com/airo-ugent/airo-mono/issues/91) and
+ [PR](https://github.com/airo-ugent/airo-mono/pull/108) for more details.
+
+### Added
+
+### Changed
+
+### Fixed
+
+### Removed
+
 ## 2024.1.0
 
 ### Breaking changes

diff --git a/README.md b/README.md
@@ -30,47 +30,55 @@ Some packages also have a command line interface. Simply run `$package-name --he
 
 
 # Installation
-There are a number of ways to install packages from this repo. As this repo is still in development and has breaking changes every now and then, we recommend locking on specific commits.
+There are a number of ways to install packages from this repo. As this repo is still in development and has breaking changes every now and then, we recommend locking on specific commits or releases if you need stability.
 
-**directly from github**
+## regular install
+Installs the packages from PyPI.
 
-if you just want to use a package for a downstream application you can install it with pip like this: `python -m pip install ' <pkg-name>[external] @ git+https://github.com/airo-ugent/airo-mono@<branch/tag>#subdirectory=<package-dir>'`. Note the [external] specification, this is a quick hack to allow for working with a monorepo while using pip is package manager, read more [here](#developer-guide/). There will now be a `src/` folder in your project where pip has downloaded the repo and from where the package is installed, but you can ignore this as it will be automatically excluded from source control. You can (and should?) lock the pip install to a specific commit in your dependency manager (pip/conda/...). The [external] specification only works on pip versions >= 22 so make sure your pip version is up-to-date with `pip install --upgrade pip`.
 
-The following table shows the required command per package:
+not available yet
+`pip install airo-camera-toolkit ....`
+
+## installing from dev builds
+Installs from dev builds, which are created for each commit on the main.
+
+not availble yet.
+
+
+
+## from github source
+
+DEPRECATED.
+
+We discourage the use of this installation method!
 
 | Package | command |
 |-------|-------|
-|`airo-camera-toolkit`|`python -m pip install 'airo-camera-toolkit[external] @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-camera-toolkit'`|
-|`airo-dataset-tools`|`python -m pip install 'airo-dataset-tools[external] @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-dataset-tools'`|
-|`airo-robots`|`python -m pip install 'airo-robots[external] @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-robots'`|
-|`airo-spatial-algebra`|`python -m pip install 'airo-spatial-algebra[external] @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-spatial-algebra' `|
-|`airo-teleop`|`python -m pip install 'airo-teleop[external] @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-teleop'`|
-|`airo-typing`  |`python -m pip install 'airo-typing[external] @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-typing'`|
 
-or alternatively, you can install all packages at once by running the [installation script](scripts/install-airo-mono.sh).
+|`airo-typing`  |`python -m pip install 'airo-typing @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-typing'`|
+|`airo-dataset-tools`|`python -m pip install 'airo-dataset-tools @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-dataset-tools'`|
+|`airo-robots`|`python -m pip install 'airo-robots @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-robots'`|
+|`airo-spatial-algebra`|`python -m pip install 'airo-spatial-algebra @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-spatial-algebra' `|
+|`airo-teleop`|`python -m pip install 'airo-teleop @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-teleop'`|
+|`airo-camera-toolkit`|`python -m pip install 'airo-camera-toolkit @ git+https://github.com/airo-ugent/airo-mono@main#subdirectory=airo-camera-toolkit'`|
 
+Make sure you install the packages according to their dependency tree. If you have not installed the airo-mono packages on which a package depends first, you will get a missing import error (or it will install the package from PyPI..)
+
+## local installation
 **git submodule**
 
-Alternatively you can add this repo as a submodule and install the relevant packages afterwards with regular pip commands. This might be useful for as long as this repo is making fast/breaking changes without good version management, as you can lock the submodule on a specific commit.
+You can add this repo as a submodule and install the relevant packages afterwards with regular pip commands. This allows to seamlessly make contributions to this repo whilst working on your own project or if you want to pin on a specific version.
 
 In your repo, run:
 ```
 git submodule init
 git submodule add https://github.com/airo-ugent/airo-mono@<commit>
 cd airo-mono
 ```
-You can now add the packages you need to your requirements or environment file. More about submodules can be found [here](https://git-scm.com/book/en/v2/Git-Tools-Submodules).
-
-**editable install**
+You can now add the packages you need to your requirements or environment file, either in development mode or through a regular pip install.
+More about submodules can be found [here](https://git-scm.com/book/en/v2/Git-Tools-Submodules). Make sure to install the packages in one pip command such that pip can install them in the appropriate order to deal with internal dependencies.
 
-If you want to make changes, you should probably clone this repo first (optionally using git submodules)
-and then install all relevant packages in [editable](https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs) mode, so that any change you make is immediately 'visible' to your python interpreter. If you make make a standalone clone of this repo, you can simply run `conda env create -f environment.yaml`, which does this for you (and also installs some binaries for convenience).
 
-```
-git clone https://github.com/airo-ugent/airo-mono@<commit>
-cd airo-mono
-conda env create -f environment.yaml
-```
 # Developer guide
 ### setting up local environment
 To set up your development environment after cloning this repo, run:
@@ -108,17 +116,18 @@ The tests are executed for each package in isolation using [github actions Matri
 We test on python 3.8 (default on ubuntu 20.04), 3.9 and 3.10 (default on Ubuntu 22.04). It is important to test these versions explicitly, e.g. typing with `list` instead of `typing.List` is not allowed in 3.8, but it is in >=3.9.
 
 ### Management of (local) dependencies
-[more background on package and dependency management in python](https://ealizadeh.com/blog/guide-to-python-env-pkg-dependency-using-conda-poetry/)
 
-An issue with using a monorepo is that you want to have packages declare their local dependencies as well, while being able to install all packages in editable mode so that all changes that you make are immediately reflected (so that you can in fact edit multiple packages at the same time).
+An issue with using a monorepo is that you want to have packages declare their local dependencies as well. But before you publish your packages or if you want to test unreleased code (as usually), this creates an issue: where should pip find these local package? Though there exist more advanced package managers such as Poetry, ([more background on package and dependency management in python](https://ealizadeh.com/blog/guide-to-python-env-pkg-dependency-using-conda-poetry/)
+) that can handle this, we have opted to stick with pip to keep the barier for new developers lower.
+
 
-Pip makes this very hard as it by default reinstalls any local package (so it will get reinstalled even if it already was in your environment!) and since you cannot specify editable dependencies in the distutils setup.py.
+This implies we simply add local dependencies in the setup file as regular dependencies, but we have to make sure pip can find the dependencies when installing the pacakges.There are two options to do so:
+1. You make sure that the local dependencies are installed before installing the package, either by running the pip install commands along the dependency tree, or by running all installs in a single pip commamd: `pip install <pkg1>  <pkg2> <pkg3>`
+2. you create distributions for the packages upfront and then tell pip where to find them (because they won't be on PyPI, which is where pip searches by default): `pip install --find-link https:// or /path/to/distributions/dir`
 
-So if you install a package in editable mode and then install another in editable mode that has the first as a dependency, pip would reinstall that first package in 'normal' mode and your changes to that package would no longer be immediately reflected.
 
-Among others, this is why [many people](https://medium.com/opendoor-labs/our-python-monorepo-d34028f2b6fa) use [Poetry](https://python-poetry.org/docs/basic-usage/) as package manager for python monorepos. Poetry config files can be handled to fix this issue.
+Initially, we used a direct link to point to the path of the dependencies, but this created some issues and hence we now use this easier approach. see [#91](https://github.com/airo-ugent/airo-mono/issues/91) for more details.
 
-However, for now we want to avoid adding this complexity for new contributors. Therefore we use an ad-hoc solution by specifying the local dependencies through an `extras_require` of the setup.py. Installing the package with its internal dependencies should hence be done with `pip install package[external]`, whereas an editable installation can still happen with `pip install -e package`. It's now on you to install the local dependencies in editable mode as well.
 ### Creating a new package
 Creating a new package is kind of a hassle atm, in the future we might add a coockiecutter template for it. For now here are the steps you have to take:
 - create the nested structure

diff --git a/airo-camera-toolkit/setup.py b/airo-camera-toolkit/setup.py
@@ -17,16 +17,12 @@
         "rerun-sdk>=0.11.0",
         "click",
         "loguru",
+        "airo-typing",
+        "airo-spatial-algebra",
+        "airo-robots",
+        "airo-dataset-tools",
     ],
-    extras_require={
-        "external": [
-            f"airo_typing @ file://localhost/{root_folder}/airo-typing",
-            f"airo_spatial_algebra @ file://localhost/{root_folder}/airo-spatial-algebra",
-            f"airo_robots @ file://localhost/{root_folder}/airo-robots",
-            f"airo_dataset_tools @ file://localhost/{root_folder}/airo-dataset-tools",
-        ]
-    },
-    packages=setuptools.find_packages(),
+    packages=setuptools.find_packages(exclude=["test"]),
     entry_points={
         "console_scripts": [
             "airo-camera-toolkit = airo_camera_toolkit.cli:cli",

diff --git a/airo-dataset-tools/setup.py b/airo-dataset-tools/setup.py
@@ -1,7 +1,6 @@
 import pathlib
 
 import setuptools
-from setuptools import find_packages
 
 root_folder = pathlib.Path(__file__).parents[1]
 setuptools.setup(
@@ -22,14 +21,10 @@
         "Pillow",
         "types-Pillow",
         "albumentations",
+        "airo-typing",
+        "airo-spatial-algebra",
     ],
-    extras_require={
-        "external": [
-            f"airo_typing @ file://localhost/{root_folder}/airo-typing",
-            f"airo_spatial_algebra @ file://localhost/{root_folder}/airo-spatial-algebra",
-        ]
-    },
-    packages=find_packages(),
+    packages=setuptools.find_packages(exclude=["test"]),
     entry_points={
         "console_scripts": [
             "airo-dataset-tools = airo_dataset_tools.cli:cli",

diff --git a/airo-robots/setup.py b/airo-robots/setup.py
@@ -9,13 +9,13 @@
     description="Interfaces, hardware implementations of those interfaces and other functionalities to control robot manipulators and grippers at the Ghent University AI and Robotics Lab",
     author="Thomas Lips",
     author_email="[email protected]",
-    install_requires=["numpy", "ur-rtde>=1.5.7", "click"],  # cf https://github.com/airo-ugent/airo-mono/issues/52
-    extras_require={
-        "external": [
-            f"airo_typing @ file://localhost/{root_folder}/airo-typing",
-            f"airo_spatial_algebra @ file://localhost/{root_folder}/airo-spatial-algebra",
-        ]
-    },
+    install_requires=[
+        "numpy",
+        "ur-rtde>=1.5.7",  # cf https://github.com/airo-ugent/airo-mono/issues/52
+        "click",
+        "airo-typing",
+        "airo-spatial-algebra",
+    ],
     packages=setuptools.find_packages(),
     package_data={"airo_robots": ["py.typed"]},
 )
diff --git a/airo-spatial-algebra/setup.py b/airo-spatial-algebra/setup.py
@@ -9,17 +9,8 @@
     description="code for working with SE3 poses,transforms,... for robotic manipulation at the Ghent University AI and Robotics Lab",
     author="Thomas Lips",
     author_email="[email protected]",
-    install_requires=[
-        "numpy",
-        "scipy",
-        "spatialmath-python",
-    ],
-    extras_require={
-        "external": [
-            f"airo_typing @ file://localhost/{root_folder}/airo-typing",
-        ]
-    },
-    packages=setuptools.find_packages(),
+    install_requires=["numpy", "scipy", "spatialmath-python", "airo-typing"],
+    packages=setuptools.find_packages(exclude=["test"]),
     # include py.typed to declare type information is available, see
     # https://mypy.readthedocs.io/en/stable/installed_packages.html#making-pep-561-compatible-packages
     package_data={"airo_spatial_algebra": ["py.typed"]},

diff --git a/airo-teleop/setup.py b/airo-teleop/setup.py
@@ -9,13 +9,6 @@
     description="teleoperation functionality for manually controlling manipulators and grippers using gaming controllers etc. at the Ghent University AI and Robotics Lab",
     author="Thomas Lips",
     author_email="[email protected]",
-    install_requires=["pygame", "click", "loguru"],
-    extras_require={
-        "external": [
-            f"airo_typing @ file://localhost/{root_folder}/airo-typing",
-            f"airo_spatial_algebra @ file://localhost/{root_folder}/airo-spatial-algebra",
-            f"airo_robots @file://localhost/{root_folder}/airo-robots",
-        ]
-    },
-    packages=setuptools.find_packages(),
+    install_requires=["pygame", "click", "loguru", "airo-typing", "airo-spatial-algebra", "airo-robots"],
+    packages=setuptools.find_packages(exclude=["test"]),
 )
diff --git a/airo-typing/setup.py b/airo-typing/setup.py
@@ -7,6 +7,6 @@
     author="Thomas Lips",
     author_email="[email protected]",
     install_requires=["numpy"],
-    packages=["airo_typing"],
+    packages=setuptools.find_packages(exclude=["test"]),
     package_data={"airo_typing": ["py.typed"]},
 )
diff --git a/docs/releasing.md b/docs/releasing.md
@@ -1,6 +1,7 @@
 # Releasing
 
 
+
 To create a new release there are two steps:
 
 
@@ -10,9 +11,8 @@ To create a new release there are two steps:
 
 
 ## Version bumping
-
 see [versioning](./versioning.md)
 
 ## Creating distribution
 
- TODO, not implemented yet.
+ TODO, not implemented yet.
diff --git a/docs/versioning.md b/docs/versioning.md
@@ -13,12 +13,14 @@ We use a [Calender Version](https://calver.org/) scheme as follows:
 YYYY.MM.N
 ```
 
+
 where YYYY.MM is the year and month in which the release is made and N is the MICRO part used to distinguish between multiple releases in the same month. No semantic meaning can be attached to the parts and breaking changes might occur in each release for now.
 
 ## Versioning strategy
 The main branch ('our trunk') will always live at last released version. All development builds will hence have the **previous** release as base.
 This is slightly counter-intuitive but actually common practice (e.g. [Twisted](https://github.com/twisted/twisted/tree/trunk)).
 
+
 ## version bumping
 
 To create a new version, you simply bump the version number as follows:
@@ -42,6 +44,7 @@ Next to updating the version strings in all relevant files, this will also creat
 
 ## Development distributions/builds
 Development builds can be used to build the development trunk (main branch) and their versioning scheme is
+
 ```
 YYYY.MM.N-build.date.sha
 ```

diff --git a/scripts/install-airo-mono.sh b/scripts/install-airo-mono.sh
@@ -18,12 +18,13 @@ branch=${1:-main}
 echo "Installing airo-mono from branch/commit $branch."
 
 package_names=(
-    "airo-camera-toolkit"
+    "airo-typing"
+    "airo-spatial-algebra"
     "airo-dataset-tools"
     "airo-robots"
-    "airo-spatial-algebra"
     "airo-teleop"
-    "airo-typing"
+    "airo-camera-toolkit"
+
 )
 
 # Base URL for the Git repository
@@ -32,7 +33,7 @@ base_url="https://github.com/airo-ugent/airo-mono@${branch}#subdirectory="
 # Loop through package names and execute pip install command
 for package_name in "${package_names[@]}"
 do
-    cmd="python -m pip install '${package_name}[external] @ git+${base_url}${package_name}'"
+    cmd="python -m pip install '${package_name} @ git+${base_url}${package_name}'"
     echo $cmd
     eval $cmd
     echo "Installed $package_name."