Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide Docker/Podman/apptainer images. #68

Open
ghuls opened this issue Jul 21, 2022 · 6 comments
Open

Provide Docker/Podman/apptainer images. #68

ghuls opened this issue Jul 21, 2022 · 6 comments

Comments

@ghuls
Copy link

ghuls commented Jul 21, 2022

It would be nice to have zsync2 as Docker/Podman/apptainer images besides AppImage.

In an HPC environment, there is in general no option to install AppImage.

Singularity/Apptainer (http://apptainer.org/) and Podman (rootless Docker) is for example the only way to run containers on our HPC infrastructure. zsync2 is quite difficult to compile natively on old RedHat versions.

Docker/Podman and SingularityApptainer images are quite easy to make from the Apptainer image.

prepare_zsync2_rootfs.sh:

#!/bin/bash

# Ubuntu Bionic OCI core root image.
ubuntu_oci_root_image_url='https://partner-images.canonical.com/oci/bionic/current/ubuntu-bionic-oci-amd64-root.tar.gz'
ubuntu_oci_root_image_file='ubuntu-bionic-oci-amd64-root.tar.gz'

# Get version/commit from AppImage release links: https://github.com/AppImage/zsync2/releases
zsync2_version='2.0.0-alpha-1-20220602'
zsync2_commit='51-4e549b6'

zsync2_app_image_url="https://github.com/AppImage/zsync2/releases/download/${zsync2_version}/zsync2-${zsync2_commit}-x86_64.AppImage"
zsync2_app_image_file="zsync2-${zsync2_commit}-x86_64.AppImage"


# Download Ubuntu Bionic OCI core root image if not available or out of date.
if [ ! -e "${ubuntu_oci_root_image_file}" ] ; then
    curl -R -L -O "${ubuntu_oci_root_image_url}";
else
   # Only download Ubuntu OCI core root image, if there is a newer version.
   curl -R -L -O -z "${ubuntu_oci_root_image_file}" "${ubuntu_oci_root_image_url}";
fi

# Download zsync2 AppImage if not availabe.
if [ ! -e "${zsync2_app_image_file}" ] ; then
    curl -R -L -O "${zsync2_app_image_url}";
    
    chmod a+x "${zsync2_app_image_file}";
fi

rm -rf ./squashfs-root/

# Extract AppImage.
"./${zsync2_app_image_file}" --appimage-extract

# Fix symlink.
rm -r ./squashfs-root/usr/share/doc/libhogweed4
ln -s libnettle6 ./squashfs-root/usr/share/doc/libhogweed4

DockerFile

FROM scratch

ADD /squashfs-root/usr/ /usr
ADD ubuntu-bionic-oci-amd64-root.tar.gz /

CMD ["bash"]

zsync2.def : Singularity/Apptainer file.

Bootstrap: scratch

%setup
    cp -dR -n --preserve=mode,timestamps ./squashfs-root/usr "${APPTAINER_ROOTFS}/"
    tar xzf ubuntu-bionic-oci-amd64-root.tar.gz -C "${APPTAINER_ROOTFS}"
# Build with Docker.
docker build -t zsync-${zsync2_version}-x86_64 .

# Build with Podman.
podman build -t zsync-${zsync2_version}-x86_64 .

# Build with Singularity.
sudo singularity build "${zsync2_app_image_file%.AppImage}.sif" zsync2.def 

# Buld with Apptainer.
sudo apptainer build "${zsync2_app_image_file%.AppImage}.sif" zsync2.def 
# Run with Docker/Podman.
podman run \
    --mount 'type=bind,source=/,destination=/local_filesystem' \
    --workdir=/local_filesystem/${PWD} \
    zsync-${zsync2_version}-x86_64 \
        zsyncmake2 -u https://test.com/bigfile.bin bigfile.bin

# Run with Singularity/Apptainer.
singularity exec \
    -B ${PWD} \
    ./zsync2-51-4e549b6-x86_64.sif \
        zsyncmake2 -u https://test.com/bigfile.bin bigfile.bin

We have files that are over 100G in some cases, so zsync2 would be nice to have to make sure the download is correct (and can be resumed).

@probonopd
Copy link
Member

In an HPC environment, there is in general no option to install AppImage.

...Nor is there a need to install AppImages. Just download, mark as executable, and run with --appimage-extract-and-run if your environment does not provide a working FUSE setup.

@ghuls
Copy link
Author

ghuls commented Jul 21, 2022

@probonopd I didn't know about that argument, but it doesn't work anyway as HPC clusters have glibc versions from a milion years ago:

❯  ./zsync2-51-4e549b6-x86_64.AppImage --appimage-extract-and-run
/tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun: /usr/lib64/libc.so.6: version `GLIBC_2.22' not found (required by /tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun)
/tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun)
/tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun)
/tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun: /usr/lib64/libgpg-error.so.0: no version information available (required by /tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/usr/bin/../lib/libgcrypt.so.20)
/tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun: /usr/lib64/libc.so.6: version `GLIBC_2.25' not found (required by /tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/usr/bin/../lib/libcrypto.so.1.1)
/tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun: /usr/lib64/libc.so.6: version `GLIBC_2.27' not found (required by /tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/usr/bin/../lib/libgssapi_krb5.so.2)
/tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun: /usr/lib64/libc.so.6: version `GLIBC_2.25' not found (required by /tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/usr/bin/../lib/libgnutls.so.30)
/tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun: /usr/lib64/libc.so.6: version `GLIBC_2.22' not found (required by /tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/usr/bin/../lib/libgnutls.so.30)
/tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/AppRun: /usr/lib64/libc.so.6: version `GLIBC_2.25' not found (required by /tmp/appimage_extracted_81b6fb70f1742dd32187033f6bf21299/usr/bin/../lib/libcrypt.so.1)

@probonopd
Copy link
Member

Interesting! Luckily this will soon be a thing of the past with

@TheAssassin
Copy link
Member

What would be needed to provide apptainer packages from this repository? I have zero experience with them (in fact, I just heard of these today for the first time).

@ghuls
Copy link
Author

ghuls commented May 15, 2024

What would be needed to provide apptainer packages from this repository? I have zero experience with them (in fact, I just heard of these today for the first time).

Nowadays providing a docker container with zsync2 should be enough for endusers to build a apptainer image from it.

@TheAssassin
Copy link
Member

Well, I'm willing to merge a PR that implements Docker image building with Docker's build-push-action. Would you prepare one, please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants