Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Azure deployment guide #840

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

TomAugspurger
Copy link
Member

This rewrites the Azure deployment guide. I'll detail the major changes below, but I wanted to ask up front: does pangeo want to be in the business of maintaining these guides, especially with 2i2c taking over maintenance roles? cc @tjcrone, @rabernat, and maybe @choldgraf. In terms of actual maintenance cost I'm happy to keep the Azure guide up to date.

Selfishly, I just want a place to link to for "How do I replicate the Planetary Computer Hub in my Azure subscription?". That could live at https://planetarycomputer.microsoft.com/docs/concepts/computing if pangeo doesn't want to maintain these guides anymore.

Last bit of house-keeping: why not just use zero to jupyterhub? Two reasons:

  1. There's a bit of awkwardness in "go to this other guide, but stop before deploying jupyterhub, since we're actually using this other helm chart". Not the worst in the world, but it's nice to read everything in one place (at the cost of maintenance).
  2. https://zero-to-jupyterhub.readthedocs.io/en/latest/kubernetes/microsoft/step-zero-azure.html has some recommendations around networking and AKS auth I need to investigate. Maybe just an issue with AKS best-practices having moved on since that was written.

Onto the changes: I don't think the diff is too meaningful. But at a high level, the guide has two sections: A "simple" deployment, just to make sure that the tools work, and a "recommended" deployment. I've also included a sample Makefile and config.yaml, to satisfy people who just want to get a cluster up and running ASAP, and don't care about the background.

The details below has the rendered HTML as a png

pangeo-azure

can use to log into the Azure web interface. If you have an existing Microsoft
account (for example, a hotmail.com or outlook.com email address) then you can use
that, or you can create a new account.
These examples create the Azure Resources in the West Europe region. This is a good
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd feel better with an explicit +1 that it's Ok to include this bit of self-promotion :) I think most pangeo users will be interested in the public data we're hosting there, and so will benefit from knowing about it and putting their compute in the same region as that data.

so we'll cover each one in turn.
This guide uses the Azure CLI to create the Azure Resources.
At the end, we provide a ``Makefile`` with targets for the AKS cluster and an updated deployment. If you're just looking to deploy a Hub, feel free to use and adapt the Makefiles. If you're looking to build understanding, read through the guide.
As an alternative to this guide, you might use `Qhub <https://docs.qhub.dev/en/latest/>`_, which provides a higher-level tool to obtain a JupyterHub and Dask deployment on Kubernetes (or HPC).
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still exploring QHub. I ran into a couple issues with my initial attempt on Azure, so I've reverted to just using the Azure CLI and Helm in this guide.

@choldgraf
Copy link

choldgraf commented Jun 8, 2021

Thanks a bunch for this PR! I haven't had a chance to look at the content (and TBH am probably not super qualified to decide haha). But here's my take on a few of your more "meta" points:

  • Should Pangeo maintain a guide like this?
    • Ultimately it's the Pangeo community's decision, but my 2 cents is that there is definitely value in having Pangeo-branded guides like this. (with my 2i2c hat on now) While 2i2c will operate the infrastructure for Pangeo, we consider to be operating Pangeo's stack on behalf of Pangeo, not a 2i2c-specific stack. So I'd rather have a community supported "best practices" guide out there that 2i2c can point to and say "2i2c is basically just following these steps".
  • Makefiles etc vs step-by-step
    • I think this depends on the kind of value you want these guides to provide to people. With Z2JH we intentionally didn't provide a bunch of "auto deploy" scripts in part because we felt that'd encourage people to just run the script without really learning about the infrastructure (which, if you're going to be running these deployments in the real world, I think you need to learn about). I think that's the danger in the "just copypaste this script or use this abstraction" approach. Obviously that's a complicated topic (AKS itself is a pretty big abstraction!) but that's my 2 cents anyway :-)
  • Could this be in Z2JH?

I also encourage you to update the Z2JH Azure guide if there are now new best practices :-)

Also more generally, @sgibson91 is joining the team soon, and will be focusing much of her time on Pangeo. She is also experience with Azure via her work at the Turing Institute, and I wonder if this might be a good collaboration point as she gets started?

@sgibson91
Copy link

Hi! @TomAugspurger I also probably wrote the networking stuff in the z2jh Azure guide, so very happy to investigate further with you and update as needed (it was a while ago that I wrote it!)

@rsignell-usgs
Copy link
Member

@TomAugspurger should this be merged?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants