feat(helm): add initial Dask support #821

Alputer · 2024-08-13T14:09:23Z

This pull request contains the ongoing work of dask integration into REANA.

codecov · 2024-08-13T14:11:42Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 31.08%. Comparing base (467be7a) to head (c0d2e92).
Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #821   +/-   ##
=======================================
  Coverage   31.08%   31.08%           
=======================================
  Files          26       26           
  Lines        2487     2487           
=======================================
  Hits          773      773           
  Misses       1714     1714

tiborsimko · 2024-09-09T08:12:02Z

helm/reana/templates/roles.yaml

@@ -22,6 +22,16 @@ rules:
 - apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]
+# Custom dask kubernetes resources


It would be good to wrap the Dask stuff into a conditional variable such as dask.enabled -- similarly as we have traefik.enabled -- because some REANA deployments may not want to (or may not be allowed to) support Dask operators in their deployments.

Also, when dask.enabled is true, it would be good to deploy Dask operator automatically, in quite the same way as we do for traefik; see the dependencies in Chart.yaml.

…b#821)

tiborsimko · 2024-10-29T16:03:20Z

helm/reana/templates/reana-server.yaml

+            value: {{ .Values.dask.cluster_default_single_worker_memory | default "2Gi" }}
+          - name: REANA_DASK_CLUSTER_MAX_SINGLE_WORKER_MEMORY
+            value: {{ .Values.dask.cluster_max_single_worker_memory | default "8Gi" }}
+          {{- end }}          


There is a trailing white space that you can remove here.

tiborsimko · 2024-10-29T16:04:06Z

helm/reana/README.md

@@ -68,6 +68,11 @@ This Helm automatically prefixes all names using the release name to avoid colli
 | `components.reana_workflow_engine_snakemake.environment` | [REANA-Workflow-Engine-Snakemake](https://github.com/reanahub/reana-workflow-engine-snakemake) environment variables | `{}`            |
 | `components.reana_workflow_engine_snakemake.image`       | [REANA-Workflow-Engine-Snakemake image](https://hub.docker.com/r/reanahub/reana-workflow-engine-snakemake) to use | `docker.io/reanahub/reana-workflow-engine-snakemake:<chart-release-version>` |
 | `compute_backends`                                       | List of supported compute backends (kubernetes, htcondorcern, slurmcern)             | "kubernetes"                                    |
+| `dask.enabled` | Install dask-kubernetes-operator custom resources in the cluster to support Dask workflows | false |


Please indent the table so that | separators would appear under previous ones.

There is also trailing white space at some of the line ends.

tiborsimko · 2024-10-29T16:05:33Z

helm/reana/values.yaml

+  cluster_max_memory_limit: "16Gi"
+  cluster_default_number_of_workers: 2
+  cluster_default_single_worker_memory: "2Gi"
+  cluster_max_single_worker_memory: "4Gi"


The value of 4Gi does not correspond to what was advertised in the README file. Please change to 8Gi.

tiborsimko · 2024-10-29T16:09:40Z

helm/reana/README.md

@@ -68,6 +68,11 @@ This Helm automatically prefixes all names using the release name to avoid colli
 | `components.reana_workflow_engine_snakemake.environment` | [REANA-Workflow-Engine-Snakemake](https://github.com/reanahub/reana-workflow-engine-snakemake) environment variables | `{}`            |
 | `components.reana_workflow_engine_snakemake.image`       | [REANA-Workflow-Engine-Snakemake image](https://hub.docker.com/r/reanahub/reana-workflow-engine-snakemake) to use | `docker.io/reanahub/reana-workflow-engine-snakemake:<chart-release-version>` |
 | `compute_backends`                                       | List of supported compute backends (kubernetes, htcondorcern, slurmcern)             | "kubernetes"                                    |
+| `dask.enabled` | Install dask-kubernetes-operator custom resources in the cluster to support Dask workflows | false |


The help could be shorter: Enable support for running Dask workflows.

tiborsimko · 2024-10-29T16:10:00Z

helm/reana/README.md

@@ -68,6 +68,11 @@ This Helm automatically prefixes all names using the release name to avoid colli
 | `components.reana_workflow_engine_snakemake.environment` | [REANA-Workflow-Engine-Snakemake](https://github.com/reanahub/reana-workflow-engine-snakemake) environment variables | `{}`            |
 | `components.reana_workflow_engine_snakemake.image`       | [REANA-Workflow-Engine-Snakemake image](https://hub.docker.com/r/reanahub/reana-workflow-engine-snakemake) to use | `docker.io/reanahub/reana-workflow-engine-snakemake:<chart-release-version>` |
 | `compute_backends`                                       | List of supported compute backends (kubernetes, htcondorcern, slurmcern)             | "kubernetes"                                    |
+| `dask.enabled` | Install dask-kubernetes-operator custom resources in the cluster to support Dask workflows | false |
+| `dask.cluster_max_memory_limit` | Max memory limit for Dask clusters | "16Gi" | 


The maximum memory limit for Dask clusters created by users

tiborsimko · 2024-10-29T16:11:13Z

helm/reana/README.md

@@ -68,6 +68,11 @@ This Helm automatically prefixes all names using the release name to avoid colli
 | `components.reana_workflow_engine_snakemake.environment` | [REANA-Workflow-Engine-Snakemake](https://github.com/reanahub/reana-workflow-engine-snakemake) environment variables | `{}`            |
 | `components.reana_workflow_engine_snakemake.image`       | [REANA-Workflow-Engine-Snakemake image](https://hub.docker.com/r/reanahub/reana-workflow-engine-snakemake) to use | `docker.io/reanahub/reana-workflow-engine-snakemake:<chart-release-version>` |
 | `compute_backends`                                       | List of supported compute backends (kubernetes, htcondorcern, slurmcern)             | "kubernetes"                                    |
+| `dask.enabled` | Install dask-kubernetes-operator custom resources in the cluster to support Dask workflows | false |
+| `dask.cluster_max_memory_limit` | Max memory limit for Dask clusters | "16Gi" | 
+| `dask.cluster_default_number_of_workers` | Number of workers in Dask clusters by default | 2 | 


The number of Dask workflows created by default

tiborsimko · 2024-10-29T16:13:07Z

helm/reana/README.md

+| `dask.enabled` | Install dask-kubernetes-operator custom resources in the cluster to support Dask workflows | false |
+| `dask.cluster_max_memory_limit` | Max memory limit for Dask clusters | "16Gi" | 
+| `dask.cluster_default_number_of_workers` | Number of workers in Dask clusters by default | 2 | 
+| `dask.cluster_default_single_worker_memory` | Amount of memory used by a single Dask worker by default | "2Gi" |


The amount of memory used by default by a single Dask worker

tiborsimko · 2024-10-29T16:13:58Z

helm/reana/README.md

+| `dask.cluster_max_memory_limit` | Max memory limit for Dask clusters | "16Gi" | 
+| `dask.cluster_default_number_of_workers` | Number of workers in Dask clusters by default | 2 | 
+| `dask.cluster_default_single_worker_memory` | Amount of memory used by a single Dask worker by default | "2Gi" |
+| `dask.cluster_max_single_worker_memory` | Maximum amount of memory that can be used by a single Dask worker | "8Gi" |


The maximum amount of memory that users can ask for the single Dask worker

BTW we may want to chat IRL about these, since the autoscaler could affect the variables and their names and/or documentation strings.

tiborsimko

Works nicely 👍

Alputer requested review from tiborsimko and mdonadoni August 13, 2024 14:09

Alputer self-assigned this Aug 13, 2024

Alputer changed the title ~~Dask integration~~ feat(dask): Dask integration (#821) Aug 13, 2024

Alputer force-pushed the dask-integration branch 2 times, most recently from bd35dcc to aed3588 Compare August 28, 2024 15:24

Alputer added a commit to Alputer/reana that referenced this pull request Aug 28, 2024

feat(dask): integrate dask into reana (reanahub#821)

81a79a8

Alputer force-pushed the dask-integration branch from aed3588 to 81a79a8 Compare August 28, 2024 15:28

Alputer added a commit to Alputer/reana that referenced this pull request Aug 28, 2024

feat(dask): update cluster roles (reanahub#821)

302172f

Alputer force-pushed the dask-integration branch from 81a79a8 to 302172f Compare August 28, 2024 15:36

Alputer added a commit to Alputer/reana that referenced this pull request Aug 29, 2024

feat(dask): update cluster roles (reanahub#821)

5d9c0c0

Alputer force-pushed the dask-integration branch from 302172f to 5d9c0c0 Compare August 29, 2024 10:36

Alputer added a commit to Alputer/reana that referenced this pull request Sep 5, 2024

feat(dask): update cluster roles (reanahub#821)

0e74810

Alputer force-pushed the dask-integration branch from 5d9c0c0 to 0e74810 Compare September 5, 2024 19:28

tiborsimko reviewed Sep 9, 2024

View reviewed changes

Alputer added a commit to Alputer/reana that referenced this pull request Sep 9, 2024

feat(dask): introduce dask variable in the helm chart (reanahub#821)

a939c31

Alputer added a commit to Alputer/reana that referenced this pull request Sep 9, 2024

feat(dask): introduce dask variable in the helm chart (reanahub#821)

79e08ba

Alputer force-pushed the dask-integration branch from a939c31 to 79e08ba Compare September 9, 2024 14:20

Alputer added a commit to Alputer/reana that referenced this pull request Sep 9, 2024

feat(dask): introduce dask variable in the helm chart (reanahub#821)

c3c3a8d

Alputer force-pushed the dask-integration branch from 79e08ba to c3c3a8d Compare September 9, 2024 14:25

Alputer added a commit to Alputer/reana that referenced this pull request Sep 9, 2024

feat(dask): add new env_vars to r-workflow-controller (reanahub#821)

7873d62

Alputer added a commit to Alputer/reana that referenced this pull request Sep 10, 2024

feat(dask): ann dask_enabled env var to r-workflow-controller(reanahu…

7f8fc41

…b#821)

Alputer force-pushed the dask-integration branch from 7873d62 to 7f8fc41 Compare September 10, 2024 08:04

Alputer added a commit to Alputer/reana that referenced this pull request Sep 10, 2024

feat(dask): add dask_enabled env var to r-workflow-controller(reanahu…

d97d228

…b#821)

Alputer force-pushed the dask-integration branch from 7f8fc41 to d97d228 Compare September 10, 2024 08:05

Alputer added a commit to Alputer/reana that referenced this pull request Sep 10, 2024

feat(dask): add dask_enabled env var to r-workflow-controller(reanahu…

c677ae9

…b#821)

Alputer added a commit to Alputer/reana that referenced this pull request Sep 10, 2024

feat(dask): update roles for traefik v3 (reanahub#821)

c48c45a

Alputer force-pushed the dask-integration branch from d97d228 to c48c45a Compare September 10, 2024 12:06

Alputer added a commit to Alputer/reana that referenced this pull request Sep 10, 2024

feat(dask): update roles for traefik v3 (reanahub#821)

dfb712a

Alputer added a commit to Alputer/reana that referenced this pull request Oct 16, 2024

feat(dask): update roles for traefik v3 (reanahub#821)

eea4b2a

Alputer added a commit to Alputer/reana that referenced this pull request Oct 16, 2024

feat(dask): add more env vars(reanahub#821)

5f2d979

Alputer added a commit to Alputer/reana that referenced this pull request Oct 16, 2024

feat(dask): adjust parameters (reanahub#821)

1d04b4d

Alputer force-pushed the dask-integration branch from c927f28 to 1d04b4d Compare October 16, 2024 09:16

Alputer added a commit to Alputer/reana that referenced this pull request Oct 24, 2024

feat(dask): update cluster roles (reanahub#821)

6e9a5f6

Alputer added a commit to Alputer/reana that referenced this pull request Oct 24, 2024

feat(dask): introduce dask variable in the helm chart (reanahub#821)

4172e70

Alputer added a commit to Alputer/reana that referenced this pull request Oct 24, 2024

feat(dask): add dask_enabled env var to r-workflow-controller(reanahu…

8907296

…b#821)

Alputer added a commit to Alputer/reana that referenced this pull request Oct 24, 2024

feat(dask): update roles for traefik v3 (reanahub#821)

151ce74

Alputer added a commit to Alputer/reana that referenced this pull request Oct 24, 2024

feat(dask): add more env vars(reanahub#821)

c5101e2

Alputer added a commit to Alputer/reana that referenced this pull request Oct 24, 2024

feat(dask): adjust parameters (reanahub#821)

9fefa51

Alputer force-pushed the dask-integration branch from 1d04b4d to 9fefa51 Compare October 24, 2024 11:04

Alputer added a commit to Alputer/reana that referenced this pull request Oct 24, 2024

feat(helm): add initial dask support (reanahub#821)

7729cc7

Alputer force-pushed the dask-integration branch from 9fefa51 to 7729cc7 Compare October 24, 2024 11:36

Alputer added a commit to Alputer/reana that referenced this pull request Oct 25, 2024

feat(helm): add initial dask support (reanahub#821)

c6faf71

Alputer force-pushed the dask-integration branch from 7729cc7 to c6faf71 Compare October 25, 2024 08:40

tiborsimko changed the title ~~feat(dask): Dask integration (#821)~~ feat(helm): add initial dask support Oct 29, 2024

tiborsimko changed the title ~~feat(helm): add initial dask support~~ feat(helm): add initial Dask support Oct 29, 2024

tiborsimko reviewed Oct 29, 2024

View reviewed changes

Alputer added a commit to Alputer/reana that referenced this pull request Oct 30, 2024

feat(helm): add initial dask support (reanahub#821)

6bd0da3

Alputer force-pushed the dask-integration branch from c6faf71 to 6bd0da3 Compare October 30, 2024 10:56

Alputer added a commit to Alputer/reana that referenced this pull request Oct 30, 2024

feat(helm): add initial dask support (reanahub#821)

89d079f

Alputer force-pushed the dask-integration branch from 6bd0da3 to 89d079f Compare October 30, 2024 12:51

feat(helm): add initial Dask support (reanahub#821)

c0d2e92

Alputer force-pushed the dask-integration branch from 89d079f to c0d2e92 Compare October 31, 2024 14:10

tiborsimko reviewed Oct 31, 2024

View reviewed changes

tiborsimko approved these changes Oct 31, 2024

View reviewed changes

tiborsimko merged commit c0d2e92 into reanahub:master Oct 31, 2024
12 checks passed

github-actions bot mentioned this pull request Oct 23, 2024

chore(master): release 0.9.4 #791

Open

Alputer added a commit to Alputer/reana that referenced this pull request Oct 31, 2024

feat(dask): make dask autoscaler configurable (reanahub#821)

128e7de

Alputer mentioned this pull request Oct 31, 2024

feat(dask): inform users when they request too few resources reanahub/reana-server#703

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(helm): add initial Dask support #821

feat(helm): add initial Dask support #821

Alputer commented Aug 13, 2024

codecov bot commented Aug 13, 2024 •

edited

Loading

tiborsimko Sep 9, 2024

tiborsimko Sep 9, 2024

tiborsimko Oct 29, 2024

tiborsimko Oct 29, 2024

tiborsimko Oct 29, 2024

tiborsimko Oct 29, 2024

tiborsimko Oct 29, 2024

tiborsimko Oct 29, 2024

tiborsimko Oct 29, 2024

tiborsimko Oct 29, 2024

tiborsimko Oct 29, 2024

tiborsimko left a comment

feat(helm): add initial Dask support #821

feat(helm): add initial Dask support #821

Conversation

Alputer commented Aug 13, 2024

codecov bot commented Aug 13, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiborsimko left a comment

Choose a reason for hiding this comment

codecov bot commented Aug 13, 2024 •

edited

Loading