system: Introduce Kueue as an optional alternative approach how to submit user jobs

[xaviertintin]

This pull request introduces changes to the reana-workflow-controller component to enable Kueue as an optional job submission method in REANA.

Key Changes:

1. Environment Variable Configuration:

• Retrieves the KUEUE_ENABLED environment variable within reana-workflow-controller/reana_workflow_controller/config.py. This variable reflects the admin's selection of Kueue during deployment via Helm values.

2. Deployment Type Selection:

• Based on the retrieved environment variable, the code in reana-workflow-controller/reana_workflow_controller/workflow_run_manager.py dynamically selects the appropriate job submission method:
  • Standard: If KUEUE_ENABLED is False, the existing Kubernetes Job API is used for job submission.
  • Kueue: If KUEUE_ENABLED is True, the pull request implements the necessary logic to submit jobs through Kueue.

3. Downstream Communication:

• The chosen deployment type (standard or kueue) is passed as an environment variable (job_controller_env_vars) to downstream components for further processing.

Benefits:

• Provides REANA admins with the flexibility to choose between the standard Kubernetes Job API and Kueue for job submission.
• Enables a more scalable and potentially efficient approach to job management depending on admin requirements. (Testing is required to confirm performance benefits.)