Add scaffold for Yunikorn batch scheduler

[jacobsalway]

🛑 Important:

Please open an issue to discuss significant work before you start. We appreciate your contributions and don't want your efforts to go to waste!

For guidelines on how to contribute, please review the CONTRIBUTING.md document.

Purpose of this PR

Add an initial scaffold for a Yunikorn batch scheduler implementation. Splitting the PRs so I can incrementally finish work without leaving a mega PR to be reviewed.

Ref #2098

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

Will close #2042 once I've got that functionality merged in separate PRs

[vara-bonthu]

Thanks for the PR @jacobsalway . This is looks good to me.

It would be great to add some examples, documentation, and unit tests for using YuniKorn with the Spark Operator.

[jacobsalway]

@vara-bonthu I'm happy to put everything under one PR with docs, test etc. but thought it might be easier to split the work into separate PRs under this issue to avoid draft PRs sitting around for a while #2098

[ChenYi015]

Hi @jacobsalway , this work is great. I have modified the batch scheduler framework in #2072, and we should release v2.0.0-rc.0 soon, so would you mind rebase your work on this new 2.0 version.

[jacobsalway]

Hey @ChenYi015 once your controller runtime PR is merged to release-2.0 I'll rebase and retarget my PR manually. Pulling in your branch commits here is a bit messy

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment