Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: support for ksctl controllers and ksctl agent #304

Merged
merged 40 commits into from
May 1, 2024
Merged

feature: support for ksctl controllers and ksctl agent #304

merged 40 commits into from
May 1, 2024

Conversation

dipankardas011
Copy link
Member

@dipankardas011 dipankardas011 commented Mar 13, 2024 •
edited

Tasks description

Issues

Completed Issue(s)

Solution

Sub-Tasks

  • milestones or sub-task which needs to be coverted

Note to reviewers

@dipankardas011 dipankardas011 linked an issue Mar 13, 2024 that may be closed by this pull request
[ACTION-TRACKING] auto-scaling of self-managed ha clusters #251
Open
@dipankardas011
Copy link
Member Author

@prateek041 you can close the PR #301 and use this branch instead

@github-actions github-actions bot added the go Pull requests that update Go code label Mar 13, 2024
@dipankardas011
Copy link
Member Author

this will be helpful
https://protobuf.dev/programming-guides/proto3/

@dipankardas011
Copy link
Member Author

dipankardas011 commented Mar 14, 2024 •
edited

For trying out the server you can use grpcui

grpcui --plaintext 127.0.0.1:8080

@github-actions github-actions bot added the area/ci ci/cd label Mar 15, 2024
@dipankardas011 dipankardas011 mentioned this pull request Mar 15, 2024
Merged
3 tasks
@dipankardas011 dipankardas011 force-pushed the 251-feature-auto-scaling-of-self-managed-ha-clusters branch 2 times, most recently from bb37070 to 29689ff Compare March 23, 2024 15:50
@dipankardas011 dipankardas011 force-pushed the 251-feature-auto-scaling-of-self-managed-ha-clusters branch 3 times, most recently from 1f859fb to 2badd33 Compare April 1, 2024 14:33
@dipankardas011 dipankardas011 marked this pull request as draft April 1, 2024 14:35
@dipankardas011 dipankardas011 changed the title feature: support for k8s controller for autoscaler and ksctl agent feature: support for k8s controller for autoscaler and ksctl agent [WIP] Apr 1, 2024
@dipankardas011
Copy link
Member Author

@prateek041 I will be working on branch 251-sub-feature-ksctl-agent

@dipankardas011 dipankardas011 mentioned this pull request Apr 1, 2024
Merged
16 tasks
@dipankardas011 dipankardas011 force-pushed the 251-feature-auto-scaling-of-self-managed-ha-clusters branch from 2badd33 to 31768c3 Compare April 2, 2024 17:59
@dipankardas011
moved the diagrams to enhancement repo
6bc369b 
Signed-off-by: Dipankar Das <[email protected]>
@dipankardas011
fix: failing unit test for cloud providers
c54dabc 
Signed-off-by: Dipankar Das <[email protected]>
@dipankardas011
patch: changes according to review
7e5ecaf 
Signed-off-by: Dipankar Das <[email protected]>
@dipankardas011
patch(e2e)!: upgraded nodeSize, no WorkerNodes during creation kubead…
b8c3f34 
…m cluster

Signed-off-by: Dipankar Das <[email protected]>
@dipankardas011
patch(e2e)!: Added cniplugin which was missing from kubeadm clusters
2614137 
Signed-off-by: Dipankar Das <[email protected]>
@dipankardas011
sub-feature: ksctl agent (#322)
243e649 
# Tasks description

This hosts as a parent pr for:
- #325
- #327

It also added implementation for

| Kind | Group | Version |
|--------|--------|--------|
| importStates | storage.ksctl.com | v1alpha1 |
| stacks | application.ksctl.com | v1alpha1 | 

Ksctl agent for the handle of requests from various controllers

Using this you can achieve the application creation, deletion, version
changes seemlessly with only the use of crd (currently single app
support)

Improved the Makefile and the testing of the core via the adding
solution for #316

## Issues
### Completed Issue(s)

### Related Issue(s)
- #251
- #304 (base-feature)
- Closes #316


# Solution

### Sub-Tasks
- [x] Ksctl agent planning
- [x] Document all the things in
ksctl/enhancements#1
- [x] ksctl grpc proto file
- [x] Ksctl storage export
- [x] Ksctl storage import
- [x] Test cases
- [x] controller for the storage export to the newly/already created
ksctl or non-ksctl(Currently not supported) cluster
- [x] controller for the application
  - [x] creation
  - [x] deletion
  - [x] store the application information in stateDocuemnt
  - [x] updating the version based on current state of the crd
- [x] improve the logging for the kind and helm client
- [x] Add support for the incluster kubernetes environment (kube go
client, helm, storageDriver)


# Note to reviewers

> [!CAUTION]
> make sure you are fixing the URL for the controllers present to more
appropriate url. Make sure it is stable and not just main branch

- [x] Ran Tests locally
- [x] Checked [Contribution's
guidelines](https://docs.ksctl.com/docs/contribution-guidelines/)
@dipankardas011 dipankardas011 marked this pull request as ready for review April 29, 2024 04:20
@dipankardas011
Copy link
Member Author

dipankardas011 commented Apr 29, 2024 •
edited

TODO 📓

  • need to provide proper versioning for the ksctl controllers
    • we are going for main branch and as there is almost no changes in the main branch
    • and we always want it to be latest so we can keep it
  • Finish the documentation for the feature in enhancement proposal
    • for kubernetes storage
    • for local storage (new methods)
    • for the export(), import() functionality
    • ksctl agent, and ksctl controllers
    • Aws docs @Horiodino
    • add your existing cluster to ksctl (aka Non-ksctl -> ksctl) Mark as TBD
  • docker buildx fix
    • update github action workflow
  • Test cases and the coverage for the controllers
  • assess any missing part
  • Improving the codebase
  • Come up with the automation for the releases of the new controllers
  • Fix the Issue and PR linking as we are not going to add feature for the ksctl autoscaler in this PR cycle

@dipankardas011 dipankardas011 changed the title feature: support for k8s controller for autoscaler and ksctl agent [WIP] feature: support for ksctl controllers and ksctl agent [WIP] Apr 29, 2024
@dipankardas011 dipankardas011 added the tests/enable it enables the PR to start tests label May 1, 2024
@codecov Codecov
Copy link

codecov bot commented May 1, 2024 •
edited

Codecov Report

Attention: Patch coverage is 69.81132% with 240 lines in your changes are missing coverage. Please review.

Project coverage is 68.02%. Comparing base (6c6bb2b) to head (b1b700a).
Report is 372 commits behind head on main.

Files Patch % Lines
internal/storage/kubernetes/kubernetes.go 77.18% 62 Missing and 24 partials ⚠️
internal/storage/kubernetes/client.go 34.56% 49 Missing and 4 partials ⚠️
internal/storage/external/mongodb/mongodb.go 72.72% 24 Missing and 6 partials ⚠️
internal/storage/local/local.go 75.40% 22 Missing and 8 partials ⚠️
internal/cloudproviders/local/custom-logger.go 0.00% 17 Missing ⚠️
internal/cloudproviders/azure/main.go 57.14% 6 Missing ⚠️
internal/cloudproviders/civo/main.go 62.50% 6 Missing ⚠️
pkg/helpers/fields.go 85.71% 4 Missing and 2 partials ⚠️
internal/cloudproviders/local/main.go 33.33% 4 Missing ⚠️
internal/cloudproviders/local/localgo.go 0.00% 1 Missing ⚠️
... and 1 more
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #304      +/-   ##
==========================================
+ Coverage   66.04%   68.02%   +1.97%     
==========================================
  Files          33       40       +7     
  Lines        5057     6811    +1754     
==========================================
+ Hits         3340     4633    +1293     
- Misses       1429     1805     +376     
- Partials      288      373      +85

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dipankardas011 dipankardas011 added the pr/lgtm This PR look good to the reviewer label May 1, 2024
@dipankardas011 dipankardas011 changed the title feature: support for ksctl controllers and ksctl agent [WIP] feature: support for ksctl controllers and ksctl agent May 1, 2024
@dipankardas011 dipankardas011 merged commit 7dc3578 into main May 1, 2024
89 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@dipankardas011 dipankardas011

@prateek041 prateek041

Labels
area/ci ci/cd area/pkg core area/test cloud/aws aws cloud/azure azure cloud/civo cloud cloud/local local distro/k3s k3s distro/kubeadm kubeadm distro/kubernetes-client-go go Pull requests that update Go code pr/lgtm This PR look good to the reviewer state/local state store in local tests/enable it enables the PR to start tests
Projects
None yet
Milestone
No milestone
2 participants
@dipankardas011 @prateek041