Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] SIGSEGV when trying to clean up a deleted eks cluster #7450

Open
gilbahat opened this issue Jan 4, 2024 · 3 comments
Open

[Bug] SIGSEGV when trying to clean up a deleted eks cluster #7450

gilbahat opened this issue Jan 4, 2024 · 3 comments

Comments

@gilbahat
Copy link

gilbahat commented Jan 4, 2024

What were you trying to accomplish?

SIGSEGV trying to clean up deleted cluster

What happened?

✗ eksctl delete cluster --region us-east-1 --force --name ci-3-graybeard-jarring

2024-01-04 11:10:49 [!] failed to create provider for cluster; force = true skipping: unable to describe cluster control plane: operation error EKS: DescribeCluster, https response error StatusCode: 404, RequestID: 6e63d05c-6a5f-442a-931b-1debe2089bc9, ResourceNotFoundException: No cluster found for name: ci-3-graybeard-jarring.
2024-01-04 11:10:50 [ℹ] deleting EKS cluster "ci-3-graybeard-jarring"
2024-01-04 11:10:52 [!] stack's status of nodegroup named eksctl-ci-3-graybeard-jarring-cluster is DELETE_FAILED
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5042d9b]

goroutine 1 [running]:
github.com/weaveworks/eksctl/pkg/actions/cluster.(*OwnedCluster).Delete(0xc000c72280, {0x6e40390?, 0x95c26a0}, 0xc000ad7500?, 0x6362420?, 0x80?, 0x1, 0x79?, 0x0?)
/private/tmp/eksctl-20231128-4712-d4i5gl/pkg/actions/cluster/owned.go:131 +0x75b
github.com/weaveworks/eksctl/pkg/ctl/delete.doDeleteCluster(0xc00020d2c0, 0x1, 0x0?, 0x0?, 0x0?)
/private/tmp/eksctl-20231128-4712-d4i5gl/pkg/ctl/delete/cluster.go:95 +0x2c2
github.com/weaveworks/eksctl/pkg/ctl/delete.deleteClusterCmd.func1(0xc000024730?, 0x0?, 0x0?, 0x0?, 0x0?)
/private/tmp/eksctl-20231128-4712-d4i5gl/pkg/ctl/delete/cluster.go:20 +0x13
github.com/weaveworks/eksctl/pkg/ctl/delete.deleteClusterWithRunFunc.func1(0xc00096fc00?, {0xc000024730?, 0x4?, 0x63b669b?})
/private/tmp/eksctl-20231128-4712-d4i5gl/pkg/ctl/delete/cluster.go:38 +0xb4
github.com/spf13/cobra.(*Command).execute(0xc000978300, {0xc0000246e0, 0x5, 0x5})
/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xabc
github.com/spf13/cobra.(*Command).ExecuteC(0xc000005800)
/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:1039
main.main()
/private/tmp/eksctl-20231128-4712-d4i5gl/cmd/eksctl/main.go:102 +0x552

How to reproduce it?

  • generate a cloudformation stack and bring it to delete_failed status
  • delete the eks cluster behind it
  • try cleaning up the rest with eksctl

Logs
✗ eksctl delete cluster --region us-east-1 --force --name ci-3-graybeard-jarring -v 4
2024-01-04 11:12:54 [▶] Setting credentials expiry window to 30 minutes
2024-01-04 11:12:55 [▶] role ARN for the current session is "REDACTED"
2024-01-04 11:12:56 [!] failed to create provider for cluster; force = true skipping: unable to describe cluster control plane: operation error EKS: DescribeCluster, https response error StatusCode: 404, RequestID: c91af533-9fdc-45fb-b1b8-6dc4cc8e0b47, ResourceNotFoundException: No cluster found for name: ci-3-graybeard-jarring.
2024-01-04 11:12:56 [▶] Setting credentials expiry window to 30 minutes
2024-01-04 11:12:57 [▶] role ARN for the current session is "REDACTED"
2024-01-04 11:12:57 [ℹ] deleting EKS cluster "ci-3-graybeard-jarring"
2024-01-04 11:12:57 [▶] cfg.json =
{
"kind": "ClusterConfig",
"apiVersion": "eksctl.io/v1alpha5",
"metadata": {
"name": "ci-3-graybeard-jarring",
"region": "us-east-1",
"version": "1.27"
},
"kubernetesNetworkConfig": {
"ipFamily": "IPv4"
},
"iam": {
"withOIDC": false,
"vpcResourceControllerPolicy": true
},
"vpc": {
"cidr": "192.168.0.0/16",
"manageSharedNodeSecurityGroupRules": true,
"autoAllocateIPv6": false,
"nat": {
"gateway": "Single"
}
},
"privateCluster": {
"enabled": false,
"skipEndpointCreation": false
},
"cloudWatch": {
"clusterLogging": {}
}
}

2024-01-04 11:12:59 [▶] cluster "ci-3-graybeard-jarring" was created by eksctl
2024-01-04 11:12:59 [▶] failed to check if cluster is operable: cluster info not available
2024-01-04 11:12:59 [!] stack's status of nodegroup named eksctl-ci-3-graybeard-jarring-cluster is DELETE_FAILED
2024-01-04 11:12:59 [▶] nodegroups = []
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5042d9b]

goroutine 1 [running]:
github.com/weaveworks/eksctl/pkg/actions/cluster.(*OwnedCluster).Delete(0xc000921640, {0x6e40390?, 0x95c26a0}, 0xc000a7dc80?, 0x6362420?, 0x0?, 0x1, 0x96?, 0x0?)
/private/tmp/eksctl-20231128-4712-d4i5gl/pkg/actions/cluster/owned.go:131 +0x75b
github.com/weaveworks/eksctl/pkg/ctl/delete.doDeleteCluster(0xc0007a32c0, 0x1, 0x0?, 0x0?, 0x0?)
/private/tmp/eksctl-20231128-4712-d4i5gl/pkg/ctl/delete/cluster.go:95 +0x2c2
github.com/weaveworks/eksctl/pkg/ctl/delete.deleteClusterCmd.func1(0xc0004c49a0?, 0x0?, 0x0?, 0x0?, 0x0?)
/private/tmp/eksctl-20231128-4712-d4i5gl/pkg/ctl/delete/cluster.go:20 +0x13
github.com/weaveworks/eksctl/pkg/ctl/delete.deleteClusterWithRunFunc.func1(0xc00096fc00?, {0xc0004c49a0?, 0x4?, 0x63b669b?})
/private/tmp/eksctl-20231128-4712-d4i5gl/pkg/ctl/delete/cluster.go:38 +0xb4
github.com/spf13/cobra.(*Command).execute(0xc000978300, {0xc0004c48c0, 0x7, 0x7})
/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:983 +0xabc
github.com/spf13/cobra.(*Command).ExecuteC(0xc000824000)
/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:1039
main.main()
/private/tmp/eksctl-20231128-4712-d4i5gl/cmd/eksctl/main.go:102 +0x552

Anything else we need to know?
MacOS sonoma latest

Versions
eksctl version: 0.165.0-dev+2064e6b32.2023-11-28T18:11:10Z
kubectl version: v1.28.4
OS: darwin

Copy link
Contributor

github-actions bot commented Jan 4, 2024

Hello gilbahat 👋 Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

Copy link
Contributor

github-actions bot commented Feb 4, 2024

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions github-actions bot added the stale label Feb 4, 2024
@TiberiuGC TiberiuGC self-assigned this Apr 9, 2024
@TiberiuGC TiberiuGC removed their assignment Apr 23, 2024
@veekaly
Copy link
Contributor

veekaly commented Apr 23, 2024

Hey @TiberiuGC
I was able to reproduce the issue. The issue is with empty c.ctl.Status.ClusterInfo variable in NewTasksToDeleteClusterWithNodeGroups function - https://github.com/eksctl-io/eksctl/blob/main/pkg/actions/cluster/owned.go#L127C194-L127C226

I can work on the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants