Merge pull request #556 from aws-quickstart/feature/karpenter

Karpenter support for v0.22.0, upgrade guide updates, node role and instance profile cfn exports available as imports
aws-quickstart · Jan 9, 2023 · 7e80943 · 7e80943
2 parents 6c4fd4c + 5e8a140
commit 7e80943
Show file tree

Hide file tree

Showing 6 changed files with 437 additions and 244 deletions.
diff --git a/docs/addons/karpenter.md b/docs/addons/karpenter.md
@@ -49,7 +49,9 @@ const karpenterAddonProps = {
   }],
   amiFamily: "AL2",
   consolidation: { enabled: true },
-  weight: 20
+  ttlSecondsUntilExpired: 2592000,
+  weight: 20,
+  interruptionHandling: true,
 }
 const vpcCniAddOn = new blueprints.addons.VpcCniAddOn();
 const karpenterAddOn = new blueprints.addons.KarpenterAddOn(karpenterAddonProps);
@@ -61,9 +63,10 @@ const blueprint = blueprints.EksBlueprint.builder()
 ```
 
 The add-on automatically sets the following Helm Chart [values](https://github.com/aws/karpenter/tree/main/charts/karpenter#values), and it is **highly recommended** not to pass these values in (as it will result in errors):
-- aws.defaultInstanceProfile
-- clusterEndpoint
-- clusterName
+- settings.aws.defaultInstanceProfile
+- settings.aws.clusterEndpoint
+- settings.aws.clusterName
+- settings.aws.interruptionQueueName (if interruption handling is enabled)
 - serviceAccount.create
 - serviceAccount.name
 - serviceAccount.annotations.eks.amazonaws.com/role-arn
@@ -90,6 +93,7 @@ blueprints-addon-karpenter-54fd978b89-hclmp   2/2     Running   0          99m
 2. Provisioner spec requirement fields are not necessary, as karpenter will dynamically choose (i.e. leaving instance-type blank will let karpenter choose approrpriate sizing).
 3. Consolidation, which is a flag that enables , is supported on versions 0.15.0 and later. It is also mutually exclusive with `ttlSecondsAfterempty`, so if you provide both properties, the addon will throw an error.
 4. Weight, which is a property to prioritize provisioners based on weight, is supported on versions 0.16.0 and later. Addon will throw an error if weight is provided for earlier versions.
+5. Interruption Handling, which is a native way to handle interruption due to involuntary interruption events, is supported on versions 0.19.0 and later. For interruption handling in the earlier versions, Karpenter supports using AWS Node Interruption Handler (which you will need to add as an add-on and ***must be in add-on array after the Karpenter add-on*** for it to work.
 
 ## Using Karpenter
 
@@ -186,6 +190,17 @@ The following are common troubleshooting issues observed when implementing Karpe
 
 1. For Karpenter version older than `0.14.0` deployed on Fargate Profiles, `values.yaml` must be overridden, setting `dnsPolicy` to `Default`. Versions after `0.14.0` has `dnsPolicy` value set default to `Default`. This is to ensure CoreDNS is set correctly on Fargate nodes.
 
+2. With the upgrade to the new OCI registry starting with `v0.17.0`, if you try to upgrade you may get a following error:
+
+```
+Received response status [FAILED] from custom resource. Message returned: Error: b'Error: path "/tmp/tmpkxgr57q5/blueprints-addon-karpenter" not found\n' 
+```
+
+Karpenter, starting from the OCI registry versions, will untar the files under `karpenter` release name only. So if you have previous version deployed under a different release name, you will run into the above error. Therefore, in order to upgrade, you will have to take the following steps:
+
+  1. Remove the existing add-on.
+  2. Re-deploy the Karpenter add-on with the release name `karpenter`.
+
 ## Upgrade Path
 
 1. Using an older version of the Karptner add-on, you may notice the difference in the "provisionerSpecs" property:
@@ -212,4 +227,11 @@ requirements: [
 
 The property is changed to align with the naming convention of the provisioner, and to allow multiple operators (In vs NotIn). The values correspond similarly between the two, with type change being the only difference.
 
-2. Certain upgrades require reapplying the CRDs since Helm does not maintain the lifecycle of CRDs. Please see the [official documentations](https://karpenter.sh/v0.16.0/upgrade-guide/#custom-resource-definition-crd-upgrades) for details.
+2. Certain upgrades require reapplying the CRDs since Helm does not maintain the lifecycle of CRDs. Please see the [official documentations](https://karpenter.sh/v0.16.0/upgrade-guide/#custom-resource-definition-crd-upgrades) for details.
+
+3. Starting with v0.17.0, Karpenter's Helm chart package is stored in OCI (Open Container Initiative) registry. With this change, [charts.karpenter.sh](https://charts.karpenter.sh/) is no longer updated to preserve older versions. You have to adjust for the following:
+
+* The full URL needs to be present (including 'oci://').
+* You need to append a `v` to the version number (i.e. v0.17.0, not 0.17.0)
+
+4. Starting with v0.22.0, Karpenter will no longer work on Kubernetes version prior to 1.21. Either upgrade your Kubernetes to 1.21 or later version and apply Karpenter, or use prior Karpenter versions.
diff --git a/examples/blueprint-construct/index.ts b/examples/blueprint-construct/index.ts
@@ -103,7 +103,7 @@ export default class BlueprintConstruct {
                     { key: 'node.kubernetes.io/instance-type', op: 'In', vals: ['m5.2xlarge'] },
                     { key: 'topology.kubernetes.io/zone', op: 'NotIn', vals: ['us-west-2c']},
                     { key: 'kubernetes.io/arch', op: 'In', vals: ['amd64','arm64']},
-                    { key: 'karpenter.sh/capacity-type', op: 'In', vals: ['spot','on-demand']},
+                    { key: 'karpenter.sh/capacity-type', op: 'In', vals: ['spot']},
                 ],
                 subnetTags: {
                     "Name": "blueprint-construct-dev/blueprint-construct-dev-vpc/PrivateSubnet1",
@@ -117,9 +117,11 @@ export default class BlueprintConstruct {
                     effect: "NoSchedule",
                 }],
                 consolidation: { enabled: true },
-                ttlSecondsUntilExpired: 360,
+                ttlSecondsUntilExpired: 2592000,
                 weight: 20,
+                interruptionHandling: true,
             }),
+            new blueprints.addons.AwsNodeTerminationHandlerAddOn(),
             new blueprints.addons.KubeviousAddOn(),
             new blueprints.addons.EbsCsiDriverAddOn(),
             new blueprints.addons.EfsCsiDriverAddOn({replicaCount: 1}),

diff --git a/lib/addons/aws-node-termination-handler/index.ts b/lib/addons/aws-node-termination-handler/index.ts
@@ -66,10 +66,13 @@ export class AwsNodeTerminationHandlerAddOn extends HelmAddOn {
    */
   deploy(clusterInfo: ClusterInfo): void {
     const cluster = clusterInfo.cluster;    
-    const asgCapacity = clusterInfo.autoscalingGroups;
+    const asgCapacity = clusterInfo.autoscalingGroups || [];
 
-    // No support for Fargate and Managed Node Groups, lets catch that
-    assert(asgCapacity && asgCapacity.length > 0, 'AWS Node Termination Handler is only supported for self-managed nodes');
+    const karpenter = clusterInfo.getScheduledAddOn('KarpenterAddOn');
+    if (!karpenter){
+      // No support for Fargate and Managed Node Groups, lets catch that
+      assert(asgCapacity && asgCapacity.length > 0, 'AWS Node Termination Handler is only supported for self-managed nodes');
+    }    
 
     // Create an SQS Queue
     let helmValues: any;
@@ -82,12 +85,12 @@ export class AwsNodeTerminationHandlerAddOn extends HelmAddOn {
 
     // Get the appropriate Helm Values depending upon the Mode selected
     if (this.options.mode === Mode.IMDS) {
-        helmValues = this.configureImdsMode(serviceAccount);
+        helmValues = this.configureImdsMode(serviceAccount, karpenter);
     }
     else {
-        helmValues = this.configureQueueMode(cluster, serviceAccount, asgCapacity);
+        helmValues = this.configureQueueMode(cluster, serviceAccount, asgCapacity, karpenter);
     }
-
+    
     // Deploy the helm chart
     const awsNodeTerminationHandlerChart = this.addHelmChart(clusterInfo, helmValues);
     awsNodeTerminationHandlerChart.node.addDependency(serviceAccount);
@@ -98,11 +101,13 @@ export class AwsNodeTerminationHandlerAddOn extends HelmAddOn {
    * @param serviceAccount 
    * @returns Helm values
    */
-    private configureImdsMode(serviceAccount: ServiceAccount): any {
+    private configureImdsMode(serviceAccount: ServiceAccount, karpenter: Promise<Construct> | undefined): any {
         return {
             enableSpotInterruptionDraining: true,
             enableRebalanceMonitoring: true,
+            enableRebalanceDraining: karpenter ? true : false,
             enableScheduledEventDraining: true,
+            nodeSelector: karpenter ? {'karpenter.sh/capacity-type': 'spot'} : {},
             serviceAccount: {
                 create: false,
                 name: serviceAccount.serviceAccountName,
@@ -117,7 +122,7 @@ export class AwsNodeTerminationHandlerAddOn extends HelmAddOn {
    * @param asgCapacity
    * @returns Helm values
    */
-    private configureQueueMode(cluster: Cluster, serviceAccount: ServiceAccount, asgCapacity: AutoScalingGroup[]): any {
+    private configureQueueMode(cluster: Cluster, serviceAccount: ServiceAccount, asgCapacity: AutoScalingGroup[], karpenter: Promise<Construct> | undefined): any {
         const queue = new Queue(cluster.stack, "aws-nth-queue", {
             retentionPeriod: Duration.minutes(5)
         });
@@ -133,27 +138,30 @@ export class AwsNodeTerminationHandlerAddOn extends HelmAddOn {
 
         const resources: string[] = [];
 
-        for (let i = 0; i < asgCapacity.length; i++) {
-            const nodeGroup = asgCapacity[i];
-            // Setup a Termination Lifecycle Hook on an ASG
-            new LifecycleHook(cluster.stack, `aws-${nodeGroup.autoScalingGroupName}-nth-lifecycle-hook`, {
-                lifecycleTransition: LifecycleTransition.INSTANCE_TERMINATING,
-                heartbeatTimeout: Duration.minutes(5), // based on https://github.com/aws/aws-node-termination-handler docs
-                notificationTarget: new QueueHook(queue),
-                autoScalingGroup: nodeGroup
-            });
-
-            // Tag the ASG
-            const tags = [{
-                Key: 'aws-node-termination-handler/managed',
-                Value: 'true'
-            }];
-            tagAsg(cluster.stack, nodeGroup.autoScalingGroupName, tags);
-            resources.push(nodeGroup.autoScalingGroupArn);
+        // This does not apply if you leverage Karpenter (which uses NTH for Spot/Fargate)
+        if (!karpenter){
+          for (let i = 0; i < asgCapacity.length; i++) {
+              const nodeGroup = asgCapacity[i];
+              // Setup a Termination Lifecycle Hook on an ASG
+              new LifecycleHook(cluster.stack, `aws-${nodeGroup.autoScalingGroupName}-nth-lifecycle-hook`, {
+                  lifecycleTransition: LifecycleTransition.INSTANCE_TERMINATING,
+                  heartbeatTimeout: Duration.minutes(5), // based on https://github.com/aws/aws-node-termination-handler docs
+                  notificationTarget: new QueueHook(queue),
+                  autoScalingGroup: nodeGroup
+              });
+
+              // Tag the ASG
+              const tags = [{
+                  Key: 'aws-node-termination-handler/managed',
+                  Value: 'true'
+              }];
+              tagAsg(cluster.stack, nodeGroup.autoScalingGroupName, tags);
+              resources.push(nodeGroup.autoScalingGroupArn);
+          }
         }
 
         // Create Amazon EventBridge Rules
-        this.createEvents(cluster.stack, queue);
+        this.createEvents(cluster.stack, queue, karpenter);
 
         // Service Account Policy
         serviceAccount.addToPrincipalPolicy(new iam.PolicyStatement({
@@ -163,7 +171,7 @@ export class AwsNodeTerminationHandlerAddOn extends HelmAddOn {
                 'autoscaling:DescribeAutoScalingInstances',
                 'autoscaling:DescribeTags'
             ],
-            resources
+            resources: karpenter ? ['*'] : resources
         }));
 
         serviceAccount.addToPrincipalPolicy(new iam.PolicyStatement({
@@ -176,10 +184,13 @@ export class AwsNodeTerminationHandlerAddOn extends HelmAddOn {
         return {
             enableSqsTerminationDraining: true,
             queueURL: queue.queueUrl,
+            awsRegion: karpenter ? cluster.stack.region: '',
             serviceAccount: {
                 create: false,
                 name: serviceAccount.serviceAccountName,
-            }
+            },
+            checkASGTagBeforeDraining: karpenter ? false : true,
+            enableSpotInterruptionDraining: karpenter ? true : false,
         };
     }
 
@@ -188,13 +199,9 @@ export class AwsNodeTerminationHandlerAddOn extends HelmAddOn {
    * @param scope 
    * @param queue 
    */
-  private createEvents(scope: Construct, queue: Queue): void {
+  private createEvents(scope: Construct, queue: Queue, karpenter: Promise<Construct> | undefined): void {
     const target = new SqsQueue(queue);
     const eventPatterns: EventPattern[] = [
-      {
-        source: ['aws.autoscaling'],
-        detailType: ['EC2 Instance-terminate Lifecycle Action']
-      },
       {
         source: ['aws.ec2'],
         detailType: ['EC2 Spot Instance Interruption Warning']
@@ -209,10 +216,19 @@ export class AwsNodeTerminationHandlerAddOn extends HelmAddOn {
       },
       {
         source: ['aws.health'],
-        detailType: ['AWS Health Even'],
+        detailType: ['AWS Health Event'],
       }
     ];
 
+    if (!karpenter){
+      eventPatterns.push(
+        {
+          source: ['aws.autoscaling'],
+          detailType: ['EC2 Instance-terminate Lifecycle Action']
+        },
+      );
+    }
+
     eventPatterns.forEach((event, index) => {
       const rule = new Rule(scope, `rule-${index}`, { eventPattern: event });
       rule.addTarget(target);

diff --git a/lib/addons/karpenter/iam.ts b/lib/addons/karpenter/iam.ts
@@ -1,28 +1,30 @@
 export const KarpenterControllerPolicy = {
-  "Version": "2012-10-17",
-  "Statement": [
-      {
-          "Effect": "Allow",
-          "Action": [
-              "ec2:CreateLaunchTemplate",
-              "ec2:CreateFleet",
-              "ec2:RunInstances",
-              "ec2:CreateTags",
-              "iam:PassRole",
-              "ec2:TerminateInstances",
-              "ec2:DeleteLaunchTemplate",
-              "ec2:DescribeLaunchTemplates",
-              "ec2:DescribeInstances",
-              "ec2:DescribeSecurityGroups",
-              "ec2:DescribeSubnets",
-              "ec2:DescribeInstanceTypes",
-              "ec2:DescribeInstanceTypeOfferings",
-              "ec2:DescribeAvailabilityZones",
-              "ssm:GetParameter",
-              "pricing:GetProducts",
-              "ec2:DescribeSpotPriceHistory",
-          ],
-          "Resource": "*"
-      }
-  ]
+"Version": "2012-10-17",
+"Statement": [
+    {
+        "Effect": "Allow",
+        "Action": [
+        // Write Operations
+            "ec2:CreateLaunchTemplate",
+            "ec2:CreateFleet",
+            "ec2:RunInstances",
+            "ec2:CreateTags",
+            "ec2:TerminateInstances",
+            "ec2:DeleteLaunchTemplate",
+        // Read Operations
+            "ec2:DescribeLaunchTemplates",
+            "ec2:DescribeInstances",
+            "ec2:DescribeSecurityGroups",
+            "ec2:DescribeSubnets",
+            "ec2:DescribeImages",
+            "ec2:DescribeInstanceTypes",
+            "ec2:DescribeInstanceTypeOfferings",
+            "ec2:DescribeAvailabilityZones",
+            "ec2:DescribeSpotPriceHistory",
+            "ssm:GetParameter",
+            "pricing:GetProducts",
+        ],
+        "Resource": "*"
+    }
+]
 };