[Help] Pod does not have the eks-pod-identity-token.

[laiminhtrung1997]

What help do you need?

I have an issue with using EKS Pod Identity with the external-dns.
After I associated the IAM Role with ServiceAccount, I deployed the external-dns by using helm install immediately.
The issue is the pod external-dns is not mounted by the volume eks-pod-identity-token, so it cannot do some actions to AWS Service Route53.
I think there is a time delay after associating ServiceAccount with the IAM Role, or maybe something else. I have no idea.
So could someone please help me out with this scenario?

[github-actions]

Hello laiminhtrung1997 👋 Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

[TiberiuGC]

Hi @laiminhtrung1997 - can you please share the exact commands you're running?

[laiminhtrung1997]

Dear @TiberiuGC ,
So much thanks for your reply.
I run all these steps with ansible playbook. These are my steps.

1. Create an IAM Role with the name $IAM_ROLE manually in the AWS Console.
2. Create an IAM Policy with name $IAM_POLICY using the ansible module and attach it to the IAM Role created above.
3. Associate the IAM Role with K8S ServiceAccount with the command below

eksctl create podidentityassociation \
    --cluster $EKS_CLUSTER \
    --namespace $NAMESPACE \
    --service-account-name $SERVICE_ACCOUNT \
    --role-arn arn:aws:iam::$AWS_ACCOUNT_ID:role/$IAM_ROLE

4. Install the external-dns with the helm command.
5. Run kubectl logs to log the external-dns pod with errors showing that it does not have an identity to authorize the action ListHostedZones.
6. Run kubectl get pod -o yaml and find out the external-pod does not have the eks-pod-identity-token file.
7. Run kubectl delete pod to restart the pod and the new pod has the eks-pod-identity-token file.

[TiberiuGC]

Thank you for laying out all the steps!

Given that you're creating the policy and role in advance, for this use case in particular, eksctl is merely calling EKS::CreatePodIdentityAssociation, which is a synchronous operation. Thus, by the time eksctl finishes creating the association, everything should be setup correctly. What you are seeing could be an upstream issue, I've searched for related issues, but unfortunately couldn't find any. 😞

On a separate note, although unrelated to the issue, eksctl can create both policies and roles for you, so that you don't have to do additional manual steps. e.g.

iam:  
  podIdentityAssociations: 
  - namespace: $NAMESPACE
    serviceAccountName: $SERVICE_ACCOUNT
    roleName: $IAM_ROLE
    permissionPolicy:
      Version: "2012-10-17"
      Statement:
      - Effect: Allow
        Action:
        - "autoscaling:DescribeAutoScalingGroups"
        - "ec2:DescribeLaunchTemplateVersions"
        Resource: '*'

and run

eksctl create podidentityassociation -f config.yaml

[laiminhtrung1997]

Dear @TiberiuGC
Thank you so much for your time.

Regarding the upstream issue, you mentioned, is it caused by the association or the IAM Policy creation? What I mean is, should I wait for 15 seconds after creating the IAM Policy or after the association?

As for the suggestion, the reason I do that is because I have multiple EKS clusters, and I want to use a single IAM Policy for all clusters by employing the ABAC concept. Each cluster will use its IAM Role. However, the AWS Service Route53 does not support ABAC, so I have to switch from creating IAM Policies to creating IAM Roles. This means I create a single IAM Role for all clusters, with each cluster having its IAM Policy. I think I will post a discussion about this in another post.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[laiminhtrung1997]

Anyone can help me?