fix(aws): Add support for eks pod identities to NewSession() #4382
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
|
k8s-ci-robot
added
the
cncf-cla: no
|
Welcome @AleCo3lho! |
k8s-ci-robot
added
the
needs-ok-to-test
|
Hi @AleCo3lho. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
size/S
|
Still working with IRSA. apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2024-04-12T01:08:14Z"
generateName: external-dns-6b64ff6ff6-
labels:
app: external-dns
name: external-dns-6b64ff6ff6-28flv
namespace: external-dns
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: external-dns-6b64ff6ff6
spec:
containers:
- args:
- --source=service
- --source=ingress
- --domain-filter=***.dev
- --provider=aws
- --policy=upsert-only
- --aws-zone-type=public
- --registry=txt
- --txt-owner-id=***
env:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
- name: AWS_DEFAULT_REGION
value: us-east-1
- name: AWS_REGION
value: us-east-1
- name: AWS_ROLE_ARN
value: arn:aws:iam::***:role/prod-eksClusterExternalDnsRole
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
image: public.ecr.aws/f1l1q5r9/prod-registry:v0.14.1-36-gc06d4ce1-dirty
imagePullPolicy: IfNotPresent
name: external-dns
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-mxztn
readOnly: true
- mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
name: aws-iam-token
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: ip-10-0-2-242.ec2.internal
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: external-dns
serviceAccountName: external-dns
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: aws-iam-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: sts.amazonaws.com
expirationSeconds: 86400
path: token
- name: kube-api-access-mxztn
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-04-12T01:08:19Z"
status: "True"
type: PodReadyToStartContainers
- lastProbeTime: null
lastTransitionTime: "2024-04-12T01:08:14Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2024-04-12T01:08:19Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2024-04-12T01:08:19Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2024-04-12T01:08:14Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://fadc445c6c09e16564360e4087a27be0980816e220de3ea6295a16b0e4a35a18
image: public.ecr.aws/f1l1q5r9/prod-registry:v0.14.1-36-gc06d4ce1-dirty
imageID: public.ecr.aws/f1l1q5r9/prod-registry@sha256:3d64e3c46f7192bf263473b3b9a80df8bc48b182c1d91bec2336782f22370bd0
lastState: {}
name: external-dns
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-04-12T01:08:18Z"
hostIP: 10.0.2.242
hostIPs:
- ip: 10.0.2.242
phase: Running
podIP: 10.0.2.36
podIPs:
- ip: 10.0.2.36
qosClass: BestEffort
startTime: "2024-04-12T01:08:14Z"
time="2024-04-12T01:08:18Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s DefaultTargets:[] GlooNamespaces:[gloo-system] SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter: LabelFilter: IngressClassNames:[] FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false GatewayNamespace: GatewayLabelFilter: Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[coelhor.dev] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] TargetNetFilter:[] ExcludeTargetNets:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType:public AWSZoneTagFilter:[] AWSAssumeRole: AWSAssumeRoleExternalID: AWSBatchChangeSize:1000 AWSBatchChangeSizeBytes:32000 AWSBatchChangeSizeValues:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AWSSDServiceCleanup:false AWSZoneMatchParent:false AWSDynamoDBRegion: AWSDynamoDBTable:external-dns AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatDNSConfiguration: BluecatConfigFile:/etc/kubernetes/bluecat.json BluecatDNSView: BluecatGatewayHost: BluecatRootZone: BluecatDNSServerName: BluecatDNSDeployType:no-deploy BluecatSkipTLSVerify:false CloudflareProxied:false CloudflareDNSRecordsPerPage:100 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 InfobloxFQDNRegEx: InfobloxNameRegEx: InfobloxCreatePTR:false InfobloxCacheDuration:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml OCICompartmentOCID: OCIAuthInstancePrincipal:false OCIZoneScope:GLOBAL OCIZoneCacheDuration:0s InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSSkipTLSVerify:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:upsert-only Registry:txt TXTOwnerID:Z07278093MVSWFHHWH1RW TXTPrefix: TXTSuffix: TXTEncryptEnabled:false TXTEncryptAESKey: Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint: ExoscaleAPIKey: ExoscaleAPISecret: ExoscaleAPIEnvironment:api ExoscaleAPIZone:ch-gva-2 CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: ResolveServiceLoadBalancerHostname:false RFC2136Host: RFC2136Port:0 RFC2136Zone:[] RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136BatchChangeSize:50 RFC2136UseTLS:false RFC2136SkipTLSVerify:false NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A AAAA CNAME] ExcludeDNSRecordTypes:[] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false OCPRouterName: IBMCloudProxied:false IBMCloudConfigFile:/etc/kubernetes/ibmcloud.json TencentCloudConfigFile:/etc/kubernetes/tencent-cloud.json TencentCloudZoneType: PiholeServer: PiholePassword: PiholeTLSInsecureSkipVerify:false PluralCluster: PluralProvider: WebhookProviderURL:http://localhost:8888 WebhookProviderReadTimeout:5s WebhookProviderWriteTimeout:10s WebhookServer:false TraefikDisableLegacy:false TraefikDisableNew:false}"
time="2024-04-12T01:08:18Z" level=info msg="Instantiating new Kubernetes client"
time="2024-04-12T01:08:18Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2024-04-12T01:08:18Z" level=info msg="Created Kubernetes client https://172.20.0.1:443"
time="2024-04-12T01:08:19Z" level=info msg="Applying provider record filter for domains: [***.dev. .***.dev.]"
time="2024-04-12T01:08:19Z" level=info msg="All records are already up to date"
time="2024-04-12T01:09:19Z" level=info msg="Applying provider record filter for domains: [***.dev. .***.dev.]"
time="2024-04-12T01:09:19Z" level=info msg="All records are already up to date"
time="2024-04-12T01:10:19Z" level=info msg="Applying provider record filter for domains: [***.dev. .***.dev.]"
time="2024-04-12T01:10:19Z" level=info msg="All records are already up to date"
time="2024-04-12T01:11:20Z" level=info msg="Applying provider record filter for domains: [***dev. .***.dev.]"
time="2024-04-12T01:11:20Z" level=info msg="All records are already up to date" |
|
Same for eks pod identity apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2024-04-12T01:27:07Z"
generateName: external-dns-6b64ff6ff6-
labels:
app: external-dns
pod-template-hash: 6b64ff6ff6
name: external-dns-6b64ff6ff6-5fbtg
namespace: external-dns
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: external-dns-6b64ff6ff6
uid: e0c0d755-a299-456e-8631-afed983c7b97
resourceVersion: "41082"
uid: 9f9bd5b3-c8b9-4325-a7e1-4037886ac233
spec:
containers:
- args:
- --source=service
- --source=ingress
- --domain-filter=***.dev
- --provider=aws
- --policy=upsert-only
- --aws-zone-type=public
- --registry=txt
- --txt-owner-id=***
env:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
- name: AWS_DEFAULT_REGION
value: us-east-1
- name: AWS_REGION
value: us-east-1
- name: AWS_CONTAINER_CREDENTIALS_FULL_URI
value: http://169.254.170.23/v1/credentials
- name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
value: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
image: public.ecr.aws/f1l1q5r9/prod-registry:v0.14.1-36-gc06d4ce1-dirty
imagePullPolicy: IfNotPresent
name: external-dns
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-pqd5j
readOnly: true
- mountPath: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount
name: eks-pod-identity-token
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: ip-10-0-2-43.ec2.internal
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: external-dns
serviceAccountName: external-dns
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: eks-pod-identity-token
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: pods.eks.amazonaws.com
expirationSeconds: 86400
path: eks-pod-identity-token
- name: kube-api-access-pqd5j
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-04-12T01:27:10Z"
status: "True"
type: PodReadyToStartContainers
- lastProbeTime: null
lastTransitionTime: "2024-04-12T01:27:07Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2024-04-12T01:27:10Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2024-04-12T01:27:10Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2024-04-12T01:27:07Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://d51b73fc4178f8bebb4c4caa23b6dc8674c3427a0625267e35aff21ec3a9c91c
image: public.ecr.aws/f1l1q5r9/prod-registry:v0.14.1-36-gc06d4ce1-dirty
imageID: public.ecr.aws/f1l1q5r9/prod-registry@sha256:3d64e3c46f7192bf263473b3b9a80df8bc48b182c1d91bec2336782f22370bd0
lastState: {}
name: external-dns
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-04-12T01:27:10Z"
hostIP: 10.0.2.43
hostIPs:
- ip: 10.0.2.43
phase: Running
podIP: 10.0.2.19
podIPs:
- ip: 10.0.2.19
qosClass: BestEffort
startTime: "2024-04-12T01:27:07Z"time="2024-04-12T01:27:10Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s DefaultTargets:[] GlooNamespaces:[gloo-system] SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter: LabelFilter: IngressClassNames:[] FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false GatewayNamespace: GatewayLabelFilter: Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[coelhor.dev] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] TargetNetFilter:[] ExcludeTargetNets:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType:public AWSZoneTagFilter:[] AWSAssumeRole: AWSAssumeRoleExternalID: AWSBatchChangeSize:1000 AWSBatchChangeSizeBytes:32000 AWSBatchChangeSizeValues:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AWSSDServiceCleanup:false AWSZoneMatchParent:false AWSDynamoDBRegion: AWSDynamoDBTable:external-dns AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatDNSConfiguration: BluecatConfigFile:/etc/kubernetes/bluecat.json BluecatDNSView: BluecatGatewayHost: BluecatRootZone: BluecatDNSServerName: BluecatDNSDeployType:no-deploy BluecatSkipTLSVerify:false CloudflareProxied:false CloudflareDNSRecordsPerPage:100 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 InfobloxFQDNRegEx: InfobloxNameRegEx: InfobloxCreatePTR:false InfobloxCacheDuration:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml OCICompartmentOCID: OCIAuthInstancePrincipal:false OCIZoneScope:GLOBAL OCIZoneCacheDuration:0s InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSSkipTLSVerify:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:upsert-only Registry:txt TXTOwnerID:Z07278093MVSWFHHWH1RW TXTPrefix: TXTSuffix: TXTEncryptEnabled:false TXTEncryptAESKey: Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint: ExoscaleAPIKey: ExoscaleAPISecret: ExoscaleAPIEnvironment:api ExoscaleAPIZone:ch-gva-2 CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: ResolveServiceLoadBalancerHostname:false RFC2136Host: RFC2136Port:0 RFC2136Zone:[] RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136BatchChangeSize:50 RFC2136UseTLS:false RFC2136SkipTLSVerify:false NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A AAAA CNAME] ExcludeDNSRecordTypes:[] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false OCPRouterName: IBMCloudProxied:false IBMCloudConfigFile:/etc/kubernetes/ibmcloud.json TencentCloudConfigFile:/etc/kubernetes/tencent-cloud.json TencentCloudZoneType: PiholeServer: PiholePassword: PiholeTLSInsecureSkipVerify:false PluralCluster: PluralProvider: WebhookProviderURL:http://localhost:8888 WebhookProviderReadTimeout:5s WebhookProviderWriteTimeout:10s WebhookServer:false TraefikDisableLegacy:false TraefikDisableNew:false}"
time="2024-04-12T01:27:10Z" level=info msg="Instantiating new Kubernetes client"
time="2024-04-12T01:27:10Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2024-04-12T01:27:10Z" level=info msg="Created Kubernetes client https://172.20.0.1:443"
time="2024-04-12T01:27:11Z" level=info msg="Applying provider record filter for domains: [***.dev. .***.dev.]"
time="2024-04-12T01:27:11Z" level=info msg="All records are already up to date" |
|
Thanks for this PR. |
k8s-ci-robot
added
cncf-cla: no
k8s-ci-robot
added
cncf-cla: yes
|
@AleCo3lho we have not the time to guide too much. From my side I also don't see a fix and don't understand the problem you are trying to solve. |
|
@szuecs the problem is well described in the PR, I even show you how to reproduce it. |
|
Me and I friend, we are trying to work on a test for this function. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Removed the config var from the NewSession function, the ideia is to discuss and evaluate a better option to implement this solution, I am not a golang developer but I am interested in contributing with my time if someone is available to guide me.
I don't understand the actual need of the Config being passed to the session.NewSessionWithOptions() function. There for I don't know the extend of the impact of this PR, I have tested and it works with Eks Pod Identities, I will be testing with IRSA and update here.
Fixes #4353
Checklist