Replies: 1 comment 1 reply
-
It seems to me this problem is most critical for those environments where cross-AZ communications are costly. In any other IaaS scenario, such as OpenStack, the Load Balancer has visibility over all the AZs, and it's free of charge. We're going to have a single Service for the whole Control Plane replicas since we're not aware of the underlying cluster topology, and also considering that deployed Kubelet wouldn't be aware of the Topology-aware traffic policies. Before implementing this, I would suggest having feedback directly from users who are expecting this feature in order to collect their feedback, and their needs, in order to design a proper solution. With the provided details you collected, @bsctl, I would suggest something as this: Deploying the TenantControlPlane as NodePort serviceKamaji would still require a valid address, and that's ok: the Cluster Administrator could create an external Load Balancer, get the VIP, and then use it as the Control Plane address.
Advertising Pods' nodes' addresses as Control Plane addressesTo keep it simple, let's say we have 3 replicas Tenant Control Plane:
These are the nodes IPs:
Each Pod replica will start with the
I hadn't yet the time to test if we can Downward APIs to inject the Node's IP as an environment variable for the
|
Beta Was this translation helpful? Give feedback.
-
Background
A Kubernetes setup made with
kubeadm
defines theControlPlaneEndpoint
:this represents how the control plane is reached by
kubelet
instances. It can be the floating IP (VIP) used by the control plane nodes or the IP of the LoadBalancer put in front of the control plane nodes.The
kube-apiserver
has an option:--advertise-address
defined as the IP address on which to advertise thekube-apiserver
to members of the cluster. This address must be reachable by the rest of the cluster.Each
kube-apiserver
has its own--advertise-address
, usually the same IP address of the machine hosting thekube-apiserver
:These addresses are then used as
endpoints
of thekubernetes
service in thedefault
namespace:The endpoints tell the pods how to reach the instances of
kube-apiserver
once thekubernetes
service is resolved by the coreDNS and, usually, they are different from theControlPlaneEndpoint
.How it works in Kamaji
In Kamaji we define the
ControlPlaneEndpoint
as how thekubelet
and other clients can reach the tenant control plane:And it is actually where the tenant control plane is exposed through the
tcp
service:Since in Kamaji, the tenant control plane pods and the workload pods running on the tenant worker nodes are not directly reachable each other, we set the
--advertise-address=tcp.spec.networkProfile.address
in thekube-apiserver
containers to make the tenant control plane pods reachable. So ùthe kubernetes service in default namespace is resolved by coreDNS into the same IP address used to reach the tenant control plane:What's the problem with current status
The model above is working fine in Kamaji when deployed on bare metal and or virtual. Same when deploying on AKS and GKE managed services. The problem arises when deploying Kamaji on EKS since AWS creates network LoadBalancers without an IP but only with a domain name:
In such case, the creation of the tenant control plane in Kamaji fails since there is no
loadBalancerIP
address assigned to the service. AWS resolves automatically the name of the service to public IP addresses but these addresses are not in the control of the Kubernetes cloud provider. Also these IPs can change over the time, so they are not reliable to use directly as addresses.How to solve this?
Beta Was this translation helpful? Give feedback.
All reactions