How to use redpanda-operator with external IPs and nodeports for AWS EKS Cluster with Public and Private subnets

[rfinner]

I am running vectorized/redpanda-operator:latest (installed 2 days ago) in an AWS EKS Cluster that has public and private subnets. It appears that the external flag for my test cluster "chat-with-me" is creating a nodeport in the private subnet and there is no external public IP created. I was wondering if this is correct and what I could or should do in order to access from rpk remotely. Here is my test

cluster CRD: apiVersion: redpanda.vectorized.io/v1alpha1
kind: Cluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"redpanda.vectorized.io/v1alpha1","kind":"Cluster","metadata":{"annotations":{},"name":"chat-cluster","namespace":"chat-with-me"},"spec":{"configuration":{"adminApi":[{"port":9644}],"developerMode":true,"kafkaApi":[{"port":9092,"tls":{"enabled":true}},{"external":{"enabled":true}}],"rpcServer":{"port":33145}},"image":"vectorized/redpanda","replicas":3,"resources":{"limits":{"cpu":1,"memory":"2Gi"},"requests":{"cpu":1,"memory":"2Gi"}},"version":"latest"}}
  creationTimestamp: "2021-05-03T01:43:01Z"
  generation: 1
  managedFields:
  - apiVersion: redpanda.vectorized.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:spec:
        .: {}
        f:configuration:
          .: {}
          f:adminApi: {}
          f:developerMode: {}
          f:kafkaApi: {}
          f:rpcServer:
            .: {}
            f:port: {}
        f:image: {}
        f:replicas: {}
        f:resources:
          .: {}
          f:limits:
            .: {}
            f:cpu: {}
            f:memory: {}
          f:requests:
            .: {}
            f:cpu: {}
            f:memory: {}
        f:version: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-05-03T01:43:01Z"
  - apiVersion: redpanda.vectorized.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        .: {}
        f:nodes:
          .: {}
          f:external: {}
          f:internal: {}
        f:replicas: {}
        f:upgrading: {}
    manager: manager
    operation: Update
    time: "2021-05-03T01:43:07Z"
  name: chat-cluster
  namespace: chat-with-me
  resourceVersion: "104435974"
  selfLink: /apis/redpanda.vectorized.io/v1alpha1/namespaces/chat-with-me/clusters/chat-cluster
  uid: 22b3d719-75ae-4202-a34d-73038b8b7799
spec:
  cloudStorage:
    enabled: false
    secretKeyRef: {}
  configuration:
    adminApi:
    - external: {}
      port: 9644
      tls: {}
    developerMode: true
    kafkaApi:
    - external: {}
      port: 9092
      tls:
        enabled: true
    - external:
        enabled: true
      tls: {}
    rpcServer:
      port: 33145
  image: vectorized/redpanda
  replicas: 3
  resources:
    limits:
      cpu: "1"
      memory: 2Gi
    requests:
      cpu: "1"
      memory: 2Gi
  storage:
    capacity: "0"
  version: latest
status:
  nodes:
    external:
    - 34.213.99.26:32087
    - 54.191.107.49:32087
    - 35.81.78.211:32087
    internal:
    - chat-cluster-0.chat-cluster.chat-with-me.svc.cluster.local.
    - chat-cluster-2.chat-cluster.chat-with-me.svc.cluster.local.
    - chat-cluster-1.chat-cluster.chat-with-me.svc.cluster.local.
  replicas: 3
  upgrading: false

---

I think the issue might be in this part of the Service yaml generated: It should have "spec:
externalIPs:

1.2.3.4
4.5.6.7
externalTrafficPolicy: Cluster
ports:
nodePort: 30001 #will give you a random 3XXXX port"
instead of externalTrafficPolicy: Local

[emaxerrno]

cc: @dimitriscruz @alenkacz and @RafalKorepta

[emaxerrno]

note, i think for testing, this will work, but for production we recommend mTLS for all outside port, superuser access, DNS, ACLs, etc. (we basically do this for customers on our cloud, so ping me if you interested in that [email protected]) but we can help you get connected to the outside world.

[rfinner]

Yes, I would like to learn how to set up using best practices like mTLS, DNS, ACLs, or anything else. That is my ultimate goal. At the moment I am just playing with the redpanda-operator and these other settings for only learning purposes as I am a newbie to Redpanda and I am rather new to making production quality deployments in general. The next thing I will try will be to use the helm chart instead of the operator and observe how it does things and what option settings are available, etc. I do soon hope to deploy to production and so any tips would be great! I am meeting with Darin Newman later this week.

[RafalKorepta]

Hi @rfinner,

We are using node port only for port generation. In the stateful set you can find hostPort that was generated by the Service

apiVersion: apps/v1
kind: StatefulSet
....
        image: vectorized/redpanda:latest
        imagePullPolicy: IfNotPresent
        name: redpanda
        ports:
        - containerPort: 33145
          name: rpc
          protocol: TCP
        - containerPort: 9644
          name: admin
          protocol: TCP
        - containerPort: 9092
          name: kafka
          protocol: TCP
        - containerPort: 9093
          hostPort: 32087   <--------------------This is your external port that should be open on all nodes
          name: kafka-external
          protocol: TCP
 ....

If the security group for worker nodes has policy that opens port 32087 to you or even to the world, then it should work.

Can you post the output of the rpk cluster info --brokers 34.213.99.26:32087,54.191.107.49:32087,35.81.78.211:32087?
What other tools/programs you were using to test the connectivity?

[rfinner]

I added this to my EKS Cluster Security Group. Also note that I reinstalled and so the NodePort is now 31200 and I changed my IP address so that I would not be publishing it here:
Custom TCP | TCP | 31200 | 72.181.111.45/32 | External Access to RedPanda tbn-cluster-1

I still get the same error message as before:

rpk cluster info --brokers 34.213.99.26:31200,54.191.107.49:31200,35.81.78.211:31200
Couldn't initialize API admin
Error: couldn't connect to redpanda at 34.213.99.26:31200, 54.191.107.49:31200, 35.81.78.211:31200. Try using --brokers to specify other brokers to connect to.

Here is the command again with the -v option:
rpk cluster info --brokers 34.213.99.26:31200,54.191.107.49:31200,35.81.78.211:31200 -v
Looking for the redpanda config file
Looking for redpanda config file in '/home/rob'
Looking for redpanda config file in '/etc/redpanda'
Skipping API TLS auth config. Set The cert file, key file and truststore file to enable it
Using --brokers: 34.213.99.26:31200, 54.191.107.49:31200, 35.81.78.211:31200
Initializing new client
client/metadata fetching metadata for all topics from broker 35.81.78.211:31200

Failed to connect to broker 35.81.78.211:31200: dial tcp 35.81.78.211:31200: i/o timeout

client/metadata got error from broker -1 while fetching metadata: dial tcp 35.81.78.211:31200: i/o timeout

client/metadata not fetching metadata from broker 54.191.107.49:31200 as we would go past the metadata timeout

client/metadata skipping last retries as we would go past the metadata timeout
Closing Client
kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
Couldn't initialize API admin
Error: couldn't connect to redpanda at 34.213.99.26:31200, 54.191.107.49:31200, 35.81.78.211:31200. Try using --brokers to specify other brokers to connect to.

[rfinner]

It is notable that this nodeport is also being used by the Service, and that the service spec does not list any external IPs. Should I try installing the Cluster without the External: True ? Also does it matter that I have TLS turned on? (In production, I would need TLS for talking across the Internet.)

[rfinner]

kubectl -n redpanda get clusters.redpanda.vectorized.io tbn-cluster -o=jsonpath='{.status.nodes.external}'
[34.213.99.26:31200 35.81.78.211:31200 54.191.107.49:31200]

[rfinner]

I also had issues using kubectl port-forward with rpk:

rpk cluster info --brokers 127.0.0.1:9092 -v
Looking for the redpanda config file
Looking for redpanda config file in '/home/rob'
Looking for redpanda config file in '/etc/redpanda'
Skipping API TLS auth config. Set The cert file, key file and truststore file to enable it
Using --brokers: 127.0.0.1:9092
Initializing new client
client/metadata fetching metadata for all topics from broker 127.0.0.1:9092

Connected to broker at 127.0.0.1:9092 (unregistered)

client/brokers registered new broker #0 at tbn-cluster-two-0.tbn-cluster-two.redpanda.svc.cluster.local.:9092
client/brokers registered new broker #2 at tbn-cluster-two-2.tbn-cluster-two.redpanda.svc.cluster.local.:9092
client/brokers registered new broker #1 at tbn-cluster-two-1.tbn-cluster-two.redpanda.svc.cluster.local.:9092
Successfully initialized new client
Failed to connect to broker tbn-cluster-two-0.tbn-cluster-two.redpanda.svc.cluster.local.:9092: dial tcp: lookup tbn-cluster-two-0.tbn-cluster-two.redpanda.svc.cluster.local. on 127.0.0.53:53: no such host

Failed to connect to broker tbn-cluster-two-0.tbn-cluster-two.redpanda.svc.cluster.local.:9092: dial tcp: lookup tbn-cluster-two-0.tbn-cluster-two.redpanda.svc.cluster.local. on 127.0.0.53:53: no such host

Closing Client
Error: Error fetching the Redpanda topic details: dial tcp: lookup tbn-cluster-two-0.tbn-cluster-two.redpanda.svc.cluster.local. on 127.0.0.53:53: no such host

[alenkacz]

@rfinner that's really strange - could you please check (just to cross out the obvious stuff) that the pods are running?

[rfinner]

Sure, I have two clusters, all six pods are healthy. I am able to create and list topics and check cluster status with kubectl exec -it rpk, etc.

[alenkacz]

looking at this once more, this is actually expected. When you port-forward, it opens the port on your local machine, then rpk connects to the broker and discovers internal IPs of other brokers but those are not reachable from your machine so it fails. To make that work you would have to do some tunneling or something like the recommendations here https://medium.com/@valercara/connecting-to-a-kafka-cluster-running-in-kubernetes-7601ae3a87d6

[dimitriscruz]

@rfinner Your spec has TLS enabled for the internal port and disabled for the external port:

    kafkaApi:
    - external: {}
      port: 9092
      tls:
        enabled: true
    - external:
        enabled: true
      tls: {}

The current combination in your spec would not work as expected with our existing implementation. If you looked at the generated redpanda.yaml within the pod, the TLS would be applied to the external listener. Hence, you'd need to have the ca.crt provided to rpk when using the external names. (Or you could disable TLS completely for testing purposes to eliminate this possibility). I don't see a reason to prevent the case in the spec above, so I'll see the logic is changed.

Having said that, I see from above that there might be further issues, but I wanted to note this down.

[alenkacz]

This is actually very good catch, also Dimitrios fixed it in here #1349

@rfinner could you try disabling tls on kafka api and seeing if the external connectivity works? I don't see any other reason why it should not...