bug: impossible to trigger more than one lambda via Kafka topics

[rumfuddle]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I have defined a Kafka cluster in localstack using your documentation, I have created 2 topics and I have deployed 2 lambda's. Lambda 1 has as event source topic 1 and lambda 2 has as event source lambda 2.
If I send in a event via topic 1, lambda 1 is triggered but this doesn't work for topic2/lambda2, nothing happens (though the topic registers the event just fine).
If I first create the event mapping between lambda 2 and topic 2, then it works for lambda 2 but not for lambda 1. So the order of event mapping is crucial.

Expected Behavior

I expect that topic 1 triggers lambda 1 and topic 2 triggers lambda 2

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

docker-compose.yml:

version: "3.8"

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_product_master_consumer}"
    image: localstack/localstack-pro
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
    environment:
      - DEBUG=1
      - DOCKER_HOST=unix:///var/run/docker.sock
      - SERVICES=lambda,dynamodb,apigateway,cloudformation,iam,logs,kafka,secretsmanager,kms
      - DYNAMODB_ENDPOINT_URL=http://localstack:4566
      - LOCALSTACK_AUTH_TOKEN=***
    volumes:
      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

brokernodegroupinfo.json:

{
  "InstanceType": "kafka.m5.xlarge",
  "BrokerAZDistribution": "DEFAULT",
  "ClientSubnets": [
    "subnet-0123456789111abcd",
    "subnet-0123456789222abcd",
    "subnet-0123456789333abcd"
  ],
  "SecurityGroups": [
    "sg-0123456789111abcd"
  ]
}

aws kafka create-cluster \
    --endpoint-url http://localhost:4566 \
    --cluster-name "EventsCluster" \
    --broker-node-group-info file://brokernodegroupinfo.json \
    --kafka-version "2.8.0" \
    --number-of-broker-nodes 1 \
    --region eu-central-1

aws lambda create-event-source-mapping \
   --event-source arn:aws:kafka:eu-central-1:000000000000:cluster/EventsCluster/ec3fd5e5-8722-4e9a-b733-3c76190c056c-25 \
   --topics product-data-topic \
   --source-access-configuration '[{"Type": "VPC_SUBNET", "URI": "subnet:subnet-0123456789111abcd"}, {"Type": "VPC_SUBNET", "URI": "subnet:subnet-0123456789222abcd"}, {"Type": "VPC_SUBNET", "URI": "subnet:subnet-0123456789333abcd"}, {"Type": "VPC_SECURITY_GROUP", "URI": "security_group:sg-0123456789111abcd"}]' \
   --function-name arn:aws:lambda:eu-central-1:000000000000:function:uninety-product-data-consumer-local \
   --self-managed-event-source '{"Endpoints":{"KAFKA_BOOTSTRAP_SERVERS":["localhost:4511"]}}' \
   --starting-position LATEST \
   --endpoint-url http://localhost:4566 --region eu-central-1 --debug

aws lambda create-event-source-mapping \
   --event-source arn:aws:kafka:eu-central-1:000000000000:cluster/EventsCluster/ec3fd5e5-8722-4e9a-b733-3c76190c056c-25 \
   --topics product-size-data-topic \
   --source-access-configuration '[{"Type": "VPC_SUBNET", "URI": "subnet:subnet-0123456789111abcd"}, {"Type": "VPC_SUBNET", "URI": "subnet:subnet-0123456789222abcd"}, {"Type": "VPC_SUBNET", "URI": "subnet:subnet-0123456789333abcd"}, {"Type": "VPC_SECURITY_GROUP", "URI": "security_group:sg-0123456789111abcd"}]' \
   --function-name arn:aws:lambda:eu-central-1:000000000000:function:uninety-product-size-data-consumer-local \
   --self-managed-event-source '{"Endpoints":{"KAFKA_BOOTSTRAP_SERVERS":["localhost:4511"]}}' \
   --starting-position LATEST \
   --endpoint-url http://localhost:4566 --region eu-central-1 --debug

Environment

- OS: Mac M2
- LocalStack: Pro latest

Anything else?

We have noticed in the localstack container output that the message regarding. the polling thread only appears during the creation of the first mapping and not during the second:

localstack_product_master_consumer | 2024-04-30T07:45:30.130 DEBUG --- [ asgi_gw_6] l.s.l.e.kafka_event_source : Starting Kafka message polling thread for Lambda API
localstack_product_master_consumer | 2024-04-30T07:45:30.596 WARN --- [ Thread-120] kafka.coordinator.consumer : group_id is None: disabling auto-commit.
localstack_product_master_consumer | 2024-04-30T07:45:31.707 INFO --- [ asgi_gw_6] localstack.request.aws : AWS lambda.CreateEventSourceMapping => 202
localstack_product_master_consumer | 2024-04-30T07:46:02.988 INFO --- [ asgi_gw_2] localstack.request.aws : AWS lambda.CreateEventSourceMapping => 202

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.