Kafka Connect Workshop

This repository contains the starter code for the How to play around with Your Apache Kafka Connect cluster? workshop.

This project shows how to install a Kafka Connect plugins and deploy connectors automatically using Docker runtime installation.

Getting Started

Use Docker Compose (docker-compose.yml) for setting up and running the whole Apache Kafka Connect infrastructure.

The following tools are available when you run the whole infrastructure:

• Kafka Connect
• Schema Registry
• Kowl UI
• Kafka Connect UI

Prerequisites

• Docker

Usage

• Run the docker compose stack

  docker compose up -d

• Check if all components are running

  docker compose ps    
                 
  # NAME                SERVICE             STATUS              PORTS
  # kafka               kafka               running             0.0.0.0:9092->9092/tcp, :::9092->9092/tcp, 0.0.0.0:9101->9101/tcp, :::9101->9101/tcp
  # kafka-connect       kafka-connect       running (healthy)   0.0.0.0:8083->8083/tcp, :::8083->8083/tcp, 9092/tcp
  # kafka-connect-ui    kafka-connect-ui    running             0.0.0.0:8000->8000/tcp, :::8000->8000/tcp
  # kowl                kowl                running             0.0.0.0:8080->8080/tcp, :::8080->8080/tcp
  # schema-registry     schema-registry     running             0.0.0.0:8081->8081/tcp, :::8081->8081/tcp
  # zookeeper           zookeeper           running             0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 2888/tcp, 3888/tcp  

• Use either Curl or Kafka Connect UI to check if autoconfigured FileStreamSourceConnector connector is running

  • Curl

    curl -L http://localhost:8083/connectors/fs-test-source-connector | jq
    
    # {
    #   "name": "fs-test-source-connector",
    #   "config": {
    #     "connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
    #     "file": "/tmp/test.txt",
    #     "tasks.max": "1",
    #     "name": "fs-test-source-connector",
    #     "topic": "test-data"
    #   },
    #   "tasks": [
    #     {
    #       "connector": "fs-test-source-connector",
    #       "task": 0
    #     }
    #   ],
    #   "type": "source"
    # }

  • Kafka Connect UI.

• Verify that custom Elastic Search Sink connector is available here - should be installed during docker image startup is

• Append messages to /tmp/test.txt file

  docker-compose exec kafka-connect bash -c "echo 'Hello World 1' >> /tmp/test.txt"
  docker-compose exec kafka-connect bash -c "echo 'Hello World 2' >> /tmp/test.txt"
  docker-compose exec kafka-connect bash -c "echo 'Hello World 3' >> /tmp/test.txt"

• Open your web browser and go to Kowl UI test-data topic

• Stop docker compose stack

  docker compose down -v

FAQ

How to install connectors differently than using Docker runtime ??

1. Custom Kafka Connect image

FROM confluentinc/cp-schema-registry:6.2.0
ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"

RUN confluent-hub install --no-prompt confluentinc/kafka-connect-elasticsearch:11.0.6

2. Docker volume mapping

    …
    environment:
      …
      CONNECT_PLUGIN_PATH: '/usr/share/java,/data/connectors/'
    volumes:
      - /path/to/custom/connectors/folder:/data/connectors/

Important Endpoints

Name	Endpoint
Kafka Connect	http://localhost:8083/
Kafka Connect UI	http://localhost:8000/
Kowl UI	http://localhost:8080/
Schema-registry	http://localhost:8081/

References

• Kafka Connect
• Confluent Kafka Connect
• FileSystem Source Connector
• Kafka Docker Images
• Kafka Connect UI
• cloudhut/kowl

License

Distributed under the MIT License. See LICENSE for more information.