Deploying Production-Grade Kafka Connect Clusters on Docker with Schema Registry (2026 Guide)

Why Custom ETL Scripts Fail for Event Streaming

Engineering teams frequently build custom Python or Node.js microservices to move data from transactional databases (PostgreSQL/MySQL) into real-time analytics data warehouses (ClickHouse/BigQuery/Snowflake). These bespoke scripts inevitably suffer from unhandled dead-letter queues, offset tracking bugs, and catastrophic schema drift. Kafka Connect combined with Confluent Schema Registry provides a distributed, fault-tolerant, and pluggable framework for real-time change data capture (CDC).

Documented by Saizul Amin for enterprise data architects and streaming infrastructure engineers.

Docker Compose Architecture for Local & Staging Clusters

version: '\''3.8'\''
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.6.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:7.6.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  schema-registry:
    image: confluentinc/cp-schema-registry:7.6.0
    depends_on:
      - kafka
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka:29092

  kafka-connect:
    image: confluentinc/cp-kafka-connect-base:7.6.0
    depends_on:
      - kafka
      - schema-registry
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:29092
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
      CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
      CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
      CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081
      CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_PLUGIN_PATH: /usr/share/java,/usr/share/confluent-hub-components

Configuring PostgreSQL Debezium CDC Connector via REST API

Submit connector configurations directly to Kafka Connect REST port 8083:

curl -X POST http://localhost:8083/connectors   -H "Content-Type: application/json"   -d '\''{
    "name": "pg-inventory-cdc",
    "config": {
      "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
      "tasks.max": "1",
      "database.hostname": "postgres",
      "database.port": "5432",
      "database.user": "postgres",
      "database.password": "secret",
      "database.dbname": "inventory",
      "database.server.name": "dbserver1",
      "plugin.name": "pgoutput"
    }
  }'\''