How to create connectors for Kafka-connect on Kubernetes?

Amit Yadav

I am deploying Kafka-connect on Google Kubernetes Engine (GKE) using cp-kafka-connect Helm chart in distributed mode.

A working Kafka cluster with broker and zookeeper is already running on the same GKE cluster. I understand I can create connectors by sending post requests to http://localhost:8083/connectors endpoint once it is available. However, Kafka-connect container goes into RUNNING state and then starts loading the jar files and till all the jar files are loaded the endpoint mentioned above is unreachable.

I am looking for a way to automate the step of manually exec the pod, check if the endpoint is ready and then send the post requests. I have a shell script that has a bunch of curl -X POST requests to this endpoint to create the connectors and also have config files for these connectors which work fine with standalone mode (using Confluent platform show in this confluent blog).

Now there are only two ways to create the connector:

  1. Somehow identify when the container is actually ready (when the endpoint has started listening) and then run the shell script containing the curl requests
  2. OR use the configuration files as we do in standalone mode (Example: $ <path/to/CLI>/confluent local load connector_name -- -d /connector-config.json)

Which of the above approach is better?

Is the second approach (config files) even doable with distributed mode?

  • If YES: How to do that?
  • If NO: How to successfully do what is explained in the first approach?

EDIT: With reference to his github issue(thanks to @cricket_007's answer below) I added the following as the container command and connectors got created after the endpoint gets ready:

...
command:
  - /bin/bash
  - -c
  - |
    /etc/confluent/docker/run &
    echo "Waiting for Kafka Connect to start listening on kafka-connect  "
    while : ; do
      curl_status=`curl -s -o /dev/null -w %{http_code} http://localhost:8083/connectors`
      echo -e `date` " Kafka Connect listener HTTP state: " $curl_status " (waiting for 200)"
      if [ $curl_status -eq 200 ] ; then
        break
      fi
      sleep 5
    done
    echo -e "\n--\n+> Creating Kafka Connector(s)"
    /tmp/scripts/create-connectors.sh
    sleep infinity
...

/tmp/scripts/create-connectors.sh is a script mounted externally containing a bunch of POST requests using CURL to the Kafka-connect API.

OneCricketeer

confluent local doesn't interact with a remote Connect cluster, such as one in Kubernetes.

Please refer to the Kafka Connect REST API

You'd connect to it like any other RESTful api running in the cluster (via a Nodeport, or an Ingress/API Gateway for example)

the endpoint mentioned above is unreachable.

Localhost is the physical machine you're typing the commands into, not the remote GKE cluster

Somehow identify when the container is actually ready

Kubernetes health checks are responsible for that

kubectl get services

there are only two ways to create the connector

That's not true. You could additional run Landoop's Kafka Connect UI or Confluent Control Center in your cluster to point and click.

But if you have local config files, you could also write code to interact with the API

Or try and see if you can make a PR for this issue

https://github.com/confluentinc/cp-docker-images/issues/467

この記事はインターネットから収集されたものであり、転載の際にはソースを示してください。

侵害の場合は、連絡してください[email protected]

編集
0

コメントを追加

0

関連記事

分類Dev

Create server and connect to it

分類Dev

How to create topics in apache kafka?

分類Dev

How to create a user provided redis service which spring auto configuration cloud connectors picks?

分類Dev

How to install mod_jk (Apache Tomcat Connectors) on Windows Server?

分類Dev

How to use kubeadm to create kubernetes cluster?

分類Dev

Kafka Connect Logstash

分類Dev

Diagrammer: how to add labels to mermaid connectors

分類Dev

How to create a mockable class to connect to mongoDB?

分類Dev

How to deploy Kafka Stream applications on Kubernetes?

分類Dev

Stripe API; How to create test connect account successfully?

分類Dev

how to connect a kubernetes pod to the outside world without a forwarding rule (google container engine)

分類Dev

Kafka Connect AWS Lambda Sink

分類Dev

Kafka Connect with CockroachDB

分類Dev

How to create a GCP Kubernetes Engine cluster spanning two regions?

分類Dev

Kafka Connect and Streams

分類Dev

Kafka + Kubernetes + Helm + `/ usr / bin / kafka-avro-console-consumer`?

分類Dev

How to create Kafka user and consumer group for ACLs in a running cluster?

分類Dev

How to create a user in a Kubernetes cluster?

分類Dev

Apache Kafka Connect With Springboot

分類Dev

Kafka Connect vs Streams for Sinks

分類Dev

How to get the table-name and database-name in the CDC event received from debezium kafka connect

分類Dev

Is it possible to specify a kafka topic in a Kafka Connect config?

分類Dev

Kafka Connect with Amazon MSK

分類Dev

KubernetesでKafka-connectのコネクタを作成するにはどうすればよいですか?

分類Dev

What if debezium source connector of kafka connect fails to produce to kafka

分類Dev

Kafka Connect SMT to add Kafka header fields

分類Dev

How to connect to a private IP from Kubernetes Pod

分類Dev

How to programmatically create topics using kafka-python?

分類Dev

Visio Static Connectors - How do you place them?

Related 関連記事

  1. 1

    Create server and connect to it

  2. 2

    How to create topics in apache kafka?

  3. 3

    How to create a user provided redis service which spring auto configuration cloud connectors picks?

  4. 4

    How to install mod_jk (Apache Tomcat Connectors) on Windows Server?

  5. 5

    How to use kubeadm to create kubernetes cluster?

  6. 6

    Kafka Connect Logstash

  7. 7

    Diagrammer: how to add labels to mermaid connectors

  8. 8

    How to create a mockable class to connect to mongoDB?

  9. 9

    How to deploy Kafka Stream applications on Kubernetes?

  10. 10

    Stripe API; How to create test connect account successfully?

  11. 11

    how to connect a kubernetes pod to the outside world without a forwarding rule (google container engine)

  12. 12

    Kafka Connect AWS Lambda Sink

  13. 13

    Kafka Connect with CockroachDB

  14. 14

    How to create a GCP Kubernetes Engine cluster spanning two regions?

  15. 15

    Kafka Connect and Streams

  16. 16

    Kafka + Kubernetes + Helm + `/ usr / bin / kafka-avro-console-consumer`?

  17. 17

    How to create Kafka user and consumer group for ACLs in a running cluster?

  18. 18

    How to create a user in a Kubernetes cluster?

  19. 19

    Apache Kafka Connect With Springboot

  20. 20

    Kafka Connect vs Streams for Sinks

  21. 21

    How to get the table-name and database-name in the CDC event received from debezium kafka connect

  22. 22

    Is it possible to specify a kafka topic in a Kafka Connect config?

  23. 23

    Kafka Connect with Amazon MSK

  24. 24

    KubernetesでKafka-connectのコネクタを作成するにはどうすればよいですか?

  25. 25

    What if debezium source connector of kafka connect fails to produce to kafka

  26. 26

    Kafka Connect SMT to add Kafka header fields

  27. 27

    How to connect to a private IP from Kubernetes Pod

  28. 28

    How to programmatically create topics using kafka-python?

  29. 29

    Visio Static Connectors - How do you place them?

ホットタグ

アーカイブ