Kafka producer and consumer on separate computers aren't communicating

aaa

I'm using kafka_2.11-1.1.0. This is my server.properties file:

broker.id=1
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.1.110:2181,192.168.1.64:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

On the second computer, broker.id=2. I got the ip numbers for the zookeeper.connect line by typing ipconfig into the command prompt and using the IPv4 Address under Ethernet adapter Local Area Connection on one computer and the IPv4 Address under Wireless LAN adapter Wi-Fi for the other.

I ran these commands on each computer (to anyone following along, run the first on both computers before running the second):

bin\windows\zookeeper-server-start.bat config\zookeeper.properties
bin\windows\kafka-server-start.bat config\server.properties

On the first computer, I made a topic and started a producer console:

bin\windows\kafka-topics.bat --create --zookeeper 192.168.1.110:2181 --replication-factor 2 --partitions 1 --topic test
bin\windows\kafka-console-producer.bat --broker-list 192.168.1.110:2181 --topic test

On the second one, I started a consumer console:

bin\windows\kafka-console-consumer.bat --bootstrap-server 192.168.1.64:2181 --topic test

When I tried sending a message, the consumer did not receive it. The zookeeper server console on each computer looped through the following messages, but with each IP value corresponding to its respective PC, and the port number increasing by one with each loop (about once a second):

INFO Accepted socket connection from /192.168.1.110:55371 (org.apache.zookeeper.server.NIOServerCnxnFactory)
WARN Exception causing close of session 0x0 due to java.io.EOFException (org.apache.zookeeper.server.NIOServerCnxn)
INFO Closed socket connection for client /192.168.1.110:55371 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)

In the producer console, this error was received after a minute:

ERROR Error when sending message to topic test with key: null, value: 6 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

How do I fix this? Any help would be appreciated.

UPDATE - SOLVED - courtesy of Victor:

Change:

bin\windows\kafka-console-producer.bat --broker-list 192.168.1.110:2181 --topic test

and

bin\windows\kafka-console-consumer.bat --bootstrap-server 192.168.1.64:2181 --topic test

To:

bin\windows\kafka-console-producer.bat --broker-list 192.168.1.110:9092 --topic test

and

bin\windows\kafka-console-consumer.bat --bootstrap-server 192.168.1.64:9092 --topic test

UPDATE 2

For anyone who is following this to set up two computers with Kafka - I found that this method doesn't always work. The permanent solution I later found was to use the same IP for both computers. I used the IP of the computer with the ethernet connection, which happened to be the one with the producer.

Victor

I believe you have to pass to the producer a list of Kafka brokers and not a Zookeeper quorum:

So change this:

bin\windows\kafka-console-producer.bat --broker-list 192.168.1.110:2181

To something like this:

bin\windows\kafka-console-producer.bat --broker-list 192.168.1.110:9092

(I´m assuming you are running your Kafka server there)

I got a similar error, writing to Kafka with Spark Streaming:

Error connecting to Zookeeper with Spark Streaming Structured

この記事はインターネットから収集されたものであり、転載の際にはソースを示してください。

侵害の場合は、連絡してください[email protected]

編集
0

コメントを追加

0

関連記事

分類Dev

Go channel didn't work for producer/consumer sample

分類Dev

Producer consumer code with wait/notify doesn't work on second produce

分類Dev

Java Producer Consumer IllegalMonitorStateException

分類Dev

Move folder structure from one computer to another computer when these computers aren't networked?

分類Dev

Powershell: Pipeline Producer and Consumer communication

分類Dev

One producer Multiple Consumer performance

分類Dev

Spring-Kafka consumer doesn't receive messages

分類Dev

Improving performance of Kafka Producer

分類Dev

Kafka Authentication Producer Unable to Connect Producer

分類Dev

Kafka Consumer with JAVA

分類Dev

Spring boot Kafka consumer

分類Dev

go routine producer-consumer pattern panics

分類Dev

Difference between Producer/Consumer pattern and Observer Pattern

分類Dev

Producer consumer using only 1 additional semaphore

分類Dev

Producer Consumer synchronization with only 1 semaphore

分類Dev

Producer & consumer threads using condition variables

分類Dev

Producer-consumer algorithm to use full buffer

分類Dev

Spring Cloud Stream(Hoxton)Kafka Producer / ConsumerがEmbeddedKafkaとの統合テストで機能しない

分類Dev

Kafka producer sending invalid characters

分類Dev

kafka: keep track of producer offset

分類Dev

Kafka Producer design - multiple topics

分類Dev

Kafka consumer manual commit offset

分類Dev

Kafka Stream: Consumer commit frequency

分類Dev

Partition specific flink kafka consumer

分類Dev

Kafka elixir consumer keeps crashing

分類Dev

kafka consumer code is not running completely

分類Dev

Kafka consumer hangs on poll when kafka is down

分類Dev

Invalid Timestamp when writing a Kafka producer with sarama

分類Dev

How to enable Kafka Producer Metrics in Spark?

Related 関連記事

  1. 1

    Go channel didn't work for producer/consumer sample

  2. 2

    Producer consumer code with wait/notify doesn't work on second produce

  3. 3

    Java Producer Consumer IllegalMonitorStateException

  4. 4

    Move folder structure from one computer to another computer when these computers aren't networked?

  5. 5

    Powershell: Pipeline Producer and Consumer communication

  6. 6

    One producer Multiple Consumer performance

  7. 7

    Spring-Kafka consumer doesn't receive messages

  8. 8

    Improving performance of Kafka Producer

  9. 9

    Kafka Authentication Producer Unable to Connect Producer

  10. 10

    Kafka Consumer with JAVA

  11. 11

    Spring boot Kafka consumer

  12. 12

    go routine producer-consumer pattern panics

  13. 13

    Difference between Producer/Consumer pattern and Observer Pattern

  14. 14

    Producer consumer using only 1 additional semaphore

  15. 15

    Producer Consumer synchronization with only 1 semaphore

  16. 16

    Producer & consumer threads using condition variables

  17. 17

    Producer-consumer algorithm to use full buffer

  18. 18

    Spring Cloud Stream(Hoxton)Kafka Producer / ConsumerがEmbeddedKafkaとの統合テストで機能しない

  19. 19

    Kafka producer sending invalid characters

  20. 20

    kafka: keep track of producer offset

  21. 21

    Kafka Producer design - multiple topics

  22. 22

    Kafka consumer manual commit offset

  23. 23

    Kafka Stream: Consumer commit frequency

  24. 24

    Partition specific flink kafka consumer

  25. 25

    Kafka elixir consumer keeps crashing

  26. 26

    kafka consumer code is not running completely

  27. 27

    Kafka consumer hangs on poll when kafka is down

  28. 28

    Invalid Timestamp when writing a Kafka producer with sarama

  29. 29

    How to enable Kafka Producer Metrics in Spark?

ホットタグ

アーカイブ