I'm using kafka_2.11-1.1.0. This is my server.properties file:
broker.id=1
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.1.110:2181,192.168.1.64:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
On the second computer, broker.id=2. I got the ip numbers for the zookeeper.connect line by typing ipconfig into the command prompt and using the IPv4 Address under Ethernet adapter Local Area Connection on one computer and the IPv4 Address under Wireless LAN adapter Wi-Fi for the other.
I ran these commands on each computer (to anyone following along, run the first on both computers before running the second):
bin\windows\zookeeper-server-start.bat config\zookeeper.properties
bin\windows\kafka-server-start.bat config\server.properties
On the first computer, I made a topic and started a producer console:
bin\windows\kafka-topics.bat --create --zookeeper 192.168.1.110:2181 --replication-factor 2 --partitions 1 --topic test
bin\windows\kafka-console-producer.bat --broker-list 192.168.1.110:2181 --topic test
On the second one, I started a consumer console:
bin\windows\kafka-console-consumer.bat --bootstrap-server 192.168.1.64:2181 --topic test
When I tried sending a message, the consumer did not receive it. The zookeeper server console on each computer looped through the following messages, but with each IP value corresponding to its respective PC, and the port number increasing by one with each loop (about once a second):
INFO Accepted socket connection from /192.168.1.110:55371 (org.apache.zookeeper.server.NIOServerCnxnFactory)
WARN Exception causing close of session 0x0 due to java.io.EOFException (org.apache.zookeeper.server.NIOServerCnxn)
INFO Closed socket connection for client /192.168.1.110:55371 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
In the producer console, this error was received after a minute:
ERROR Error when sending message to topic test with key: null, value: 6 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
How do I fix this? Any help would be appreciated.
UPDATE - SOLVED - courtesy of Victor:
Change:
bin\windows\kafka-console-producer.bat --broker-list 192.168.1.110:2181 --topic test
and
bin\windows\kafka-console-consumer.bat --bootstrap-server 192.168.1.64:2181 --topic test
To:
bin\windows\kafka-console-producer.bat --broker-list 192.168.1.110:9092 --topic test
and
bin\windows\kafka-console-consumer.bat --bootstrap-server 192.168.1.64:9092 --topic test
UPDATE 2
For anyone who is following this to set up two computers with Kafka - I found that this method doesn't always work. The permanent solution I later found was to use the same IP for both computers. I used the IP of the computer with the ethernet connection, which happened to be the one with the producer.
I believe you have to pass to the producer a list of Kafka brokers and not a Zookeeper quorum:
So change this:
bin\windows\kafka-console-producer.bat --broker-list 192.168.1.110:2181
To something like this:
bin\windows\kafka-console-producer.bat --broker-list 192.168.1.110:9092
(I´m assuming you are running your Kafka server there)
I got a similar error, writing to Kafka with Spark Streaming:
Error connecting to Zookeeper with Spark Streaming Structured
この記事はインターネットから収集されたものであり、転載の際にはソースを示してください。
侵害の場合は、連絡してください[email protected]
コメントを追加