検索
検索

Kafka Consumer configuration - How does auto.offset.reset controls the message consumption

debugcn 投稿 Dev
6
Raj

I'm trying to understand, how does the ConsumerConfig.auto.offset.reset = latest would affect the message consumption.

For example I've a consumer, sending 100 messages initially at time t1 and then my consumer is up and running at t1+30 sec, then would my consumer consume the messages published after t1+30 sec or will it consume messages published t1 onwards?

Gary Russell

It depends.

auto.offset.reset only applies when there is no stored offset for the consumer group.

It applies to the following conditions:

  • the first time a consumer group consumes
  • if a consumer doesn't commit any offsets, the next time it is started
  • if a consumer group has been expired (7 days by default with modern brokers)

If a consumer commits an offset; it will start at the last committed offset the next time it is started.

この記事はインターネットから収集されたものであり、転載の際にはソースを示してください。

侵害の場合は、連絡してください[email protected]

編集
0

コメントを追加

0
サインイン

関連記事

分類Dev

Kafka - How to commit offset after every message using High-Level consumer?

分類Dev

Who keeps track of the last read message offset of the consumer in Apache Kafka?

分類Dev

Kafka consumer does not start from latest message

分類Dev

Kafka Consumer not able to read all message after offset commit (error=OFFSET_OUT_OF_RANGE

分類Dev

Kafka consumer manual commit offset

分類Dev

spark-streaming-kafka-0-10 auto.offset.reset is always set to none

分類Dev

Incorrect Kafka offset across consumer groups

分類Dev

How to set offset committed by the consumer group using Spark's Direct Stream for Kafka?

分類Dev

spark-streaming-kafka-0-10auto.offset.resetは常にnoneに設定されます

分類Dev

auto-offset-reset = latestはspring-kafkaでは機能しません

分類Dev

Kafka - Why fresh groupId doesn't return all messages in topic when setting AUTO_OFFSET_RESET_CONFIG as "latest"

分類Dev

Spark 1.6 Streaming consumer reading in kafka offset stuck at createDirectStream

分類Dev

How does Kafka handle a consumer which is running slower than other consumers?

分類Dev

Spring kafka Consume multiple Message types in one consumer

分類Dev

Spring kafka Consume multiple Message types in one consumer

分類Dev

Apache Kafka: Can we restrict message to be read by only 1 consumer?

分類Dev

Kafka what will happen with message if a consumer group member goes down?

分類Dev

How Can i pace a producer actor so that it does not pump to many message to a consumer actor

分類Dev

How do I implement Kafka Consumer in Scala

分類Dev

Send message to different Kafka topics based on configuration

分類Dev

RabbitMQ - How to save message for new consumer

分類Dev

How to get the processing kafka topic name dynamically in Flink Kafka Consumer?

分類Dev

Is consumer offset commited even when failing to post to output topic in Kafka Streams?

分類Dev

Kafkaコンシューマー構成-auto.offset.resetはメッセージ消費をどのように制御しますか

分類Dev

Why does kafka-console-consumer timeout on a small number of messages?

分類Dev

Message pre-processing (topic - topic) - Kafka Connect API vs. Streams vs Kafka Consumer?

分類Dev

How to keep the date in DateTimeOffset and reset the offset to zero (UTC)?

分類Dev

How to detect if kafka broker is not available from consumer in java?

分類Dev

How to have multiple kafka consumer groups in application properties

Related 関連記事

  1. 1

    Kafka - How to commit offset after every message using High-Level consumer?

  2. 2

    Who keeps track of the last read message offset of the consumer in Apache Kafka?

  3. 3

    Kafka consumer does not start from latest message

  4. 4

    Kafka Consumer not able to read all message after offset commit (error=OFFSET_OUT_OF_RANGE

  5. 5

    Kafka consumer manual commit offset

  6. 6

    spark-streaming-kafka-0-10 auto.offset.reset is always set to none

  7. 7

    Incorrect Kafka offset across consumer groups

  8. 8

    How to set offset committed by the consumer group using Spark's Direct Stream for Kafka?

  9. 9

    spark-streaming-kafka-0-10auto.offset.resetは常にnoneに設定されます

  10. 10

    auto-offset-reset = latestはspring-kafkaでは機能しません

  11. 11

    Kafka - Why fresh groupId doesn't return all messages in topic when setting AUTO_OFFSET_RESET_CONFIG as "latest"

  12. 12

    Spark 1.6 Streaming consumer reading in kafka offset stuck at createDirectStream

  13. 13

    How does Kafka handle a consumer which is running slower than other consumers?

  14. 14

    Spring kafka Consume multiple Message types in one consumer

  15. 15

    Spring kafka Consume multiple Message types in one consumer

  16. 16

    Apache Kafka: Can we restrict message to be read by only 1 consumer?

  17. 17

    Kafka what will happen with message if a consumer group member goes down?

  18. 18

    How Can i pace a producer actor so that it does not pump to many message to a consumer actor

  19. 19

    How do I implement Kafka Consumer in Scala

  20. 20

    Send message to different Kafka topics based on configuration

  21. 21

    RabbitMQ - How to save message for new consumer

  22. 22

    How to get the processing kafka topic name dynamically in Flink Kafka Consumer?

  23. 23

    Is consumer offset commited even when failing to post to output topic in Kafka Streams?

  24. 24

    Kafkaコンシューマー構成-auto.offset.resetはメッセージ消費をどのように制御しますか

  25. 25

    Why does kafka-console-consumer timeout on a small number of messages?

  26. 26

    Message pre-processing (topic - topic) - Kafka Connect API vs. Streams vs Kafka Consumer?

  27. 27

    How to keep the date in DateTimeOffset and reset the offset to zero (UTC)?

  28. 28

    How to detect if kafka broker is not available from consumer in java?

  29. 29

    How to have multiple kafka consumer groups in application properties

ホットタグ

アーカイブ