Kafka Stream: Consumer commit frequency

Raman

With atleast-once guarantee, I understand that there is a possibility of duplicates in case of failures. However,
1) How frequent does the Kafka Stream library performs commits?
2) Does the users ever need consider committing in addition to the above?
3) Is there a best practice on how frequent the commit should be performed?

Matthias J. Sax

Kafka Streams commits in regular intervals that can be configured via parameter commit.interval.ms (default is 30 seconds; if exactly-once processing is enabled, default is 100ms).

Usually, it's not necessary for users to commit manually. Note thought, that users don't have complete control over committing, but can only request commits: cf. How to commit manually with Kafka Stream?

Commits are synchronizations point and if you commit too frequently (for an extreme example after every processed record) your throughput can drop significantly. It also highly application dependent, because the commit frequency determines how many potential duplicates the application processes (this also depends on the input data rate). Thus, you need to consider how many duplicates in case of failure your are willing to tolerate. It also depends how long it will take for the application to reprocess the data: during this time the application might not be fully available. Overall, it's hard to give a recommendation and you need to consider the described trade-offs for each application individually.

この記事はインターネットから収集されたものであり、転載の際にはソースを示してください。

侵害の場合は、連絡してください[email protected]

編集
0

コメントを追加

0

関連記事

分類Dev

Kafka consumer manual commit offset

分類Dev

Kafka - How to commit offset after every message using High-Level consumer?

分類Dev

Kafka Consumer not able to read all message after offset commit (error=OFFSET_OUT_OF_RANGE

分類Dev

How to set offset committed by the consumer group using Spark's Direct Stream for Kafka?

分類Dev

Kafka Consumer with JAVA

分類Dev

Spring boot Kafka consumer

分類Dev

Reactive-Kafka Stream Consumer:デッドレターが発生しました

分類Dev

Partition specific flink kafka consumer

分類Dev

Kafka elixir consumer keeps crashing

分類Dev

kafka consumer code is not running completely

分類Dev

Kafka consumer hangs on poll when kafka is down

分類Dev

Kafka Stream Ksql Json

分類Dev

Kafka Stream BoundedMemoryRocksDBConfig

分類Dev

Kafka consumer - what's the relation of consumer processes and threads with topic partitions

分類Dev

Kafka consumer group script to see all consumer group not working

分類Dev

Spring Cloud Stream(Hoxton)Kafka Producer / ConsumerがEmbeddedKafkaとの統合テストで機能しない

分類Dev

Kafka Avro Consumer with Decoderの問題

分類Dev

Better way of error handling in Kafka Consumer

分類Dev

Kafka Consumer get assigned partitions for a specific topic

分類Dev

How do I implement Kafka Consumer in Scala

分類Dev

Kafka consumer does not start from latest message

分類Dev

Incorrect Kafka offset across consumer groups

分類Dev

Connecting Apache Consumer to a single node on a Kafka Cluster

分類Dev

Spring Boot Kafka consumer lags and reads wrong

分類Dev

Issue in implementing infinite retry in kafka consumer

分類Dev

Kafka - Assign messages to specific Consumer Groups

分類Dev

Apache Kafka 2.0 get the consumer lag

分類Dev

How to count the frequency of each ID in kafka Topic

分類Dev

How to get the processing kafka topic name dynamically in Flink Kafka Consumer?

Related 関連記事

  1. 1

    Kafka consumer manual commit offset

  2. 2

    Kafka - How to commit offset after every message using High-Level consumer?

  3. 3

    Kafka Consumer not able to read all message after offset commit (error=OFFSET_OUT_OF_RANGE

  4. 4

    How to set offset committed by the consumer group using Spark's Direct Stream for Kafka?

  5. 5

    Kafka Consumer with JAVA

  6. 6

    Spring boot Kafka consumer

  7. 7

    Reactive-Kafka Stream Consumer:デッドレターが発生しました

  8. 8

    Partition specific flink kafka consumer

  9. 9

    Kafka elixir consumer keeps crashing

  10. 10

    kafka consumer code is not running completely

  11. 11

    Kafka consumer hangs on poll when kafka is down

  12. 12

    Kafka Stream Ksql Json

  13. 13

    Kafka Stream BoundedMemoryRocksDBConfig

  14. 14

    Kafka consumer - what's the relation of consumer processes and threads with topic partitions

  15. 15

    Kafka consumer group script to see all consumer group not working

  16. 16

    Spring Cloud Stream(Hoxton)Kafka Producer / ConsumerがEmbeddedKafkaとの統合テストで機能しない

  17. 17

    Kafka Avro Consumer with Decoderの問題

  18. 18

    Better way of error handling in Kafka Consumer

  19. 19

    Kafka Consumer get assigned partitions for a specific topic

  20. 20

    How do I implement Kafka Consumer in Scala

  21. 21

    Kafka consumer does not start from latest message

  22. 22

    Incorrect Kafka offset across consumer groups

  23. 23

    Connecting Apache Consumer to a single node on a Kafka Cluster

  24. 24

    Spring Boot Kafka consumer lags and reads wrong

  25. 25

    Issue in implementing infinite retry in kafka consumer

  26. 26

    Kafka - Assign messages to specific Consumer Groups

  27. 27

    Apache Kafka 2.0 get the consumer lag

  28. 28

    How to count the frequency of each ID in kafka Topic

  29. 29

    How to get the processing kafka topic name dynamically in Flink Kafka Consumer?

ホットタグ

アーカイブ