how many is too many Kafka commits per second
shmish111
I have read that you shouldn't commit offsets with the Kafka high level consumer too often as each commit is a write to Zookeeper and so you can slow down Zookeeper because it's strongly consistent across its quorum.
My question is, what is considered a lot of writes to Zookeeper? I am sending 10 messages per second and it would make my system a lot nicer if I could afford to commit after every message but is this a lot? I really don't know. I would ideally like some idea before I start testing on our fully loaded kafka cluster.
Dirk
If you look at this graph, you can see, that their example setup (3x dual 2Ghz Xeon and two SATA 15K RPM drives) can handle ~80000 writes/s if you have only writes.
Collected from the Internet
Please contact [email protected] to delete if infringement.
edited at
Related
Related Related
- 1
Error: too many emails per second on Heroku
- 2
Too many commits in SourceTree, how to remove them?
- 3
How many document reads per minute is too many in Firestore?
- 4
Kafka connect - how many tasks per connector
- 5
How to solve git conflict with master that has too many commits ahead?
- 6
How many threads is too many?
- 7
How many parameters are too many?
- 8
How many repositories is too many?
- 9
How many coroutines is too many?
- 10
kafka Too many open files
- 11
Kafka: too many SubsribableChannels and MessageChannels
- 12
How to Measure How Many Results App Can Produce Per Second
- 13
Calling many methods of many objects many times per second
- 14
Too Many commits in react profiler without renders
- 15
How many attempts per second can a password cracker actually make?
- 16
How many interrupts does my cpu have per second?
- 17
Is it possible to limit how many goroutines run per second?
- 18
affdex-sdk. How many frames can processed per second?
- 19
Scroll event firing too many times. I only want it to fire a maximum of, say, once per second
- 20
Getting "Script invoked too many times per second" when running script in Google Spreadsheet
- 21
How do I execute commands many many times per second in Golang?
- 22
Ceph: too many PGs per OSD
- 23
How many is too many ID in an html page?
- 24
How many bad blocks in a HDD is too many?
- 25
How many parameters are too many in JavaScript?
- 26
How many components are too many in ReactJS?
- 27
How many Futures is too many in Java?
- 28
Laravel - How many queries is too many?
- 29
How many commas are too many commas in CSS
Comments