I'm working with kafka and I want to monitor topics. Basically I output several topics to stdout and to file for later review. For single topic monitoring I came up with the following command:
${KAFKA_HOME}/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic mytopic | \
while IFS= read -r line; do
printf '[%s | %20s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "mytopic" "${line}";
done | tee -a kafka.out
However, there're many topics to consume, so I'm trying to do something like
consumer=${KAFKA_HOME}/bin/kafka-console-consumer.sh
mapfile -t topics < <(${KAFKA_HOME}/bin/kafka-topics.sh --zookeeper localhost:2181 --list)
for i in ${topics[@]}; do
xterm -T ${i} -e "my_command" &
done
tail -F kafka.out
where my_command is the above command. It doesn't work. I suspect I'm messing up with quotes, unfortunately I can't figure out how to apply them correctly
If you want to store some text verbatim, it's usually easier with this syntax:
mycommand=$(cat << 'EOF'
"${KAFKA_HOME}/bin/kafka-console-consumer.sh" --zookeeper localhost:2181 --topic "$TOPIC" |
while IFS= read -r line; do
printf '[%s | %20s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$TOPIC" "${line}";
done | tee -a kafka.out
EOF
)
In ksh93, bash or zsh, you can also make it:
mycommand=$(<<'EOF'
....
EOF
In ksh93 that optimises out the exec of cat and the fork, while with bash only the exec (in zsh, it makes no difference).
In mksh:
mycommand=<<'EOF'
...
EOF
And then:
export TOPIC KAFKA_HOME
for TOPIC in "${topics[@]}"; do
xterm -T "$TOPIC" -e "$mycommand" &
done
Here, I'd use ts to time-stamp the input (or do it in gawk or perl if ts is not available) instead of that very inefficient while read loop.
ts '[%F %T | mytopic]'
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments