How to get numbers from a log message in logstash?

Youngmin Kim

I am very new to logstash. I can just run logstash jar file and see a kibana web page. It's cool~~

Now, I want to change a following line (syslog message) to the next line.

Feb 19 18:45:29 SD550 Jack: REG,0x1000,4,10,20,30,40
==>
{ 'timestamp': 'Feb 19 18:45:29', 
  'host': 'SD550', 0x1000:10, 0x1001:20, 0x1002:30, 0x1003:40 }

In log message, '0x1000' is a starting register address, '4' is the number of register values, and next values are just value. So, that means 0x1000:10, 0x1001:20, 0x1002:30, 0x1003:40. An important point is that the number of register values is able to change. As a result, the length of log message can be variable. Even though it has any length, I'd like to get a proper result. (e.g., 0x2000,2,12,22 ==> 0x2000:12, 0x2001:22)

This is my incomplete config file for logstash. I found some filters such as grok, mutate and extractnumbers. But, I don't know how to do what I want to do.

input { 
  file { 
        path => "/var/log/syslog"
        type => "syslog"
  } 
}

filter {
   ???
}

output {
  elasticsearch { }
}

I know I want a lot, sorry guys. In addition, My final goal is to draw a TIME(x)/VALUE(y) chart for a specific register in kibana. Is it possible? Can I have some advice from you?

Thank you, Youngmin Kim

Youngmin Kim

Thank you everybody who answers my question.. Especially, Ben Lim.

With your help, I got this result.

{
      "@version" => "1",
    "@timestamp" => "2014-02-20T11:07:28.125Z",
          "type" => "syslog",
          "host" => "ymkim-SD550",
          "path" => "/var/log/syslog",
            "ts" => "Feb 20 21:07:27",
          "user" => "ymkim",
          "func" => "REG",
          "8192" => 16,
          "8193" => 32,
          "8194" => 17,
          "8195" => 109
}

from $ logger REG,2000,4,10,20,11,6d

This is my config file.

input { 
  file { 
        path => "/var/log/syslog"
        type => "syslog"
  } 
}

filter {
  grok {
        match => ["message", "%{SYSLOGTIMESTAMP:ts} %{SYSLOGHOST:hostname} %{WORD:user}: %{WORD:func},%{WORD:address},%{NUMBER:regNumber},%{GREEDYDATA:regValue}"]
  }

  if [func] == "REG" {  
      modbus_csv {
          start_address => "address"
          num_register => "regNumber"
          source => "regValue"
          remove_field => ["regValue", "hostname", "message", 
                "address", "regNumber"]
      }
  }

}

output {
    stdout { debug => true }
  elasticsearch { }
}

and modified csv filter, named modbus_csv.rb.

# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"

require "csv"

# CSV filter. Takes an event field containing CSV data, parses it,
# and stores it as individual fields (can optionally specify the names).
class LogStash::Filters::MODBUS_CSV < LogStash::Filters::Base
  config_name "modbus_csv"
  milestone 2

  # The CSV data in the value of the source field will be expanded into a
  # datastructure.
  config :source, :validate => :string, :default => "message"

  # Define a list of column names (in the order they appear in the CSV,
  # as if it were a header line). If this is not specified or there
  # are not enough columns specified, the default column name is "columnX"
  # (where X is the field number, starting from 1).
  config :columns, :validate => :array, :default => []
  config :start_address, :validate => :string, :default => "0"
  config :num_register, :validate => :string, :default => "0"

  # Define the column separator value. If this is not specified the default
  # is a comma ','.
  # Optional.
  config :separator, :validate => :string, :default => ","

  # Define the character used to quote CSV fields. If this is not specified
  # the default is a double quote '"'.
  # Optional.
  config :quote_char, :validate => :string, :default => '"'

  # Define target for placing the data.
  # Defaults to writing to the root of the event.
  config :target, :validate => :string

  public
  def register

    # Nothing to do here

  end # def register

  public
  def filter(event)
    return unless filter?(event)

    @logger.debug("Running modbus_csv filter", :event => event)

    matches = 0

    @logger.debug(event[@num_register].hex)
    for i in 0..(event[@num_register].hex)
        @columns[i] = event[@start_address].hex + i
    end
    if event[@source]
      if event[@source].is_a?(String)
        event[@source] = [event[@source]]
      end

      if event[@source].length > 1
        @logger.warn("modbus_csv filter only works on fields of length 1",
                     :source => @source, :value => event[@source],
                     :event => event)
        return
      end

      raw = event[@source].first
      begin
        values = CSV.parse_line(raw, :col_sep => @separator, :quote_char => @quote_char)

        if @target.nil?
          # Default is to write to the root of the event.
          dest = event
        else
          dest = event[@target] ||= {}
        end

        values.each_index do |i|
          field_name = @columns[i].to_s || "column#{i+1}"
          dest[field_name] = values[i].hex
        end

        filter_matched(event)
      rescue => e
        event.tag "_modbus_csvparsefailure"
        @logger.warn("Trouble parsing modbus_csv", :source => @source, :raw => raw,
                      :exception => e)
        return
      end # begin
    end # if event

    @logger.debug("Event after modbus_csv filter", :event => event)

  end # def filter

end # class LogStash::Filters::Csv

Finally, I got a chart what I want. (*func = REG (13) 4096 mean per 10m | (13 hits))

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Get list of tasks numbers from git log based on message

From Dev

Logstash filter timestamp from log message

From Dev

Logstash filter timestamp from log message

From Dev

Get log message from Assertions

From Dev

logstash split message from log4net into two fields

From Dev

How to pipe a sample log message manually to logstash for processing

From Dev

push log from logstash-forwarder to logstash

From Dev

How can I get the Apache Camel Message History in logfile from log4j?

From Dev

log4j: how to get the last inserted log message?

From Dev

how to get a message from this Json

From Dev

How to get numbers from a string

From Dev

How to get Numbers from a String

From Dev

How to get numbers from html?

From Dev

How to get numbers from content

From Dev

How to disable a particular log message from being saved to var/log?

From Dev

How to extract variables from log file path, test log file name for pattern in Logstash?

From Dev

Logstash: How to save an entry from earlier in a log for use across multiple lines later in the log?

From Dev

How to extract variables from log file path, test log file name for pattern in Logstash?

From Dev

([ numbers ], numbers) as function parameter, how to get numbers from outside of [ ]?

From Dev

Can I delete the message field from Logstash?

From Dev

extract numbers from json with logstash and send to elasticsearch

From Dev

How do you get rid of ConfigurationPropertiesRebinderAutoConfiguration INFO log message

From Dev

How i can get log message with yii2

From Dev

Can logstash read directly from remote log?

From Dev

logstash to receive log from android? or is this elasticsearch?

From Dev

Reading from rotating log files in logstash

From Dev

logstash to receive log from android? or is this elasticsearch?

From Dev

Logstash and Multiline Log Entry from Cloud Foundry

From Dev

Logstash does not update @timestamp from apache log

Related Related

  1. 1

    Get list of tasks numbers from git log based on message

  2. 2

    Logstash filter timestamp from log message

  3. 3

    Logstash filter timestamp from log message

  4. 4

    Get log message from Assertions

  5. 5

    logstash split message from log4net into two fields

  6. 6

    How to pipe a sample log message manually to logstash for processing

  7. 7

    push log from logstash-forwarder to logstash

  8. 8

    How can I get the Apache Camel Message History in logfile from log4j?

  9. 9

    log4j: how to get the last inserted log message?

  10. 10

    how to get a message from this Json

  11. 11

    How to get numbers from a string

  12. 12

    How to get Numbers from a String

  13. 13

    How to get numbers from html?

  14. 14

    How to get numbers from content

  15. 15

    How to disable a particular log message from being saved to var/log?

  16. 16

    How to extract variables from log file path, test log file name for pattern in Logstash?

  17. 17

    Logstash: How to save an entry from earlier in a log for use across multiple lines later in the log?

  18. 18

    How to extract variables from log file path, test log file name for pattern in Logstash?

  19. 19

    ([ numbers ], numbers) as function parameter, how to get numbers from outside of [ ]?

  20. 20

    Can I delete the message field from Logstash?

  21. 21

    extract numbers from json with logstash and send to elasticsearch

  22. 22

    How do you get rid of ConfigurationPropertiesRebinderAutoConfiguration INFO log message

  23. 23

    How i can get log message with yii2

  24. 24

    Can logstash read directly from remote log?

  25. 25

    logstash to receive log from android? or is this elasticsearch?

  26. 26

    Reading from rotating log files in logstash

  27. 27

    logstash to receive log from android? or is this elasticsearch?

  28. 28

    Logstash and Multiline Log Entry from Cloud Foundry

  29. 29

    Logstash does not update @timestamp from apache log

HotTag

Archive