Фильтр Logstash Grok для медленных запросов Mongodb

Я пробовал приведенный ниже шаблон фильтра grok, и он работает в отладчике grok, но не работает при развертывании в logstash.

Шаблон:

'%{GREEDYDATA}:"%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA}"s":"%{WORD:severity}",%{SPACE}"c":"%{WORD:component}",%{SPACE}"id":%{NUMBER:id},%{SPACE}"ctx":%{QUOTEDSTRING:context},"msg":%{QUOTEDSTRING:msg},"attr":{"remote":"%{IPV4:client_ip}:%{NUMBER:port}","connectionId":%{NUMBER:connection_id},"connectionCount":%{NUMBER:connection_count}%{GREEDYDATA}',

Ввод:

{"t":{"$date":"2020-11-09T09:51:41.936+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn2468512","msg":"Connection ended","attr":{"remote":"172.21.41.24:58546","connectionId":2468512,"connectionCount":1617}}

Ошибка LogStash:

{"level":"ERROR","loggerName":"logstash.agent","timeMillis":1604933044844,"thread":"Converge PipelineAction::Create<main>","logEvent":{"message":"Failed to execute action","action":{"metaClass":{"metaClass":{"metaClass":{"action":"PipelineAction::Create<main>","exception":"LogStash::ConfigurationError","message":"Expected one of [ \\t\\r\\n], \"#\", [A-Za-z0-9_-], '\"', \"'\", [A-Za-z_], \"-\", [0-9], \"[\", \"{\" at line 26, column 9 (byte 13997) after filter {\n  if [container][image] =~ \"mongodb\" {\n    grok {\n      patterns_dir => [\"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns\"]\n      match => { \"message\" => [\n        '%{GREEDYDATA}:\"%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA}\"s\":\"%{WORD:severity}\",%{SPACE}\"c\":\"%{WORD:component}\",%{SPACE}\"id\":%{NUMBER:id},%{SPACE}\"ctx\":%{QUOTEDSTRING:context},\"msg\":%{QUOTEDSTRING:msg},\"attr\":{\"remote\":\"%{IPV4:client_ip}:%{NUMBER:port}\",\"connectionId\":%{NUMBER:connection_id},\"connectionCount\":%{NUMBER:connection_count}%{GREEDYDATA}',\n        ","backtrace":["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'","org/logstash/execution/AbstractPipelineExt.java:183:in `initialize'","org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'","/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'","/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in `block in converge_state'"]}}}}}}

Вот файл conf, который я использую

filter {
  if [container][image] =~ "mongodb" {
    grok {
      patterns_dir => ["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns"]
      match => { "message" => [
        '%{GREEDYDATA}:"%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA}"s":"%{WORD:severity}",%{SPACE}"c":"%{WORD:component}",%{SPACE}"id":%{NUMBER:id},%{SPACE}"ctx":%{QUOTEDSTRING:context},"msg":%{QUOTEDSTRING:msg},"attr":{"remote":"%{IPV4:client_ip}:%{NUMBER:port}","connectionId":%{NUMBER:connection_id},"connectionCount":%{NUMBER:connection_count}%{GREEDYDATA}',
        ]
        break_on_match => false
        tag_on_failure => ["failed_match"]
      }
    }
  }
}

Сообщите мне, есть ли у кого-нибудь способ решить эту проблему. TIA


mongodb logstash logstash-grok
person Vamshi Siddarth    schedule 09.11.2020    source источник

Ответы (1)


arrow_upward
1
arrow_downward

В вашем grok фильтре отсутствует закрывающая фигурная скобка в опции match`.

grok {
  patterns_dir => ["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns"]
  match => { "message" => ['%{GREEDYDATA}:"%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA}"s":"%{WORD:severity}",%{SPACE}"c":"%{WORD:component}",%{SPACE}"id":%{NUMBER:id},%{SPACE}"ctx":%{QUOTEDSTRING:context},"msg":%{QUOTEDSTRING:msg},"attr":{"remote":"%{IPV4:client_ip}:%{NUMBER:port}","connectionId":%{NUMBER:connection_id},"connectionCount":%{NUMBER:connection_count}%{GREEDYDATA}']}
  break_on_match => false
  tag_on_failure => ["failed_match"]
}

Но ваше сообщение уже является объектом json, вам не нужно использовать grok, вы можете использовать json filter для анализа сообщения и mutate filter, чтобы переименовать поля после синтаксического анализа.

Попробуйте что-то вроде этого

json {
    source => "message"
}
mutate {
    rename => ["[t][$date]","timestamp"]
    rename => ["s","severity"]
    rename => ["c","component"]
    ... the rest of your fields ...
}
person leandrojmp    schedule 09.11.2020