跳至主要内容

Common Grok Rules

Common Grok Rules

In this blog, we are going to share some common grok patterns used in Logstash to parse logs from: Nginx (access log & error log with Naxsi), ufw.

Nginx

access.log

Example:

192.168.1.1 - - [30/Oct/2018:09:38:28 +0800] "GET /question?id=yyy HTTP/1.1" 200 808 "https://xxx.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15"

Grok rules:

%{IPORHOST:clientip} (?:-|(%{WORD}.%{WORD})) %{USER:ident} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) "%{GREEDYDATA:referrer}" "%{GREEDYDATA:agent}"( "%{GREEDYDATA:forwarder}")?

error.log

Examples:

2018/10/30 16:35:19 [error] 39685#0: *117137 NAXSI_FMT: ip=192.168.1.1&server=api.xxx&uri=/material&learning=1&vers=0.56&total_processed=24687&total_blocked=198&block=1&cscore0=$SQL&score0=8&zone0=ARGS&id0=1015&var_name0=types, client: 192.168.1.1, server: api.xxx, request: "GET /material?key=&types=REGISTER HTTP/1.1", host: "api.xxx:1443", referrer: "http://192.168.1.204:12306/"

Grok rules:

(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}, client: %{IP:client}, server: %{GREEDYDATA:domain}, request: "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})", host: "%{GREEDYDATA:host}", referrer: "%{GREEDYDATA:referrer}"

Naxsi

If you installed the Naxsi module of nginx as web application firewall, and want to parse error message into more detailed pieces, we need to Grok one more time:

Example:

117137 NAXSI_FMT: ip=192.168.1.1&server=api.xxx&uri=/material&learning=1&vers=0.56&total_processed=24687&total_blocked=198&block=1&cscore0=$SQL&score0=8&zone0=ARGS&id0=1015&var_name0=types

Rule is relative simple:

\*%{NUMBER} NAXSI_FMT: %{GREEDYDATA:param}

Then, we can use kv filter to parse param into more readable json like:

kv {
  source => "param"
  field_split => "&"
  target => "param"
}

UFW

Example:

Oct 30 11:04:57 ubuntu-100 kernel: [11497227.995058] [UFW BLOCK] IN=docker0 OUT= PHYSIN=vethf628260 MAC=02:42:52:2d:ba:c5:02:42:ac:11:00:24:08:00 SRC=172.17.0.36 DST=192.168.1.100 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=48748 DF PROTO=TCP SPT=42634 DPT=19998 WINDOW=237 RES=0x00 ACK FIN URGP=0

Grok rules to do pre-process:

%{SYSLOGTIMESTAMP:ufw_timestamp} %{SYSLOGHOST:ufw_hostname} %{DATA:ufw_program}(?:\[%{POSINT:ufw_pid}\])?: %{GREEDYDATA:ufw_message}

Then, parse ufw_message:

[11497227.995058] [UFW BLOCK] IN=docker0 OUT= PHYSIN=vethf628260 MAC=02:42:52:2d:ba:c5:02:42:ac:11:00:24:08:00 SRC=172.17.0.36 DST=192.168.1.100 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=48748 DF PROTO=TCP SPT=42634 DPT=19998 WINDOW=237 RES=0x00 ACK FIN URGP=0

second step rules:

\[%{DATA}\] \[UFW %{WORD:ufw_action}\] IN=%{DATA:ufw_interface} OUT= (MAC|PHYSIN)=%{DATA:ufw_mac} SRC=%{IP:ufw_src_ip} DST=%{IP:ufw_dest_ip} %{GREEDYDATA:ufw_tcp_opts} PROTO=%{WORD:ufw_protocol} SPT=%{INT:ufw_src_port} DPT=%{INT:ufw_dst_port} %{GREEDYDATA:ufw_tcp_opts}

Grok Debugger

Tools & documentation to debug our own rules:

Written with StackEdit.

评论

发表评论

此博客中的热门博文

Spring Boot: Customize Environment

Spring Boot: Customize Environment Environment variable is a very commonly used feature in daily programming: used in init script used in startup configuration used by logging etc In Spring Boot, all environment variables are a part of properties in Spring context and managed by Environment abstraction. Because Spring Boot can handle the parse of configuration files, when we want to implement a project which uses yml file as a separate config file, we choose the Spring Boot. The following is the problems we met when we implementing the parse of yml file and it is recorded for future reader. Bind to Class Property values can be injected directly into your beans using the @Value annotation, accessed via Spring’s Environment abstraction or bound to structured objects via @ConfigurationProperties. As the document says, there exists three ways to access properties in *.properties or *.yml : @Value : access single value Environment : can access multi...
5 条评论
阅读全文

LevelDB Source Reading (1): Structure

LevelDB Source Reading (1): Structure LevelDB “is an open source on-disk key-value store.” After I read some documents, I have some basic understanding of LevelDB. So I come up with some questions about structure of LevelDB to answer when reading the source code. Structure Log File: repair/recover db A log file (*.log) stores a sequence of recent updates. Each update is appended to the current log file. The log file contents are a sequence of 32KB blocks. The only exception is that the tail of the file may contain a partial block. Block format: Each block consists of a sequence of records: block := record* trailer? record := checksum: uint32 // crc32c of type and data[] ; little-endian length: uint16 // little-endian type: uint8 // One of FULL, FIRST, MIDDLE, LAST data: uint8[length] // data is LengthPrefixedSlice with type from batch data definition in Block : data: also named `writeBatch` in levelDB // WriteBatch header has an 8-byte ...
发表评论
阅读全文

LevelDB Source Reading (3): Compaction

LevelDB Source Reading (3): Compaction In the last blog that analyzes read/write process of Leveldb, we can see writing only happens to log file and memory table, then it relies on the compaction process to move the new updates into persistent sorted table for future use. So the compaction is a crucial part for the design, and we will dive into it in this blog. Compaction LevelDB compacts its underlying storage data in the background to improve read performance. The upper sentence is cited from the document of Leveldb , and we will see how it is implemented via code review. Background compaction // db_impl.cc void DBImpl :: MaybeScheduleCompaction ( ) { // use background thread to run compaction env_ - > Schedule ( & DBImpl :: BGWork , this ) ; } Two main aspects // arrange background compaction when Get, Open, Write void DBImpl :: BackgroundCompaction ( ) { // compact memtable CompactMemTable ( ) ; // compact ...
发表评论
阅读全文