跳至主要内容

Logstash Learning (3): Application

In this blog, we will apply Logstash in our project. In this process, we first choose the tech stack, then solve the problem one by one.

Design

Because there exists many different types of input source, we have to choose the one that suitable for our system to use:

  • business application directly write to Logstash by socket/http
  • parse log file using Filebeat, then send to Logstash
  • via message queue: message queue used as buffer

Taking the availability and data consistency into account, we decided to use the following data flow:

business app -> slf4j & logback -> file 
-> filebeat -> logstash -> ES -> kibana

Source: SLF4J & Logback

SLF4J & Logback are very powerful log library for Java that we have introduced some features of it here.

Log Pattern

In our scenario, we customize our project as following, which is very similar to the Spring Boot’s default logging pattern:

logging.pattern.file=%d %5p --- [%t] %-40.40c{39} : %m%n%ex
logging.pattern.console=%d %5p --- [%t] %-40.40c{39} : %m%n%ex

Pattern Rule

  • %{variable}: represent a variable, can be set in application.properties
  • :-: used in ${}, represent the default value of variable

The following is part of config cited from Spring Boot’s config:

<property name="FILE_LOG_PATTERN" value="${FILE_LOG_PATTERN:-%d{yyyy-MM-dd HH:mm:ss.SSS} ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } --- [%t] %-40.40logger{39} : %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}"/>

Sender: Filebeat

Then, we can use Filebeat to read log file and parse and send them. The following is the config we need to care and use.

Tail Log

When application is running, new log will append to log file. Filebeat will automatically detect log file change and send new log entries.

The backoff option defines how long Filebeat waits before checking a file again after EOF is reached. The default is 1s, which means the file is checked every second if new lines were added.

Log Rotation

As normal log library, Logback will rotate the log file when some condition is met, i.e. date or size.

Filebeat can handle this situation automatically: it will continue to read from the rotated log even after it is renamed until the file reaches a certain age (base on modified time) or is deleted. It tracks the file by inode number which doesn’t change when renamed. It will also periodically look for a new files matching the xxx.log file name so that can start reading from the new log file when it is created.

New File Discovery

We can controll the Filebeat new file discovery behavior by setting scan_frequency.

Notice

As we have said in last blog, trying to implement multiline event handling in Logstash may result in the mixing of streams and corrupted data. So we can use following option to handle multiline exception in Filebeat:

multiline.pattern: '(^[a-zA-Z.]+(Error|Exception): .+)|(^\s+at .+)|(^\s+... \d+ )|(^\s*Caused by:.+)'
multiline.negate: false
multiline.match: after

Terminal: Logstash

Because we have enabled Logstash’s persist queue, which can be used as a buffer layer, we don’t need to use message queue. The log data now reach the Logstash.

Example Config

Here is the Logstash and Filebeat config file of our projects:

  • enable the persistent queue
  • enable debug & set debug level
  • enable slow log detection

Performance Troubleshooting Guide

Because logstash is a Java application with many JRuby plugins, i.e. running on JVM, we can profile and tune it like any normal Java application. The following is the checklist to follow if we come across performance problem in production:

  • performance of input/output plugins
  • system statistics:
    • cpu
    • memory
    • I/O
      • disk
      • network
  • JVM heap
  • Logstash worker settings customization

The system and JVM statistic can be monitored by X-Pack and viewed by Kibana, which is very handy to use.

Problem List

Warning at Startup

When start Logstash from command line, it will complain the following warning:

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties.

But the config file do exists in /etc/logstash.

Through some searching, we found the discussion in ELK community, which says if we start Logstash from cmd, it will only look at config file in $LS_HOME/config. Only when we start Logstash by service, i.e. in a daemon process, will it look in /etc/logstash.

So we don’t need to worry about the warning when use it.

TO BE CONTINUED

Ref

Written with StackEdit.

标签：

评论

发表评论

此博客中的热门博文

Spring Boot: Customize Environment

Spring Boot: Customize Environment Environment variable is a very commonly used feature in daily programming: used in init script used in startup configuration used by logging etc In Spring Boot, all environment variables are a part of properties in Spring context and managed by Environment abstraction. Because Spring Boot can handle the parse of configuration files, when we want to implement a project which uses yml file as a separate config file, we choose the Spring Boot. The following is the problems we met when we implementing the parse of yml file and it is recorded for future reader. Bind to Class Property values can be injected directly into your beans using the @Value annotation, accessed via Spring’s Environment abstraction or bound to structured objects via @ConfigurationProperties. As the document says, there exists three ways to access properties in *.properties or *.yml : @Value : access single value Environment : can access multi...
5 条评论
阅读全文

Elasticsearch: Join and SubQuery

Elasticsearch: Join and SubQuery Tony was bothered by the recent change of search engine requirement: they want the functionality of SQL-like join in Elasticsearch! “They are crazy! How can they think like that. Didn’t they understand that Elasticsearch is kind-of NoSQL 1 in which every index should be independent and self-contained? In this way, every index can work independently and scale as they like without considering other indexes, so the performance can boost. Following this design principle, Elasticsearch has little related supports.” Tony thought, after listening their requirements. Leader notice tony’s unwillingness and said, “Maybe it is hard to do, but the requirement is reasonable. We need to search person by his friends, didn’t we? What’s more, the harder to implement, the more you can learn from it, right?” Tony thought leader’s word does make sense so he set out to do the related implementations Application-Side Join “The first implementation ...
发表评论
阅读全文

Learn Spring Expression Language

When reading the source code of some Spring based projects, we can see some code like following: @Value( "${env}" ) private int value ; and like following: @Autowired public void configure (MovieFinder movieFinder, @ Value ("#{ systemProperties[ 'user.region' ] } ") String defaultLocale) { this.movieFinder = movieFinder; this.defaultLocale = defaultLocale; } In this way, we can inject values from different sources very conveniently, and this is the features of Spring EL. What is Spring EL? How to use this handy feature to assist our developments? Today, we are going to learn some basics of Spring EL. Features The full name of Spring EL is Spring Expression Language, which exists in form of Java string and evaluated by Spring. It supports many syntax, from simple property access to complex safe navigation – method invocation when object is not null. And the following is the feature list from Spring EL document : ...
发表评论
阅读全文