On teh way

In order to collect, visualize, analyze logs, we decided to use ELK to finish those related jobs. In two serials of blog, we have already introduced some basics about Logstash and Elasticsearch: Elasticsearch Learning (1): Introduction Logstash Learning (1): Basic If you are not familiar with Elasticsearch and Logstash, you may find those posts useful. ELK Architecture ELK architecture First, we would like to go through the recommended architecture in official document : Multiple nodes – for robustness and resilience against node failure; Filebeat – which ensure the at-least-once delivery and enable load balance to send logs across multiple Logstash nodes; Logstash – enable persistent queue, to provide protection across node failures; Elasticsearch Now, we come to how to setup Elasticsearch clusters. The basic configs to set up a cluster is very easy in Elasticsearch. Bind Address We can choose an array of addresses to let Elasticsearch to bind, so Elasticsear...

In order to use the new features of Elasticsearch/Kibana, we decided to upgrade to newer version of Spring. Because the new version of Spring is not released version and not stable, we met some problems and bugs. This post records them for future reader. In last blog, we have specified that the version of Spring Boot Starter is 1.5.3, we have to upgrade to 2.0.0.M3 to support ELK 5.x. Add Repo The first step is to update the maven dependency. Because the new version of Spring Boot that supports the ES 5.x is not released, we have to add customized repositories to download pom and jar. < repository > < id > spring-milestone </ id > < name > spring-milestone </ name > < url > http://repo.spring.io/milestone/ </ url > </ repository > Except the code repo, we also need update the plugin repo for we used the Spring Boot maven plugin: < plugin > < groupId > org.springframework.boot </ groupId > ...

In this blog, we will apply Logstash in our project. In this process, we first choose the tech stack, then solve the problem one by one. Design Because there exists many different types of input source, we have to choose the one that suitable for our system to use: business application directly write to Logstash by socket/http parse log file using Filebeat, then send to Logstash via message queue: message queue used as buffer Taking the availability and data consistency into account, we decided to use the following data flow: business app -> slf4j & logback -> file -> filebeat -> logstash -> ES -> kibana Source: SLF4J & Logback SLF4J & Logback are very powerful log library for Java that we have introduced some features of it here . Log Pattern In our scenario, we customize our project as following, which is very similar to the Spring Boot’s default logging pattern: logging.pattern. file = %d %5p --- [ %t ] %- 40.40 c{...

In the last blog, we have introduced some concepts in Logstash: the log data flow from input to filter to output, the buffer & batch etc. In the this blog, we focus on how to setup Logstash. Settings Files After installing Logstash, we can find its settings files under /etc/logstash (in linux): logstash.yml: Logstash parameter config file log4j2.properties: Logstash logging config jvm.options: Logstash JVM config startup.options: It is used by system-install script in /usr/share/logstash/bin to build the startup script. It will setup options like user, group, service name, and service decription. logstash.yml Except the normal form of yml config file functionality, the logstash.yml file also supports bash-style interpolation of environment variables in setting values. pipeline: batch: size: ${BATCH_SIZE} delay: ${BATCH_DELAY:5} node: name: "node_ ${LS_NODE_NAME} " path: queue: "/tmp/ ${QUEUE_DIR:queue} " We can al...

Today, we will learn some basic concepts of Logstash, which is a important component in ELK and is responsible for log aggregation. This serial will include following topics: Introduction: this post Configuration Application Concepts Logstash has three main conponents: input, filter and output. It is designed to follow the principle of loose coupling between components. So, it adopt the Pipe and Filter design patterns, making the plugins of Logstash very easy to be added or removed in execution pipeline of log. Pipeline Configuration When using Logstash to handle logs, we use pipeline to define the flow of logs. A simple pipeline configuration file can be looked like following: input { stdin {} } output { stdout {} } This config only defines input and output components, because filter is a optional component. There exists many different plugins can be used in different components and we will introduce them in next blog . Batch & Buffer In o...

Introduction SLF4J is a very commonly used log library in many Java application. It is favored because it allows the end-user to plug in the desired logging framework at deployment time. How is this feature implemented? How to use it? Let’s continue. Binding API & Implementation SLF4J library has two main parts: api and implementations. Its implementations are referred as bindings. In order to use SLF4J we have to both the api jar and bindings. We can replace slf4j bindings on our class path to switch logging frameworks. If we start a project with only slf4j-api-x.jar dependency, we will see the following warning: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder" . SLF4J: Defaulting to no-operation ( NOP ) logger implementation SLF4J: See http://www .slf 4j .org /codes .html #StaticLoggerBinder for further details. This is because if no binding is found on the class path, SLF4J will default to a no-operation implementation. C...

In last blog, we have introduced some problems about Elasticsearch basic concepts confusions and some config problems we met. Now, we come to the problems came cross when using Elasticsearch in application with the help of Spring Data Elasticsearch. With Spring After the understanding of Elasticsearch and configuration of server, we need to write code to interact with it. We choose the Spring Data Elasticsearch framework to assist our implementations. So the following is the problem we met when using Spring to access Elasticsearch. spring-boot-starter-data-elasticsearch: 1.5.3-RELEASE Elasticsearch server: 2.4.x Connection Clients When using Java to access Elasticsearch, we have two types of clients to choose to communicate with server: Transport Client: this client won’t be part of cluster, It just communicate with server Node Client: this client will be part of cluster – store data shards and respond search request In our cases, we just want to communicate...