On teh way

Microservice Log Practice(1): Log Log The project tony worked on is now in a painful change: from the old monolithic application to new style microservice model. The leader gives him a job to do the new logging system, from logging the log, log collection, to log analyzing. Log in Monolithic Applications “We attach great importance to the logging system, do you understand why?” Leader asked. “Yep, the log can be used to debug the application in production environment in which we can’t set breakpoints. It can also be used to monitor the application status about the its health. We can even do the intrusion detection and related security by log.” Tony answered. "Yes, you are right. In a monolithic application, we package all functionality into a single Jar / War , then deploy it using application server like Tomcat or even Glassfish . It has some disadvantages like bad scalability, small changes leading to whole application re-deployment etc. But it also som...

Spring Boot Introduction What is Spring Boot? What the usage of it and what’s the differences between Spring Boot and Spring? In this blog, we are going to solve those questions by introducing some basic idea. Project Dependency Structure First, we can see the projects hierarchy from the pom dependency: spring-boot-dependencies # list all dependencies of spring boot: include spring boot jar and some third party jar spring-boot-parent spring-boot-starters # list all starter project, which is a set of convenient dependency descriptor spring-boot-starter-xxx ( data-elastcisearch, web, etc ) # specific tech dependencies spring-boot-starter-parent # can include `spring-boot-starter-xxx` So, what is starter and boot ? Boot Boot is the project to start web application more easily with the help of: Embedded tomcat or jetty in spring-boot-starter-web Spring ApplicationContext etc Starter Starters are a set of convenient dependency descrip...

Design a Logging System? Long long ago, when there is not any mature logging system, Tom is facing the problem of logging. The native logging system form Java SDK has some issues of class loading, in which it may cause memory leaks 1 . Requirement He discussed this problem with leader and they decided to write their own logging system with the following requirements: The logging system can be used by different package, class at the same time and we can easily differentiate them; The logging system can output to different location, for example, console, file or even network; The logging system can output formatted log content: plain text, JSON, XML etc; The logging system can have different levels to control the behavior in different environments: in development env, we can log more, but in production env, we maybe log only exception; Design Tom now begin to analyze the requirements of logging system using OO related patterns and try to make some basic...

Elasticsearch MySQL Sync Challenge (3): Impl Tony and leader has discussed many different ways to sync data from MySQL to Elasticsearch, from simple write split (add more code to send two copy of data to different persistent layers) and bulk load from MySQL using Logstash, to more sophisticated event-driven binlog sync. Why Remake Having decided to use the binlog sync method, tony was asked to choose a suitable tools to finish the job. After doing some search, tony decided to make one by using binlog connector because The complex search requirement needs some extra join in data which is not supported in current tools; Incremental and resume ability: using this way can restart from any location of binlog; Maybe to expend this sync tool to other data stores: like redis, Hadoop etc; Wildcard database support, for reduce the work of horizontal split; Pipe and Filter “Considering this tool is a data hub which just do little mapping/join work but many transferr...

Elasticsearch MySQL Sync Challenge (2): Event Driven Tony was reminded that whether there exists some event-driven update way to sync data from MySQL to Elasticsearch yesterday, so he made more effort to this direction. Observer Pattern "As you have suggested, I have dived into more elegant way of sync data – event-driven way, or in design pattern: a observer pattern. I found three different ways to do. JDBC Logging “Whenever we use JDBC to connect to data base, JDBC can log the SQL statement of our execution. So we can enable this functionality and listening the log. We can easily detect the newly added log entry using Java NIO watch service 1 or other mature tools like FileBeat, but we need to translate the logged SQL statement to Elasticsearch query, which is a relative big work to do. Considering the work of make a simple SQL parser, I think we can deny this option” Tony finished. DB Trigger "The second way works in event-driven model is trigge...