On teh way

Microservice Log Practice(1): Log Log The project tony worked on is now in a painful change: from the old monolithic application to new style microservice model. The leader gives him a job to do the new logging system, from logging the log, log collection, to log analyzing. Log in Monolithic Applications “We attach great importance to the logging system, do you understand why?” Leader asked. “Yep, the log can be used to debug the application in production environment in which we can’t set breakpoints. It can also be used to monitor the application status about the its health. We can even do the intrusion detection and related security by log.” Tony answered. "Yes, you are right. In a monolithic application, we package all functionality into a single Jar / War , then deploy it using application server like Tomcat or even Glassfish . It has some disadvantages like bad scalability, small changes leading to whole application re-deployment etc. But it also som...

Spring Boot Introduction What is Spring Boot? What the usage of it and what’s the differences between Spring Boot and Spring? In this blog, we are going to solve those questions by introducing some basic idea. Project Dependency Structure First, we can see the projects hierarchy from the pom dependency: spring-boot-dependencies # list all dependencies of spring boot: include spring boot jar and some third party jar spring-boot-parent spring-boot-starters # list all starter project, which is a set of convenient dependency descriptor spring-boot-starter-xxx ( data-elastcisearch, web, etc ) # specific tech dependencies spring-boot-starter-parent # can include `spring-boot-starter-xxx` So, what is starter and boot ? Boot Boot is the project to start web application more easily with the help of: Embedded tomcat or jetty in spring-boot-starter-web Spring ApplicationContext etc Starter Starters are a set of convenient dependency descrip...

Design a Logging System? Long long ago, when there is not any mature logging system, Tom is facing the problem of logging. The native logging system form Java SDK has some issues of class loading, in which it may cause memory leaks 1 . Requirement He discussed this problem with leader and they decided to write their own logging system with the following requirements: The logging system can be used by different package, class at the same time and we can easily differentiate them; The logging system can output to different location, for example, console, file or even network; The logging system can output formatted log content: plain text, JSON, XML etc; The logging system can have different levels to control the behavior in different environments: in development env, we can log more, but in production env, we maybe log only exception; Design Tom now begin to analyze the requirements of logging system using OO related patterns and try to make some basic...

Elasticsearch MySQL Sync Challenge (3): Impl Tony and leader has discussed many different ways to sync data from MySQL to Elasticsearch, from simple write split (add more code to send two copy of data to different persistent layers) and bulk load from MySQL using Logstash, to more sophisticated event-driven binlog sync. Why Remake Having decided to use the binlog sync method, tony was asked to choose a suitable tools to finish the job. After doing some search, tony decided to make one by using binlog connector because The complex search requirement needs some extra join in data which is not supported in current tools; Incremental and resume ability: using this way can restart from any location of binlog; Maybe to expend this sync tool to other data stores: like redis, Hadoop etc; Wildcard database support, for reduce the work of horizontal split; Pipe and Filter “Considering this tool is a data hub which just do little mapping/join work but many transferr...

Elasticsearch MySQL Sync Challenge (2): Event Driven Tony was reminded that whether there exists some event-driven update way to sync data from MySQL to Elasticsearch yesterday, so he made more effort to this direction. Observer Pattern "As you have suggested, I have dived into more elegant way of sync data – event-driven way, or in design pattern: a observer pattern. I found three different ways to do. JDBC Logging “Whenever we use JDBC to connect to data base, JDBC can log the SQL statement of our execution. So we can enable this functionality and listening the log. We can easily detect the newly added log entry using Java NIO watch service 1 or other mature tools like FileBeat, but we need to translate the logged SQL statement to Elasticsearch query, which is a relative big work to do. Considering the work of make a simple SQL parser, I think we can deny this option” Tony finished. DB Trigger "The second way works in event-driven model is trigge...

SLF4J Introduction(2): Output In this blog, we focus on how SLF4J and related logging system handle the output of logging. Layout Layout, which is maybe called pattern, is responsible to transform an log event into string with some format. The format of the log event is somewhat like printf method. There are many conversion characters to represent things like class name, date, thread, file name, stack trace, message, caller data etc. Log4j In log4j, the default conversion pattern is %r [%t] %p %c %x - %m%n % : the prefix of conversion character. If we want to print a single % sign, we can use %% r : the number of milliseconds elapsed from the construction of the layout until the creation of the logging event. t : name of thread p : logging priority c : the category of the logging event x : NDC (nested diagnostic context) m : the application supplied message associated with the logging event n : the platform dependent line separator character or c...

Elasticsearch Adventure(3): Dev & Ops The earch engine project now is in test environment. Tony now do some operations of search system to maintain the Elasticsearch cluster. This process is a new adventure in Elasticsearch. har oct s nteerais a e inencers hicss i ane er n lssearch uste to toy er sarinte n. atineieer sad no aee de t tisarhe t ctrhameoce s, used tau reeae e r srd aalot o i shards led o or #### Cluster Health "Tony, tony. Come here. Your Elasticsearch cluster in state of red now." Operation engineer said loudly. So, Tony log in server and check the Elasticsearch cluster state: $curl '192.168.1.100:9200/_cluster/health?pretty' { eeaner rustrealthre "cluster_name" : "searcher-dev" , "status" : "red" , ... "unassigned_shards" : 3 , ... "number_of_pending_tasks" : xxx , } “Statuys is red because some primary shards are not allocated 1 ....

Elasticsearch Learning (4): Mappings “Today, we are going to dive into one of the most important settings when we use Elasticsearch, mappings, which is related to how is ES interpreting our document JSON, how is ES analyzing our field and indexing document, how is ES searching for our request.”, Tony said in the technology sharing time. Mapping and Analysis “As we all known, there exists different data types in common databases, and ES also has some common data types.”, Tony. Text: string for full text search Keyword: string for exact match Whole number: byte, short, integer, long Floating-point: float, double Boolean: boolean Date: date " Text type and Keyword type seems different, can you explain what’s the difference?", someone asked. Tony, “Good question. But before we explain their differences, we need to understand there exists two large categories of data types: exact values and full text” Exact Values vs Full Text "Data in Ela...

With the development and testing of search engine, Tony met more problems. Now, he learned much basic things about Elasticsearch, and he is seeking better practice for problems. Query Condition Combination Tony, “Mentor, I am using rest API of Elasticsearch to assist the access of Elasticsearch for the time being. Now we have a requirement of very flexible searching: user can choose and combine different fields using checkbox to search. I am wondering how to implement it gracefully. For now, I can only think about using different pre-write query string of ES, but it seems cumbersome and I think I may need some string builder to compose a customized query.” Tim, “Em, maybe QueryBuidlers of Elasticsearch Java API is what you want. Let’s say we want a bool query 1 , and the condition for query may exists or not. So we can do like following:” BoolQueryBuilder bool = boolQuery ( ) . should ( matchQuery ( "must_have1name" , info . getQuery ( ) ) ) . should ( neste...

Elasticsearch Adventure(1): Trial and Error Tony is an energetic graduate from University who is admitted by a software company. He is interested in many new technology and recently, he is responsible for developing a simple search engine for test. He skim the basic tutorials and books online ( ES Introduction , Elasticsearch Concepts Confusion , etc ) and start developing his search engine. With the development of searching system, hwe came across more problems of Elasticsearch usage. Fortunately, he has a mentor, Tim, who has rich experience of ES. Query String Tony: “Mentor, our search engine has the requirement of special search techniques, like xx AND yy for intersection of result, aa OR bb for union. Does ES support the interception of similar content?” Tim: “Yes, ES support this features. It has query called query string query . Let’s see an example” GET / _search { "query" : { "query_string" : { ...