On teh way

Elasticsearch: Join and SubQuery Tony was bothered by the recent change of search engine requirement: they want the functionality of SQL-like join in Elasticsearch! “They are crazy! How can they think like that. Didn’t they understand that Elasticsearch is kind-of NoSQL 1 in which every index should be independent and self-contained? In this way, every index can work independently and scale as they like without considering other indexes, so the performance can boost. Following this design principle, Elasticsearch has little related supports.” Tony thought, after listening their requirements. Leader notice tony’s unwillingness and said, “Maybe it is hard to do, but the requirement is reasonable. We need to search person by his friends, didn’t we? What’s more, the harder to implement, the more you can learn from it, right?” Tony thought leader’s word does make sense so he set out to do the related implementations Application-Side Join “The first implementation ...

Netty(2): Handler In last blog, we skim the basic abstraction of Netty network model: events cause the state change, user get involved in via Pipeline & Handler . In this blog, we will dive into more design details. Overall Review In Netty , the source of data is not the abstraction in Java, like Stream , but Event . When an event (from remove peer or user defined) is received by Channel , it will use Buffer to hold the data, and give it to Pipeline , which is a serial of Handler . Then, the thread in EventLoop will run the code of Handler ( Callback job, Listener ) to react to it. Today, we focus on core of user code – Pipeline & Handler . Handler Detail Inbound & Outbound As the core of user interaction with Netty , we need to dive into ChannelHandler . It can be categorized as two broad kinds according to the event types: ChannelInboundHandler & ChannelOutboundHandler . The ChannelInboundHandler will respond to network related ev...

Git Abstraction Git is widely used in current software development routine to do the version control of code. With the help of IDE like IntelliJ Idea , we can avoid manually inputing git command in command line, like git add , git commit , git push etc. But when some things goes wrong, it always need command line operations. Today, we are not going to introduce git command, but to understand how git works and what abstraction it have. As soon as we make it clear, we can understand many restrictions of git usage and many command in depth. So, when we met a new problem, we can infer a new solution from what we known, rather than just relying on Google. Abstraction As Torvalds explains, Git is first designed as a file system, so it has different concepts as old VCS system. In many ways you can just see git as a filesystem – it’s content-addressable , and it has a notion of versioning Git is a content-addressable filesystem, which means that we can retrieve th...

Iterator Design Pattern We have talked the usage of iterator when we need to interact with collections: Not expose internal state to outside world: Increasing cohesion of class and avoid mis-operation; A unified interface to operate on different kinds of operation; Enforce the order of method call; Iterator Pitfall public interface Iterator < E > { boolean hasNext ( ) ; E next ( ) ; default void remove ( ) { throw new UnsupportedOperationException ( "remove" ) ; } default void forEachRemaining ( Consumer < ? super E > action ) { Objects . requireNonNull ( action ) ; while ( hasNext ( ) ) action . accept ( next ( ) ) ; } } Checkout the interface of iterator (two more default method added in Java8), and we can see the common usage routine of Iterator . while ( hasNext ( ) ) T t = next ( ) ; // do with t But it is not so friendly for n...

SonarQube Gitlab Integration In order to improve the code quality, our team decided to install the SonarQube to do the static code analysis. Easy Install The installation can be done very easily using docker compose: version: "2" services: sonarqube: image: sonarqube ports: - "9002:9000" environment: - SONARQUBE_JDBC_URL = jdbc:postgresql://db:5432/sonar volumes: - ./sonarqube_conf:/opt/sonarqube/conf - ./sonarqube_data:/opt/sonarqube/data - ./sonarqube_extensions:/opt/sonarqube/extensions - ./sonarqube_bundled-plugins:/opt/sonarqube/lib/bundled-plugins db: image: postgres environment: - POSTGRES_USER = sonar - POSTGRES_PASSWORD = sonar volumes: - ./postgresql:/var/lib/postgresql # This needs explicit mapping due to https://github.com/docker-library/postgres/blob/4e48e3228a30763913ece952c611e5e9b95c8759/Dockerfile.template#L52 - ./postgresql...

Elasticsearch MySQL Sync Challenge (5): Redesign Multiple Syncer? “You may have already heard the new requirement: listening for the data change of main DB and send the data to auth system. What is your idea?” Leader said. "Em, the simple way is of course to write a similar config files as we did before like following: input : masters : - connection : address : 192.168.1.1 port : 3306 filter : ... output : mysql : - connection : address : auth - system --- - input : masters : - connection : address : 192.168.1.1 port : 3306 filter : ... output : elsticsearch : connection : search - system “Then we can start two separate syncer to finish the job of data synchronization for both search system and auth system.” Tony finished. “Yes, it works. But seems not so good.” “Em, yep. Two syncer connect to same data source in which they receive t...

GC Roots This blog is about the a question I received: Are the GC roots of Minor GC and Major GC the same? GC Algorithm If we are asked to come up with a GC collection algorithm, we may think about a solution : in which we maintain the count of reference to some objects, if count drop to 0, we claim the memory. This is called Reference Count , which can’t solve the cyclic reference, is a wrong algorithm. The GC collection algorithm adopted by JVM is marking algorithm from GC roots. The GC roots sounds very mysterious, but its meaning is very simple: GC roots is the object that live outside Java heap and referring to object in Java heap. From the Hotspot JVM source code, we can see the following GC roots types: enum RootType { universe = 1 , jni_handles = 2 , threads = 3 , object_synchronizer = 4 , system_dictionary = 5 , class_loader_data = 6 , management = 7 , ...