On teh way

Elasticsearch MySQL Sync Challenge (1) Today, Tony was called into team leader’s office. The leader said, “You have done a great job to finish the basic search engine implementations recently. Now, we have another very challenging task for you to do: To sync the data between database and ES. Do you have any ideas?” Tony said, “em, for now, I can only come up with the idea that we write some codes in business layer to write another copy of data to ES.” Leader, “This is a way, but there exists other methods that deserve to be dive into. You can do some research and we will talk about it later. But remember our requirements of this sync job” Aims Updated asynchronously - The user’s database query request search request should be delayed as little as possible. Eventually consistent - While it can lag behind slightly, serving stale results indefinitely isn’t an option. Easy to rebuild - Updates can be lost before reaching Elasticsearch, and Elasticsearch itsel...

Elasticsearch: Join and SubQuery Tony was bothered by the recent change of search engine requirement: they want the functionality of SQL-like join in Elasticsearch! “They are crazy! How can they think like that. Didn’t they understand that Elasticsearch is kind-of NoSQL 1 in which every index should be independent and self-contained? In this way, every index can work independently and scale as they like without considering other indexes, so the performance can boost. Following this design principle, Elasticsearch has little related supports.” Tony thought, after listening their requirements. Leader notice tony’s unwillingness and said, “Maybe it is hard to do, but the requirement is reasonable. We need to search person by his friends, didn’t we? What’s more, the harder to implement, the more you can learn from it, right?” Tony thought leader’s word does make sense so he set out to do the related implementations Application-Side Join “The first implementation ...

Netty(2): Handler In last blog, we skim the basic abstraction of Netty network model: events cause the state change, user get involved in via Pipeline & Handler . In this blog, we will dive into more design details. Overall Review In Netty , the source of data is not the abstraction in Java, like Stream , but Event . When an event (from remove peer or user defined) is received by Channel , it will use Buffer to hold the data, and give it to Pipeline , which is a serial of Handler . Then, the thread in EventLoop will run the code of Handler ( Callback job, Listener ) to react to it. Today, we focus on core of user code – Pipeline & Handler . Handler Detail Inbound & Outbound As the core of user interaction with Netty , we need to dive into ChannelHandler . It can be categorized as two broad kinds according to the event types: ChannelInboundHandler & ChannelOutboundHandler . The ChannelInboundHandler will respond to network related ev...

Git Abstraction Git is widely used in current software development routine to do the version control of code. With the help of IDE like IntelliJ Idea , we can avoid manually inputing git command in command line, like git add , git commit , git push etc. But when some things goes wrong, it always need command line operations. Today, we are not going to introduce git command, but to understand how git works and what abstraction it have. As soon as we make it clear, we can understand many restrictions of git usage and many command in depth. So, when we met a new problem, we can infer a new solution from what we known, rather than just relying on Google. Abstraction As Torvalds explains, Git is first designed as a file system, so it has different concepts as old VCS system. In many ways you can just see git as a filesystem – it’s content-addressable , and it has a notion of versioning Git is a content-addressable filesystem, which means that we can retrieve th...

Iterator Design Pattern We have talked the usage of iterator when we need to interact with collections: Not expose internal state to outside world: Increasing cohesion of class and avoid mis-operation; A unified interface to operate on different kinds of operation; Enforce the order of method call; Iterator Pitfall public interface Iterator < E > { boolean hasNext ( ) ; E next ( ) ; default void remove ( ) { throw new UnsupportedOperationException ( "remove" ) ; } default void forEachRemaining ( Consumer < ? super E > action ) { Objects . requireNonNull ( action ) ; while ( hasNext ( ) ) action . accept ( next ( ) ) ; } } Checkout the interface of iterator (two more default method added in Java8), and we can see the common usage routine of Iterator . while ( hasNext ( ) ) T t = next ( ) ; // do with t But it is not so friendly for n...

SonarQube Gitlab Integration In order to improve the code quality, our team decided to install the SonarQube to do the static code analysis. Easy Install The installation can be done very easily using docker compose: version: "2" services: sonarqube: image: sonarqube ports: - "9002:9000" environment: - SONARQUBE_JDBC_URL = jdbc:postgresql://db:5432/sonar volumes: - ./sonarqube_conf:/opt/sonarqube/conf - ./sonarqube_data:/opt/sonarqube/data - ./sonarqube_extensions:/opt/sonarqube/extensions - ./sonarqube_bundled-plugins:/opt/sonarqube/lib/bundled-plugins db: image: postgres environment: - POSTGRES_USER = sonar - POSTGRES_PASSWORD = sonar volumes: - ./postgresql:/var/lib/postgresql # This needs explicit mapping due to https://github.com/docker-library/postgres/blob/4e48e3228a30763913ece952c611e5e9b95c8759/Dockerfile.template#L52 - ./postgresql...

Elasticsearch MySQL Sync Challenge (5): Redesign Multiple Syncer? “You may have already heard the new requirement: listening for the data change of main DB and send the data to auth system. What is your idea?” Leader said. "Em, the simple way is of course to write a similar config files as we did before like following: input : masters : - connection : address : 192.168.1.1 port : 3306 filter : ... output : mysql : - connection : address : auth - system --- - input : masters : - connection : address : 192.168.1.1 port : 3306 filter : ... output : elsticsearch : connection : search - system “Then we can start two separate syncer to finish the job of data synchronization for both search system and auth system.” Tony finished. “Yes, it works. But seems not so good.” “Em, yep. Two syncer connect to same data source in which they receive t...