On teh way

Elasticsearch MySQL Sync Challenge (4): Quality Attributes Tony was called into the leader’s office. After the basic functionality is fulfilled, Tony has started to review the quality attributes of the project. Extensibility “The syncer you have completed works great and it has good extensibility for its relative loose restriction between different modules (input, filter, output). Now, we need one more output choice: to output to MySQL server. How long do you think to finish this feature?” “Em, at least three days, for both coding and testing.” “Fine, go for it.” The process to extend one more output choice is relative simple: Add MySQL output config config package, which is used to mapping config item in yaml to class; Add MySQL output channel, which is the abstraction remote destination; Add SqlMapper , which convert the SyncData to sql statement when SyncData reach the MySQL output channel; The first item is implemented very fast but the seco...

Elasticsearch MySQL Sync Challenge (3): Impl Tony and leader has discussed many different ways to sync data from MySQL to Elasticsearch, from simple write split (add more code to send two copy of data to different persistent layers) and bulk load from MySQL using Logstash, to more sophisticated event-driven binlog sync. Why Remake Having decided to use the binlog sync method, tony was asked to choose a suitable tools to finish the job. After doing some search, tony decided to make one by using binlog connector because The complex search requirement needs some extra join in data which is not supported in current tools; Incremental and resume ability: using this way can restart from any location of binlog; Maybe to expend this sync tool to other data stores: like redis, Hadoop etc; Wildcard database support, for reduce the work of horizontal split; Pipe and Filter “Considering this tool is a data hub which just do little mapping/join work but many transferr...

Elasticsearch MySQL Sync Challenge (2): Event Driven Tony was reminded that whether there exists some event-driven update way to sync data from MySQL to Elasticsearch yesterday, so he made more effort to this direction. Observer Pattern "As you have suggested, I have dived into more elegant way of sync data – event-driven way, or in design pattern: a observer pattern. I found three different ways to do. JDBC Logging “Whenever we use JDBC to connect to data base, JDBC can log the SQL statement of our execution. So we can enable this functionality and listening the log. We can easily detect the newly added log entry using Java NIO watch service 1 or other mature tools like FileBeat, but we need to translate the logged SQL statement to Elasticsearch query, which is a relative big work to do. Considering the work of make a simple SQL parser, I think we can deny this option” Tony finished. DB Trigger "The second way works in event-driven model is trigge...