On teh way

LevelDB Source Reading (1): Structure LevelDB “is an open source on-disk key-value store.” After I read some documents, I have some basic understanding of LevelDB. So I come up with some questions about structure of LevelDB to answer when reading the source code. Structure Log File: repair/recover db A log file (*.log) stores a sequence of recent updates. Each update is appended to the current log file. The log file contents are a sequence of 32KB blocks. The only exception is that the tail of the file may contain a partial block. Block format: Each block consists of a sequence of records: block := record* trailer? record := checksum: uint32 // crc32c of type and data[] ; little-endian length: uint16 // little-endian type: uint8 // One of FULL, FIRST, MIDDLE, LAST data: uint8[length] // data is LengthPrefixedSlice with type from batch data definition in Block : data: also named `writeBatch` in levelDB // WriteBatch header has an 8-byte ...

LevelDB Source Reading (3): Compaction In the last blog that analyzes read/write process of Leveldb, we can see writing only happens to log file and memory table, then it relies on the compaction process to move the new updates into persistent sorted table for future use. So the compaction is a crucial part for the design, and we will dive into it in this blog. Compaction LevelDB compacts its underlying storage data in the background to improve read performance. The upper sentence is cited from the document of Leveldb , and we will see how it is implemented via code review. Background compaction // db_impl.cc void DBImpl :: MaybeScheduleCompaction ( ) { // use background thread to run compaction env_ - > Schedule ( & DBImpl :: BGWork , this ) ; } Two main aspects // arrange background compaction when Get, Open, Write void DBImpl :: BackgroundCompaction ( ) { // compact memtable CompactMemTable ( ) ; // compact ...

LevelDB Source Reading (2): Read Write Last blog introduces the structure of leveldb, this blog will introduce how leveldb handle reading/writing and some related question. Reading Reading Procedure: check memtable (skiplist) check immutable memtable iterate sorted table in different level[0…] to find possible file using table index ([key => offset]) to find the suitable block via binary search binary search in restart array to find the last restart point with a key < target linear search possible entries between two restarting point to check Step 1 & 2: // dp_impl.cc // First look in the memtable, then in the immutable memtable (if any). LookupKey lkey ( key , snapshot ) ; if ( mem - > Get ( lkey , value , & s ) ) { // Done } else if ( imm != NULL && imm - > Get ( lkey , value , & s ) ) { // Done Step 3: // version_set.cc: Version::Get // We can search level-by-level since entries ne...

Netty(4): Performance Besides the event style design that we can learn from Netty , we can also learn experiences it used to optimize networking communication. Object Pool Buffer Recycle: Reference Count As we all known, Java has many powerful garbage collector and algorithms ( G1 , CMS ; and notice that reference count is not a valid GC algorithm, which can’t solve cyclic reference) to clean up unused object and free memory. In most cases, we don’t need to care about object creation and de-allocation thanks to the work of GC . But, sometimes, we need object pools to do some optimizations ( in most cases, this is not necessary and may make thing worse ). One of examples like this is connection pool which is expensive to create. In order to do object pooling, we need to adopt the reference count in case of explicit de-allocate and the Buffer recycle in Netty is implemented by this feature. More general, objects supporting recycling all implement io.netty.uti...

Netty(1): Abstration As the one of the most widely used network library in Java, Netty has become the de facto standard of network communication in Java. In this blog, we learn some basic concept and designs of Netty by examples in their website . The following is the template code to start a server: EventLoopGroup group = new NioEventLoopGroup ( ) ; EventLoopGroup worker = new NioEventLoopGroup ( ) ; try { ServerBootstrap b = new ServerBootstrap ( ) ; b . group ( group , worker ) . channel ( NioServerSocketChannel . class ) . handler ( new xxxHandler ( ) ) . childHandler ( new yyyHandler ( ) ) ; b . bind ( PORT ) . sync ( ) . channel ( ) . closeFuture ( ) . await ( ) ; } finally { worker . shutdownGracefully ( ) ; group . shutdownGracefully ( ) ; } In the view of normal Java plain IO & Servlet developer, this code snippet can be very confusing. In plain Java IO way, we bind, listen,...

Spring Boot Port Rebind Sometimes, our application need to be opened in one instance, we can bind to a port to make sure it. Sometimes, out application is dedicated to started in multiple instances and each one should have a different port for listening connection, we may better to let our application automatically choose in some ranges which can save much config pain. Recently, we are trying to add health check endpoint in Syncer (a Spring Boot based data sync application), and we need to add the auto rebind functionality with Spring Boot . Implementation One of Spring Boot 's feature is the embedded server to ease the pain to deploy, but it also encapsulate too much, making it very hard to customize by ourself, because we don’t know how it works. We searched like Spring Boot rebind if failed , Spring Boot change server port , but only found how to customize embedded container . So, we have to do by ourselves. In order to understand how it works, we st...