On teh way

LevelDB Source Reading (2): Read Write Last blog introduces the structure of leveldb, this blog will introduce how leveldb handle reading/writing and some related question. Reading Reading Procedure: check memtable (skiplist) check immutable memtable iterate sorted table in different level[0…] to find possible file using table index ([key => offset]) to find the suitable block via binary search binary search in restart array to find the last restart point with a key < target linear search possible entries between two restarting point to check Step 1 & 2: // dp_impl.cc // First look in the memtable, then in the immutable memtable (if any). LookupKey lkey ( key , snapshot ) ; if ( mem - > Get ( lkey , value , & s ) ) { // Done } else if ( imm != NULL && imm - > Get ( lkey , value , & s ) ) { // Done Step 3: // version_set.cc: Version::Get // We can search level-by-level since entries ne...

Netty(4): Performance Besides the event style design that we can learn from Netty , we can also learn experiences it used to optimize networking communication. Object Pool Buffer Recycle: Reference Count As we all known, Java has many powerful garbage collector and algorithms ( G1 , CMS ; and notice that reference count is not a valid GC algorithm, which can’t solve cyclic reference) to clean up unused object and free memory. In most cases, we don’t need to care about object creation and de-allocation thanks to the work of GC . But, sometimes, we need object pools to do some optimizations ( in most cases, this is not necessary and may make thing worse ). One of examples like this is connection pool which is expensive to create. In order to do object pooling, we need to adopt the reference count in case of explicit de-allocate and the Buffer recycle in Netty is implemented by this feature. More general, objects supporting recycling all implement io.netty.uti...

Netty(1): Abstration As the one of the most widely used network library in Java, Netty has become the de facto standard of network communication in Java. In this blog, we learn some basic concept and designs of Netty by examples in their website . The following is the template code to start a server: EventLoopGroup group = new NioEventLoopGroup ( ) ; EventLoopGroup worker = new NioEventLoopGroup ( ) ; try { ServerBootstrap b = new ServerBootstrap ( ) ; b . group ( group , worker ) . channel ( NioServerSocketChannel . class ) . handler ( new xxxHandler ( ) ) . childHandler ( new yyyHandler ( ) ) ; b . bind ( PORT ) . sync ( ) . channel ( ) . closeFuture ( ) . await ( ) ; } finally { worker . shutdownGracefully ( ) ; group . shutdownGracefully ( ) ; } In the view of normal Java plain IO & Servlet developer, this code snippet can be very confusing. In plain Java IO way, we bind, listen,...

Spring Boot Port Rebind Sometimes, our application need to be opened in one instance, we can bind to a port to make sure it. Sometimes, out application is dedicated to started in multiple instances and each one should have a different port for listening connection, we may better to let our application automatically choose in some ranges which can save much config pain. Recently, we are trying to add health check endpoint in Syncer (a Spring Boot based data sync application), and we need to add the auto rebind functionality with Spring Boot . Implementation One of Spring Boot 's feature is the embedded server to ease the pain to deploy, but it also encapsulate too much, making it very hard to customize by ourself, because we don’t know how it works. We searched like Spring Boot rebind if failed , Spring Boot change server port , but only found how to customize embedded container . So, we have to do by ourselves. In order to understand how it works, we st...

支付宝微信支付的坑 沙箱 Alipay 根据 这里 商家的账号信息，配置沙箱环境下的 appId ，商户 private key ，支付宝公钥（设置商户私钥之后会自动在账户信息处生成）。然后下载 这里 的沙箱app，登陆客户账户，然后就可以付款。付款成功之后，便会回调我们设置的回调地址。 Wechat 微信沙箱环境的key需要根据真实的key生成，并且有时间限制，需要注意 // 微信的sdk需要去官网下载 WXPay wxpay = new WXPay ( config , true , true ) ; String retrieveSandboxSignKey ( WXPayConfig config ) { try { Map < String , String > params = new HashMap < > ( ) ; params . put ( "mch_id" , config . getMchID ( ) ) ; params . put ( "nonce_str" , WXPayUtil . generateNonceStr ( ) ) ; params . put ( "sign" , WXPayUtil . generateSignature ( params , config . getKey ( ) ) ) ; String strXML = wxpay . requestWithoutCert ( "/sandboxnew/pay/getsignkey" , params , config . getHttpConnectTimeoutMs ( ) , config . getHttpReadTimeoutMs ( ) ) ; if ( StringUtils . isBlank ( strXML ) ) { throw new SandboxUnavailableException ( ) ; ...

Netty(4): Event, Channel, Buffer In Netty , the source of data is not the abstraction in Java OIO , like Stream , but Event . One of obvious differences between Stream & Event is how we handle IO request. In OIO , we read/write from Stream and block if it is not available; In Netty , we are notified when corresponding event happens. Netty have much common in this design with Java NIO . In this blog, we will dive into Netty 's similarity between Java NIO : event, channel, buffer. Event The following is event propagation methods we have introduced in Handler part. Inbound event propagation methods: [ ChannelHandlerContext.fireChannelRegistered() ] [ ChannelHandlerContext.fireChannelActive() ] [ ChannelHandlerContext.fireChannelRead(Object) ] [ ChannelHandlerContext.fireChannelReadComplete() ] [ ChannelHandlerContext.fireExceptionCaught(Throwable) ] [ ChannelHandlerContext.fireUserEventTriggered(Object) ] [ ChannelHandlerContext.fireChannelW...

Netty(3): EventLoop We have discussed the Handler design in last blog : the InboundHandler & OutboundHandler , the handler & childHandler , the Servlet vs Handler . In this blog, we will dive into the thread pool abstraction in Netty – EventLoop . EventLoop Basic From our previous code example, we can see that what we used are EventLoopGroup , which is a group of EventLoop s, but what is EventLoop on earth? Loop & Pool The EventLoop , as its name indicates, is a infinite loop to handle IO events. A common implementation ( NioEventLoop ) is like following: protected void run ( ) { for ( ; ; ) { try { switch ( selectStrategy . calculateStrategy ( selectNowSupplier , hasTasks ( ) ) ) { case SelectStrategy . CONTINUE : continue ; case SelectStrategy . SELECT : select ( wakenUp . getAndSet ( false ) ) ; if ( wakenUp . get ( ) ) { selec...