On teh way

In last blog, we have introduced some problems about Elasticsearch basic concepts confusions and some config problems we met. Now, we come to the problems came cross when using Elasticsearch in application with the help of Spring Data Elasticsearch. With Spring After the understanding of Elasticsearch and configuration of server, we need to write code to interact with it. We choose the Spring Data Elasticsearch framework to assist our implementations. So the following is the problem we met when using Spring to access Elasticsearch. spring-boot-starter-data-elasticsearch: 1.5.3-RELEASE Elasticsearch server: 2.4.x Connection Clients When using Java to access Elasticsearch, we have two types of clients to choose to communicate with server: Transport Client: this client won’t be part of cluster, It just communicate with server Node Client: this client will be part of cluster – store data shards and respond search request In our cases, we just want to communicate

When using Elasticsearch, we are confronted with many problems, this post will list the problems and solutions for future reader. Elasticsearch Concepts The first set of problems are related to the some concepts in Elasticsearch which we are confused about at the very beginning of adoption. Indices or Shards In Elasticsearch, shards is the minimal runnable unit for searching and scaling. When we search within a single index, Elasticsearch forwards the search request to a primary or replica of every shard in that index, and then gathers the results from each shard. So, we can understand that Searching one index that has five primary shards is exactly equivalent to searching five indices that have one primary shard each. And, that is the reason why Elasticsearch suggests us to rolling the index to scale our applications, because shards number are fixed. Parent Child vs Nested Saying we have a person class to search, and we have some tags describing them: c

In the last blog, we have discussed some natures of Elasticsearch: how it search text and how it store document. Now, we will focus on search deeper. Inverted Index Revisit We have talked about that Elasticsearch use the inverted index to assist searching functionality, but Elasticsearch build a more complex inverted index than we have talked. Analyzed Inverted Index So we have following two documents: Doc_1: The quick brown fox jumped over the lazy dog Doc_2: Quick brown foxes leap over lazy dogs in summer A simple inverted index looks like following: Term Doc_1 Doc_2 ------------------------- Quick | | X The | X | brown | X | X dog | X | dogs | | X fox | X | foxes | | X in | | X jumped | X | lazy | X | X leap | | X over | X | X quick | X | summer | | X the | X | ------------------------ A normalized index in which docume

Concepts node, cluster master: creating or deleting an index, or adding or removing a node from the cluster index, mapping/type, document Term Elasticsearch Equivalent in RDMS Index collection of different type of documents and document properties database Shards the horizontal separation of a index X Type/Mapping collection of dicuments sharing a set of common fields table Document collection of fields defined in JSON row ID Every document is associated with ID primary key Example - - - - - - - - - - - - - - - - - - - - - - - Node0 Node1 Node2 shard0 shard1 shard1 shard0 - - - - - - - - - - - - - - - - - - - - - - - - one copy of replicas two copy of data & two shards (shard0, shard1) three nodes Application clients Java Restful curl - XGET 'localhost:92

Today, we start the learning of Elasticsearch, a widely used search engine based on Lucene. This will be a serial of blogs with following or more subjects: Introduction: this post Implementation Search Application: Similar Concepts With Spring Now, we starts from the basic concepts of Elasticsearch and then how to apply it in our programs. Basic Concepts As a distributed system, Elasticsearch has some common concepts like Node , Cluster , but as a storage system in general, it has some specific terms to describe its structure, as following introduced. Node: single instance of Elasticsearch Cluster: collection of nodes Term Elasticsearch Equivalent in RDMS Index collection of different type of documents and document properties database Shards the horizontal separation of a index Type/Mapping collection of dicuments sharing a set of common fields table Document collection of fields defined in JSON row UI