跳至主要内容

Elasticsearch Adventure(1): Trial and Error

Elasticsearch Adventure(1): Trial and Error

Tony is an energetic graduate from University who is admitted by a software company. He is interested in many new technology and recently, he is responsible for developing a simple search engine for test.

He skim the basic tutorials and books online (ES Introduction, Elasticsearch Concepts Confusion, etc ) and start developing his search engine.

With the development of searching system, hwe came across more problems of Elasticsearch usage. Fortunately, he has a mentor, Tim, who has rich experience of ES.

Query String

Tony: “Mentor, our search engine has the requirement of special search techniques, like xx AND yy for intersection of result, aa OR bb for union. Does ES support the interception of similar content?”

Tim: “Yes, ES support this features. It has query called query string query. Let’s see an example”

GET /_search
{
    "query": {
        "query_string" : {
            "fields" : ["content", "name"], # search against both
            "query" : "a AND b" # both have to appear
        }
    }
}

“Query string support many syntax, from commonly used booperator, ranges, groups, to advanced wildcards, regex fuzziness etc. You may like to refer the document for more details.”

Not Exist Value

Tony: “When we index our data structure into Elasticsearch, we sometimes not have all of the data fields of this type. So, we may have null values for some fields. TWhen, when we searching, we need to [deal withing null values otherwise we will encounter errors. So, mentor, do you have any solutions?”

Tim: “I believe that you can fix the similar problem in SQL world.”

Tony: “Maybe, I can do operations select non null field in SQL like following”

SELECT tags
FROM   posts
WHERE  tags IS NOT NULL and xxx

Tim: “Good job, Elasticsearch also has similar syntax called exists/missing query. And corresponding Elasticsearch query may looks like this”:

GET /my_index/posts/_search
{
    "query" : {
        "constant_score" : { # just filter, use constant score
            "filter" : {
                "exists" : { "field" : "tags" } # is tags exists?
            }
        }
    }
}

Tony: “But, as far as I understand,It seems that what thise Elasticsearch query is just checking is whether this field exists or not but not checking value?”

Tim: “Yes, you are very keen about the syntax. Hvalue, however, we should understand if a value is not exists, we can’t store it in the inverted index1, which store our value for searching document. So, in Elasticsearch, we can’t tell the differences between the we don’t have this field and we don’t have this value.”

Tony: “Ok, I understand that. But sometimes, we may need to distinguish the situation that we set it to null and we don’t set it at all. How to achieve it?”

Tim: “Good question. ES also met this question, so it provides us a placeholder for null when we setting our mappings. What we should notice are matching te data type and not setting a real value that could appears.”

Tony: “Wonderful! Thanks for your help.”

Tony: “Mentor, I came across another tricky problem now. When dealing with Chinese, the analyzer become more important and complex, because the standard built-in analyzer2 of Elasticsearch can only split Chinese in one by one char, like this”

data 你好北京
analyzed 你 好 北 京

Tim: “So what’s going on?”

Tony: “After some search I chose the Elasticsearch official document recorded Chinese analyzer: smartcn, which gives better result”

data 你好北京
analyzed 你好 北京

Tim: “Yes, It looks much better.”

Tony: “But, today, when testing the functionality of our search engine, we met some problems.”.

data 你好北京 你好1212

search 你好
result 你好1212

search 你好北京
result 你好北京

Tim: “em, it’s not working. Did you find what’s wrong?”

Tony: “Yes, I found it is because the analyzer smartcn's tokenizer”

data 你好北京 
analyzed 你好 北京 

data 你好1212
analyzed 你 好 1212 

data 你好
analyzed 你 好

“As we can see, It provides different tokenized results for same word, which causes the problem.”

Tim: “In this situation, I suggest you to try more different analyzers and compare the results with more sentence to choose a better one. Here is some analyzers that I know.”

  • analysis-icu
    • tokenizer: icu_tokenizer
  • analysis-ik
    • analyzer: ik_smart
    • analyzer: ik_max_word

The next day, Tony come to Tim again.

Tony: “Mentor, I finished the installation and test, you may like to see the result.”

data IPO企业清障“三类股东”还需监管层发力

smartcn IPO 企业 清 障 “三类 股东” 还 需 监管 层 发 力
icu IPO 企业 清 障 “三 类 股东” 还 需 监管 层 发 力
ik_smart IPO 企业 清障 “三类 股东” 还需 监管 层 发 力
ik_max_word: this analyzer will count all valid combinations of Chinese char in sentence, not suitable for our use case

“From above result, we can see ik_smart works better, with only one error. And even better, ik can add dictionary very easily.”

Tim: “Fine, we can choose ik_smart for our analyzer.”

Ref

Written with StackEdit.

  1. Inverted index is a data structure ES used to assist quick search more here ↩︎

  2. The analyzer is composed by character filer, tokenizer, token filer. Details can be found here. ↩︎

标签：

评论

发表评论

此博客中的热门博文

Spring Boot: Customize Environment

Spring Boot: Customize Environment Environment variable is a very commonly used feature in daily programming: used in init script used in startup configuration used by logging etc In Spring Boot, all environment variables are a part of properties in Spring context and managed by Environment abstraction. Because Spring Boot can handle the parse of configuration files, when we want to implement a project which uses yml file as a separate config file, we choose the Spring Boot. The following is the problems we met when we implementing the parse of yml file and it is recorded for future reader. Bind to Class Property values can be injected directly into your beans using the @Value annotation, accessed via Spring’s Environment abstraction or bound to structured objects via @ConfigurationProperties. As the document says, there exists three ways to access properties in *.properties or *.yml : @Value : access single value Environment : can access multi
5 条评论
阅读全文

Elasticsearch: Join and SubQuery

Elasticsearch: Join and SubQuery Tony was bothered by the recent change of search engine requirement: they want the functionality of SQL-like join in Elasticsearch! “They are crazy! How can they think like that. Didn’t they understand that Elasticsearch is kind-of NoSQL 1 in which every index should be independent and self-contained? In this way, every index can work independently and scale as they like without considering other indexes, so the performance can boost. Following this design principle, Elasticsearch has little related supports.” Tony thought, after listening their requirements. Leader notice tony’s unwillingness and said, “Maybe it is hard to do, but the requirement is reasonable. We need to search person by his friends, didn’t we? What’s more, the harder to implement, the more you can learn from it, right?” Tony thought leader’s word does make sense so he set out to do the related implementations Application-Side Join “The first implementation
发表评论
阅读全文

Implement isdigit

It is seems very easy to implement c library function isdigit , but for a library code, performance is very important. So we will try to implement it and make it faster. Function So, first we make it right. int isdigit ( char c) { return c >= '0' && c <= '9' ; } Improvements One – Macro When it comes to performance for c code, macro can always be tried. #define isdigit (c) c >= '0' && c <= '9' Two – Table Upper version use two comparison and one logical operation, but we can do better with more space: # define isdigit(c) table[c] This works and faster, but somewhat wasteful. We need only one bit to represent true or false, but we use a int. So what to do? There are many similar functions like isalpha(), isupper ... in c header file, so we can combine them into one int and get result by table[c]&SOME_BIT , which is what source do. Source code of ctype.h : # define _ISbit(bit) (1 << (
发表评论
阅读全文