跳至主要内容

Elasticsearch Adventure(1): Trial and Error

Elasticsearch Adventure(1): Trial and Error

Tony is an energetic graduate from University who is admitted by a software company. He is interested in many new technology and recently, he is responsible for developing a simple search engine for test.

He skim the basic tutorials and books online (ES Introduction, Elasticsearch Concepts Confusion, etc ) and start developing his search engine.

With the development of searching system, hwe came across more problems of Elasticsearch usage. Fortunately, he has a mentor, Tim, who has rich experience of ES.

Query String

Tony: “Mentor, our search engine has the requirement of special search techniques, like xx AND yy for intersection of result, aa OR bb for union. Does ES support the interception of similar content?”

Tim: “Yes, ES support this features. It has query called query string query. Let’s see an example”

GET /_search
{
    "query": {
        "query_string" : {
            "fields" : ["content", "name"], # search against both
            "query" : "a AND b" # both have to appear
        }
    }
}

“Query string support many syntax, from commonly used booperator, ranges, groups, to advanced wildcards, regex fuzziness etc. You may like to refer the document for more details.”

Not Exist Value

Tony: “When we index our data structure into Elasticsearch, we sometimes not have all of the data fields of this type. So, we may have null values for some fields. TWhen, when we searching, we need to [deal withing null values otherwise we will encounter errors. So, mentor, do you have any solutions?”

Tim: “I believe that you can fix the similar problem in SQL world.”

Tony: “Maybe, I can do operations select non null field in SQL like following”

SELECT tags
FROM   posts
WHERE  tags IS NOT NULL and xxx

Tim: “Good job, Elasticsearch also has similar syntax called exists/missing query. And corresponding Elasticsearch query may looks like this”:

GET /my_index/posts/_search
{
    "query" : {
        "constant_score" : { # just filter, use constant score
            "filter" : {
                "exists" : { "field" : "tags" } # is tags exists?
            }
        }
    }
}

Tony: “But, as far as I understand,It seems that what thise Elasticsearch query is just checking is whether this field exists or not but not checking value?”

Tim: “Yes, you are very keen about the syntax. Hvalue, however, we should understand if a value is not exists, we can’t store it in the inverted index1, which store our value for searching document. So, in Elasticsearch, we can’t tell the differences between the we don’t have this field and we don’t have this value.”

Tony: “Ok, I understand that. But sometimes, we may need to distinguish the situation that we set it to null and we don’t set it at all. How to achieve it?”

Tim: “Good question. ES also met this question, so it provides us a placeholder for null when we setting our mappings. What we should notice are matching te data type and not setting a real value that could appears.”

Tony: “Wonderful! Thanks for your help.”

Tony: “Mentor, I came across another tricky problem now. When dealing with Chinese, the analyzer become more important and complex, because the standard built-in analyzer2 of Elasticsearch can only split Chinese in one by one char, like this”

data 你好北京
analyzed 你 好 北 京

Tim: “So what’s going on?”

Tony: “After some search I chose the Elasticsearch official document recorded Chinese analyzer: smartcn, which gives better result”

data 你好北京
analyzed 你好 北京

Tim: “Yes, It looks much better.”

Tony: “But, today, when testing the functionality of our search engine, we met some problems.”.

data 你好北京 你好1212

search 你好
result 你好1212

search 你好北京
result 你好北京

Tim: “em, it’s not working. Did you find what’s wrong?”

Tony: “Yes, I found it is because the analyzer smartcn's tokenizer”

data 你好北京 
analyzed 你好 北京 

data 你好1212
analyzed 你 好 1212 

data 你好
analyzed 你 好

“As we can see, It provides different tokenized results for same word, which causes the problem.”

Tim: “In this situation, I suggest you to try more different analyzers and compare the results with more sentence to choose a better one. Here is some analyzers that I know.”

  • analysis-icu
    • tokenizer: icu_tokenizer
  • analysis-ik
    • analyzer: ik_smart
    • analyzer: ik_max_word

The next day, Tony come to Tim again.

Tony: “Mentor, I finished the installation and test, you may like to see the result.”

data IPO企业清障“三类股东”还需监管层发力

smartcn IPO 企业 清 障 “三类 股东” 还 需 监管 层 发 力
icu IPO 企业 清 障 “三 类 股东” 还 需 监管 层 发 力
ik_smart IPO 企业 清障 “三类 股东” 还需 监管 层 发 力
ik_max_word: this analyzer will count all valid combinations of Chinese char in sentence, not suitable for our use case

“From above result, we can see ik_smart works better, with only one error. And even better, ik can add dictionary very easily.”

Tim: “Fine, we can choose ik_smart for our analyzer.”

Ref

Written with StackEdit.

  1. Inverted index is a data structure ES used to assist quick search more here ↩︎

  2. The analyzer is composed by character filer, tokenizer, token filer. Details can be found here. ↩︎

标签：

评论

发表评论

此博客中的热门博文

Spring Boot: Customize Environment

Spring Boot: Customize Environment Environment variable is a very commonly used feature in daily programming: used in init script used in startup configuration used by logging etc In Spring Boot, all environment variables are a part of properties in Spring context and managed by Environment abstraction. Because Spring Boot can handle the parse of configuration files, when we want to implement a project which uses yml file as a separate config file, we choose the Spring Boot. The following is the problems we met when we implementing the parse of yml file and it is recorded for future reader. Bind to Class Property values can be injected directly into your beans using the @Value annotation, accessed via Spring’s Environment abstraction or bound to structured objects via @ConfigurationProperties. As the document says, there exists three ways to access properties in *.properties or *.yml : @Value : access single value Environment : can access multi...
5 条评论
阅读全文

Elasticsearch: Join and SubQuery

Elasticsearch: Join and SubQuery Tony was bothered by the recent change of search engine requirement: they want the functionality of SQL-like join in Elasticsearch! “They are crazy! How can they think like that. Didn’t they understand that Elasticsearch is kind-of NoSQL 1 in which every index should be independent and self-contained? In this way, every index can work independently and scale as they like without considering other indexes, so the performance can boost. Following this design principle, Elasticsearch has little related supports.” Tony thought, after listening their requirements. Leader notice tony’s unwillingness and said, “Maybe it is hard to do, but the requirement is reasonable. We need to search person by his friends, didn’t we? What’s more, the harder to implement, the more you can learn from it, right?” Tony thought leader’s word does make sense so he set out to do the related implementations Application-Side Join “The first implementation ...
发表评论
阅读全文

Learn Spring Expression Language

When reading the source code of some Spring based projects, we can see some code like following: @Value( "${env}" ) private int value ; and like following: @Autowired public void configure (MovieFinder movieFinder, @ Value ("#{ systemProperties[ 'user.region' ] } ") String defaultLocale) { this.movieFinder = movieFinder; this.defaultLocale = defaultLocale; } In this way, we can inject values from different sources very conveniently, and this is the features of Spring EL. What is Spring EL? How to use this handy feature to assist our developments? Today, we are going to learn some basics of Spring EL. Features The full name of Spring EL is Spring Expression Language, which exists in form of Java string and evaluated by Spring. It supports many syntax, from simple property access to complex safe navigation – method invocation when object is not null. And the following is the feature list from Spring EL document : ...
发表评论
阅读全文