跳至主要内容

Elasticsearch Problem Lists(3): Spring Upgrade

In order to use the new features of Elasticsearch/Kibana, we decided to upgrade to newer version of Spring. Because the new version of Spring is not released version and not stable, we met some problems and bugs. This post records them for future reader.
In last blog, we have specified that the version of Spring Boot Starter is 1.5.3, we have to upgrade to 2.0.0.M3 to support ELK 5.x.

Add Repo

The first step is to update the maven dependency. Because the new version of Spring Boot that supports the ES 5.x is not released, we have to add customized repositories to download pom and jar.
<repository>
    <id>spring-milestone</id>
    <name>spring-milestone</name>
    <url>http://repo.spring.io/milestone/</url>
</repository>
Except the code repo, we also need update the plugin repo for we used the Spring Boot maven plugin:
  <plugin>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-maven-plugin</artifactId>
  </plugin>

<pluginRepositories>
    <pluginRepository>
        <id>spring-milestone</id>
        <name>spring-milestone</name>
        <url>http://repo.spring.io/milestone/</url>
    </pluginRepository>
</pluginRepositories>

Migration

@Field

  • FieldType add new type: text – analyzed string, keyword – not analyzed string;
  • Remote FieldIndex, replace with a boolean flag;

findById

  • Return type change to Optional

PageRequest

  • Change from constructor to factory method

Settings Error

After fixing the compiler error, we run the project. But it complains the following error message:
Node settings must not contain any index level settings.
Searching in google give me this issuse, which explains the change in ES 5.x:
Since elasticsearch 5.x index level settings can NOT be set on the nodes
configuration like the elasticsearch.yaml, in system properties or command line arguments.In order to upgrade all indices the settings must be updated via the /${index}/_settings API.
Following stacktrace, we debug source and find out which setting causes the error:
spring.data.elasticsearch.properties.index.search.slowlog.threshold.query.info=1ms
Searching this setting in project, we remove it from property file and it works.

Index Closed

Now, the project can run but the regression test fails.
It seems that the index created in 2.x is closed and not show in _aliases. Open by command fix the problem.

No Node Available

The next exception is NoNodeAvailableException:
ERROR --- [main] .d.e.r.s.AbstractElasticsearchRepository : failed to load elasticsearch nodes : org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{xQ0rTHudQg6kP3203NEEMg}{192.168.1.100}{192.168.1.100:9300}]

Possible Reasons

According to the experience, there exists two possible reasons:
  • Wrong address & port
  • No username and password: We have installed the X-Pack in ELK tech stack for the features of monitoring, alter etc. Security is also one of the features of X-Pack, which enable the authentication of using ELK.

Debug Source

Even though we guess it is the second reason, we decide to debug to make sure we are at right direction.
No node available is a somewhat vague exception and the code where it were thrown is where the ES client is used, so we need to find where the client is init.
Setting the breakpoint at it is init, we can find it is a PreBuiltTransportClient, which is a auto-configured bean by String Boot (which can be seen from set Spring Boot in Debug model).
Following the invocation chain of adding and testing connection of our remote server:
TransportClientFactoryBean#buildClient
=>TransportClient#addTransportAddress
=>TransportClientNodesService#addTransportAddresses
=>NodeSampler#doSample
=>SimpleNodeSampler#doSample
And finally, in doSample, it try to connect to remote server and throws the exception:
RemoteTransportException[[KvLjp4K][192.168.1.100:9300][cluster:monitor/nodes/liveness]]; nested: ElasticsearchSecurityException[missing authentication token for action [cluster:monitor/nodes/liveness]];
But this exception is not logged as expected. ES use log4j to log and Spring Data Elasticsearch use slf4j log4j bridge. In normal case, it should work and log through slf4j, but it is not logged.

Solution

Found the source of problem, we need to add the username and password when connect to ES server. But things is not so simple as expected.
Searching sometimes, we finally find we have to used customized TransportClient from X-Pack with user and password.
<dependency>
  <groupId>org.elasticsearch.client</groupId>
  <artifactId>x-pack-transport</artifactId>
  <version>${es.version}</version>
</dependency>
PreBuiltXPackTransportClient client = new PreBuiltXPackTransportClient(settings());

Settings.builder()
  .put("cluster.name", properties.getClusterName())
  .put("xpack.security.user", "elastic:changeme")

Index Not Found

Then, next exception is IndexNotFoundException.

Wrong Stacktrace

The confusing and hard-to-solve part of this problem is the wrong stacktrace. The line in stacktrace is not reached. 
[affair-] IndexNotFoundException[no such index
]
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.infe(IndexNameExpressionResolver.java:676)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:630)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:578)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:168)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:144)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:77)
    ...

Capture Packet

No other way, we have to capture packet to detect where the name is send. In wireshark, we start capture the packet:
tcp.port == 9300
and a better filter can be:
elasticsearch
Then we find the action before the error packet:

From the view, we can see it is a refresh affair- action. Then we set the breakpoint here:
public <S extends T> S save(S entity) {
    elasticsearchOperations
      .index(createIndexQuery(entity));
    elasticsearchOperations
      .refresh(entityInformation.getIndexName());
    return entity;
}
Debug told us the entityInformation gives us the wrong index name:
public String getIndexName() {
    return indexName;
}
which in older version is:
public String getIndexName() {
    return indexName != null ? indexName : entityMetadata.getIndexName();
}

Real Problem

The source of problem is Someone set the field eagerly after 3.0.0.M2, which disables the dynamic index/type via Spring EL like following:
@Document(indexName = "affair-#{suffix.toString()}", type = "affair", refreshInterval = "1s", createIndex = false)
public class AffairPO {}

Solution

We have already submit the bug to Spring site and we use a local maven repo to host a temp bug fixed version to make project run.

Ref

Written with StackEdit.
标签：

评论

发表评论

此博客中的热门博文

Spring Boot: Customize Environment

Spring Boot: Customize Environment Environment variable is a very commonly used feature in daily programming: used in init script used in startup configuration used by logging etc In Spring Boot, all environment variables are a part of properties in Spring context and managed by Environment abstraction. Because Spring Boot can handle the parse of configuration files, when we want to implement a project which uses yml file as a separate config file, we choose the Spring Boot. The following is the problems we met when we implementing the parse of yml file and it is recorded for future reader. Bind to Class Property values can be injected directly into your beans using the @Value annotation, accessed via Spring’s Environment abstraction or bound to structured objects via @ConfigurationProperties. As the document says, there exists three ways to access properties in *.properties or *.yml : @Value : access single value Environment : can access multi
5 条评论
阅读全文

Elasticsearch: Join and SubQuery

Elasticsearch: Join and SubQuery Tony was bothered by the recent change of search engine requirement: they want the functionality of SQL-like join in Elasticsearch! “They are crazy! How can they think like that. Didn’t they understand that Elasticsearch is kind-of NoSQL 1 in which every index should be independent and self-contained? In this way, every index can work independently and scale as they like without considering other indexes, so the performance can boost. Following this design principle, Elasticsearch has little related supports.” Tony thought, after listening their requirements. Leader notice tony’s unwillingness and said, “Maybe it is hard to do, but the requirement is reasonable. We need to search person by his friends, didn’t we? What’s more, the harder to implement, the more you can learn from it, right?” Tony thought leader’s word does make sense so he set out to do the related implementations Application-Side Join “The first implementation
发表评论
阅读全文

Learn Spring Expression Language

When reading the source code of some Spring based projects, we can see some code like following: @Value( "${env}" ) private int value ; and like following: @Autowired public void configure (MovieFinder movieFinder, @ Value ("#{ systemProperties[ 'user.region' ] } ") String defaultLocale) { this.movieFinder = movieFinder; this.defaultLocale = defaultLocale; } In this way, we can inject values from different sources very conveniently, and this is the features of Spring EL. What is Spring EL? How to use this handy feature to assist our developments? Today, we are going to learn some basics of Spring EL. Features The full name of Spring EL is Spring Expression Language, which exists in form of Java string and evaluated by Spring. It supports many syntax, from simple property access to complex safe navigation – method invocation when object is not null. And the following is the feature list from Spring EL document :
发表评论
阅读全文