跳至主要内容

Compile Java Using Java

Compile Java Using Java

Recently, we decided to refactor the Filter module of [Syncer](https://github.com/zzt93/syncer/), which is used to be implemented by Spring EL. We want to refactor it for three reasons:

  • The Spring EL is not so fast compared with native Java code;
  • The config specified by Spring EL is rather hard to write and maintain;
  • The config expression defined by myself is rather limited in supported syntax;

So, we came up with two options to upgrade it:

  • First, we upgrade config expression with better hierarchy and support more powerful expression;
  • Second, we introduced the Java code to do the config;

The first option is not related with this post, so we will skip to next one: config application using Java code.

Background

In syncer, we listen on change of different input, and manipulate the data via filter module, then output to target output like following:

version: 1.2  
  
consumerId: sample1  
  
input:  
  masters:  
    - connection:  
        address: 192.168.1.100  
        port: 1234  
      scheduler: mod  
      repos:  
        - name: "simple"  
          entities:  
          - name: test  
            fields: [id, test]  
 
filter:  
- switcher:  
    switch: "table"  
    case: # support default branch  
      "affair": ["#suffix = '-' + row['id']","#type = 'INDEX_AFFAIR'", "renameField('xx', 'yy')" ]  
      "file": ["#suffix = '-' + row['id']","#type = 'INDEX_FILE'", "addRow('type', '0')"]  
      "folder": ["#suffix = '-' + row['id']","#type = 'INDEX_FILE'", "addRow('type', '1')"]  
- statement: [ "#tags = row['tags']", "updateField('tags', new java.util.ArrayList())", "removeFields('id', 'xid')"]  
- foreach:  
    var: "tag"  
    in: "#tags?.split('\n')"  
    statement: ["#map = new java.util.HashMap()", "#map.put('des', #tag)", "row.get('tags').add(#map)"]  
- if:  
    condition: "table == 'affair'"  
    ifBody:  
      create:  
        copy: [row]  
        postCreation: ["table = 'role'", "row['id']"]  
    elseBody:  
      drop: {}
  
# filter result class: com.github.zzt93.syncer.common.data.SyncData  

output:  
  http:  
    connection:  
      address: 192.168.1.100  
      port: 9700  
    jsonMapping:  
      "data":  
        "parentId": "row['parent_id']"  
  "superId": "row['superid']"  
  "allianceId": "row['id']"  
  "row.*": "row.*.flatten"  
  "type": "extra['type']"

So, for a better config usability, we decided to support Java code to do the filter module, which is relative simpler to write and more powerful.

Implementation

In order to do it, we need to do following steps:

  • Read config code from file;
  • Compile the code to byte code;
  • Load class and run it to process data;

The first task is relative simple and we will focus on second and third jobs. By some searching, we can found some sample code of Java Compiler API:

JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();  
compiler.run(null, null, null, classPath.toString());  
  
Class<?> cls;  
try {  
  URLClassLoader classLoader = URLClassLoader.newInstance(new URL[]{path.getParent().toUri().toURL()}, this.getClass().getClassLoader());  
  cls = Class.forName(className, true, classLoader);  
  return (SyncFilter) cls.newInstance();  
} catch (ClassNotFoundException | IllegalAccessException | InstantiationException | MalformedURLException e) {  
  logger.error("Syncer bug", e);  
  return null;  
}

Notice that in the init of classLoader, we pass the current class’s classLoader because in the code to do the config, we may use some clazz of Syncer and this can resolve classpath problem of classLoader:

filter:
 - method: `public void filter(List<SyncData> list) {
      for (SyncData d : list) {
        // ...
      }
    }`

Inside of Jar

The above demo runs fine in IDEA, but fails when we package it into Jar. It complains that some import can’t be found. Though some trials, we realize that the classpath of compiler is different inside of Jar. By setting classpath, finally it works with simple Jar.

JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();  
StandardJavaFileManager fm = compiler.getStandardFileManager(diagnostic -> logger.error("{}, {}", diagnostic.getLineNumber(), diagnostic.getSource().toUri()), null, null);  
try {  
  fm.setLocation(StandardLocation.CLASS_PATH, Lists.newArrayList(new File(System.getProperty("java.class.path"))));  
} catch (IOException e) {  
  logger.error("Fail to set location for compiler file manager", e);  
}  
if (compiler.run(null, null, null, path) != 0) {  
  ShutDownCenter.initShutDown(new InvalidConfigException());  
}

Spring Boot

Actually, we use Spring Boot at the beginning, which has different Jar file structure and fails the solution. We come up with following possible solutions:

  • Implement FileManager to find clazz in Spring Boot liked Jar;
  • Build a shaded Jar from Spring Boot liked Jar;
  • Start application in simple Jar;

The second option fails and considering we are planned to reduce the dependency of Syncer, we finally use the third option to finish the job.

Ref

Written with StackEdit.

评论

发表评论

此博客中的热门博文

Spring Boot: Customize Environment

Spring Boot: Customize Environment Environment variable is a very commonly used feature in daily programming: used in init script used in startup configuration used by logging etc In Spring Boot, all environment variables are a part of properties in Spring context and managed by Environment abstraction. Because Spring Boot can handle the parse of configuration files, when we want to implement a project which uses yml file as a separate config file, we choose the Spring Boot. The following is the problems we met when we implementing the parse of yml file and it is recorded for future reader. Bind to Class Property values can be injected directly into your beans using the @Value annotation, accessed via Spring’s Environment abstraction or bound to structured objects via @ConfigurationProperties. As the document says, there exists three ways to access properties in *.properties or *.yml : @Value : access single value Environment : can access multi
5 条评论
阅读全文

Elasticsearch: Join and SubQuery

Elasticsearch: Join and SubQuery Tony was bothered by the recent change of search engine requirement: they want the functionality of SQL-like join in Elasticsearch! “They are crazy! How can they think like that. Didn’t they understand that Elasticsearch is kind-of NoSQL 1 in which every index should be independent and self-contained? In this way, every index can work independently and scale as they like without considering other indexes, so the performance can boost. Following this design principle, Elasticsearch has little related supports.” Tony thought, after listening their requirements. Leader notice tony’s unwillingness and said, “Maybe it is hard to do, but the requirement is reasonable. We need to search person by his friends, didn’t we? What’s more, the harder to implement, the more you can learn from it, right?” Tony thought leader’s word does make sense so he set out to do the related implementations Application-Side Join “The first implementation
发表评论
阅读全文

Implement isdigit

It is seems very easy to implement c library function isdigit , but for a library code, performance is very important. So we will try to implement it and make it faster. Function So, first we make it right. int isdigit ( char c) { return c >= '0' && c <= '9' ; } Improvements One – Macro When it comes to performance for c code, macro can always be tried. #define isdigit (c) c >= '0' && c <= '9' Two – Table Upper version use two comparison and one logical operation, but we can do better with more space: # define isdigit(c) table[c] This works and faster, but somewhat wasteful. We need only one bit to represent true or false, but we use a int. So what to do? There are many similar functions like isalpha(), isupper ... in c header file, so we can combine them into one int and get result by table[c]&SOME_BIT , which is what source do. Source code of ctype.h : # define _ISbit(bit) (1 << (
发表评论
阅读全文