跳至主要内容

Compile Java Using Java

Compile Java Using Java

Recently, we decided to refactor the Filter module of [Syncer](https://github.com/zzt93/syncer/), which is used to be implemented by Spring EL. We want to refactor it for three reasons:

  • The Spring EL is not so fast compared with native Java code;
  • The config specified by Spring EL is rather hard to write and maintain;
  • The config expression defined by myself is rather limited in supported syntax;

So, we came up with two options to upgrade it:

  • First, we upgrade config expression with better hierarchy and support more powerful expression;
  • Second, we introduced the Java code to do the config;

The first option is not related with this post, so we will skip to next one: config application using Java code.

Background

In syncer, we listen on change of different input, and manipulate the data via filter module, then output to target output like following:

version: 1.2  
  
consumerId: sample1  
  
input:  
  masters:  
    - connection:  
        address: 192.168.1.100  
        port: 1234  
      scheduler: mod  
      repos:  
        - name: "simple"  
          entities:  
          - name: test  
            fields: [id, test]  
 
filter:  
- switcher:  
    switch: "table"  
    case: # support default branch  
      "affair": ["#suffix = '-' + row['id']","#type = 'INDEX_AFFAIR'", "renameField('xx', 'yy')" ]  
      "file": ["#suffix = '-' + row['id']","#type = 'INDEX_FILE'", "addRow('type', '0')"]  
      "folder": ["#suffix = '-' + row['id']","#type = 'INDEX_FILE'", "addRow('type', '1')"]  
- statement: [ "#tags = row['tags']", "updateField('tags', new java.util.ArrayList())", "removeFields('id', 'xid')"]  
- foreach:  
    var: "tag"  
    in: "#tags?.split('\n')"  
    statement: ["#map = new java.util.HashMap()", "#map.put('des', #tag)", "row.get('tags').add(#map)"]  
- if:  
    condition: "table == 'affair'"  
    ifBody:  
      create:  
        copy: [row]  
        postCreation: ["table = 'role'", "row['id']"]  
    elseBody:  
      drop: {}
  
# filter result class: com.github.zzt93.syncer.common.data.SyncData  

output:  
  http:  
    connection:  
      address: 192.168.1.100  
      port: 9700  
    jsonMapping:  
      "data":  
        "parentId": "row['parent_id']"  
  "superId": "row['superid']"  
  "allianceId": "row['id']"  
  "row.*": "row.*.flatten"  
  "type": "extra['type']"

So, for a better config usability, we decided to support Java code to do the filter module, which is relative simpler to write and more powerful.

Implementation

In order to do it, we need to do following steps:

  • Read config code from file;
  • Compile the code to byte code;
  • Load class and run it to process data;

The first task is relative simple and we will focus on second and third jobs. By some searching, we can found some sample code of Java Compiler API:

JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();  
compiler.run(null, null, null, classPath.toString());  
  
Class<?> cls;  
try {  
  URLClassLoader classLoader = URLClassLoader.newInstance(new URL[]{path.getParent().toUri().toURL()}, this.getClass().getClassLoader());  
  cls = Class.forName(className, true, classLoader);  
  return (SyncFilter) cls.newInstance();  
} catch (ClassNotFoundException | IllegalAccessException | InstantiationException | MalformedURLException e) {  
  logger.error("Syncer bug", e);  
  return null;  
}

Notice that in the init of classLoader, we pass the current class’s classLoader because in the code to do the config, we may use some clazz of Syncer and this can resolve classpath problem of classLoader:

filter:
 - method: `public void filter(List<SyncData> list) {
      for (SyncData d : list) {
        // ...
      }
    }`

Inside of Jar

The above demo runs fine in IDEA, but fails when we package it into Jar. It complains that some import can’t be found. Though some trials, we realize that the classpath of compiler is different inside of Jar. By setting classpath, finally it works with simple Jar.

JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();  
StandardJavaFileManager fm = compiler.getStandardFileManager(diagnostic -> logger.error("{}, {}", diagnostic.getLineNumber(), diagnostic.getSource().toUri()), null, null);  
try {  
  fm.setLocation(StandardLocation.CLASS_PATH, Lists.newArrayList(new File(System.getProperty("java.class.path"))));  
} catch (IOException e) {  
  logger.error("Fail to set location for compiler file manager", e);  
}  
if (compiler.run(null, null, null, path) != 0) {  
  ShutDownCenter.initShutDown(new InvalidConfigException());  
}

Spring Boot

Actually, we use Spring Boot at the beginning, which has different Jar file structure and fails the solution. We come up with following possible solutions:

  • Implement FileManager to find clazz in Spring Boot liked Jar;
  • Build a shaded Jar from Spring Boot liked Jar;
  • Start application in simple Jar;

The second option fails and considering we are planned to reduce the dependency of Syncer, we finally use the third option to finish the job.

Ref

Written with StackEdit.

评论

发表评论

此博客中的热门博文

Spring Boot: Customize Environment

Spring Boot: Customize Environment Environment variable is a very commonly used feature in daily programming: used in init script used in startup configuration used by logging etc In Spring Boot, all environment variables are a part of properties in Spring context and managed by Environment abstraction. Because Spring Boot can handle the parse of configuration files, when we want to implement a project which uses yml file as a separate config file, we choose the Spring Boot. The following is the problems we met when we implementing the parse of yml file and it is recorded for future reader. Bind to Class Property values can be injected directly into your beans using the @Value annotation, accessed via Spring’s Environment abstraction or bound to structured objects via @ConfigurationProperties. As the document says, there exists three ways to access properties in *.properties or *.yml : @Value : access single value Environment : can access multi...
5 条评论
阅读全文

LevelDB Source Reading (4): Concurrent Access

In this thread, we come to the issue of concurrent access of LevelDB. As a database, it can be concurrently accessed by users. But, it wouldn’t be easy to provide high throughput under product load. What effort does LevelDB make to achieve this goal both in design and implementation? Goal of Design From this github issue , we can see LevelDB is designed for not allowing multi-process access. this (supporting multiple processes) doesn’t seem like a good feature for LevelDB to implement. They believe let multiple process running would be impossible to share memory/buffer/cache, which may affect the performance of LevelDB. In the case of multiple read-only readers without altering the code base, you could simply copy the file for each reader. Yes, it will be inefficient (though not on file systems that dedupe data), but then again, so would having multiple leveldb processes running as they wouldn’t be able to share their memory/buffer/etc. They achieve it by adding a l...
发表评论
阅读全文

Elasticsearch: Join and SubQuery

Elasticsearch: Join and SubQuery Tony was bothered by the recent change of search engine requirement: they want the functionality of SQL-like join in Elasticsearch! “They are crazy! How can they think like that. Didn’t they understand that Elasticsearch is kind-of NoSQL 1 in which every index should be independent and self-contained? In this way, every index can work independently and scale as they like without considering other indexes, so the performance can boost. Following this design principle, Elasticsearch has little related supports.” Tony thought, after listening their requirements. Leader notice tony’s unwillingness and said, “Maybe it is hard to do, but the requirement is reasonable. We need to search person by his friends, didn’t we? What’s more, the harder to implement, the more you can learn from it, right?” Tony thought leader’s word does make sense so he set out to do the related implementations Application-Side Join “The first implementation ...
发表评论
阅读全文