Monitor: Spring Boot Actuator & Prometheus (2)

In the last blog of this topic, we have make a simple runnable example using Spring Boot 2.0 and Prometheus to do the monitoring of system state. But to make it accessible for old version Spring & Jersey, we need to do more works.

Metric in Actuator

In order to migrate the functionality to old versioned Spring & Jersey, we have to understand how actuator achieve it.

The actuator provides a serials of endpoints (like headdump, logfile, metrics, env) to monitor and get information from our applications and we focus only on metric related endpoints. The metric module has three main abstractions:

• Metric: the class hold the metric in form of key & value with timestamp;
• Counter: the service used to increment/decrease count for a metric, like visitor count;
• Gauge: the service used to set a value for a metric, like CPU usage, Memory usage;

The auto configured default implementation is relative simple. With the following simple config:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
    <version>1.5.8.RELEASE</version>
</dependency>

endpoints.enabled=true
# some endpoints are sensitive
endpoints.sensitive=false

We can get the following metric infos.

curl 'http://localhost:8080/metrics'

{"mem":374575,"mem.free":276095,"processors":4,"instance.uptime":612745,"uptime":620414,"systemload.average":7.50830078125,"heap.committed":321024,"heap.init":262144,"heap.used":44928,"heap":3728384,"nonheap.committed":55184,"nonheap.init":2496,"nonheap.used":53551,"nonheap":0,"threads.peak":25,"threads.daemon":21,"threads.totalStarted":28,"threads":23,"classes":6767,"classes.loaded":6767,"classes.unloaded":0,"gc.ps_scavenge.count":8,"gc.ps_scavenge.time":83,"gc.ps_marksweep.count":2,"gc.ps_marksweep.time":92,"httpsessions.max":-1,"httpsessions.active":0,"gauge.response.metrics":163972.0,"gauge.response.star-star.favicon.ico":2.0,"counter.api./favicon.ico":6,"counter.status.200.star-star.favicon.ico":6,"counter.status.200.metrics":6,"counter.api./metrics":6}

This default implementation use MetricFilter to intercept request and record request info into Counter & Gauge and accompany more metrics (like mem & uptime etc) from SystemPublicMetrics like metric collector.

In order to enrich the result of metric, we can use CounterService & GaugeService to customize. The following is an example to count login times.

@Service
public class LoginServiceImpl {
    private final CounterService counterService;   
    public LoginServiceImpl(CounterService counterService) {
        this.counterService = counterService;
    }
     
    public boolean login(String userName, char[] password) {
        boolean success;
        if (userName.equals("admin") && "secret".toCharArray().equals(password)) {
            counterService.increment("counter.login.success");
            success = true;
        } else {
            counterService.increment("counter.login.failure");
            success = false;
        }
        return success;
    }
}

Micrometer

We can customize endpoints and metrics, but there exists an easier way. Spring Boot 2.0 adopted the micrometer dependency as the implementation for metric endpoint and micrometer is back port to Spring 1.5. So we can use micrometer, which is powerful and easy to upgrade.

Comparing with the default implementations, Micrometer has more abstractions except Counter & Gauge.

MeterRegistry

Creates and manages our application’s meters. Exporters use the meter registry to iterate over the set of meters instrumenting your application, and then further iterate over each meter’s metrics, generally resulting in a time series in the metrics backend for each combination of metrics and dimensions.

If we have a dependency on micrometer-registry-{system} in our runtime classpath, Spring Boot will be able to auto configure the registry to use.

Meter Binder

Meter Binders register one or more metrics to Registry to provide information about the state of some aspect of the application or its container. We can find some default binders in MeterBindersConfiguration:

public JvmGcMetrics jvmGcMetrics() {  
    return new JvmGcMetrics();  
}

public JvmMemoryMetrics jvmMemoryMetrics() {  
    return new JvmMemoryMetrics();  
}

More in Metric

In Micrometer, metric has more rich infos than just key, value and timestamp, which is more like the data items in Prometheus (Remember the prometheus output? There exists some tags examples)

Id getId();
Iterable<Measurement> measure();
class Id {  
    private final String name;  
    private final List<Tag> tags;  
    private Type type;
}
public class Measurement {  
    private final Supplier<Double> f;  // measurement value
    private final Statistic statistic; // measurement types
}

Web Monitor

Micrometer contains built-in instrumentation for timings of requests made to Spring MVC and Spring WebFlux server endpoints and it is also implemented via Filters.

Server

# default to be true
management.metrics.web.server.auto-time-requests=true

And if we need customization, we can do it easily via annotations.

@RestController
@Timed (1)
public class MyController {
    @GetMapping("/api/people")
    @Timed(extraTags = { "region", "us-east-1" }) (2)
    @Timed(value = "all.people", longTask = true) (3)
    public List<Person> listPeople() { ... }

Mis

Enabled Endpoint 404 Not Found

If you use the Jersey and set the @ApplicationPath("/") with root path, you will encounter this problem. The console output tells you the /Prometheus is registered, but access it only returns 404.

In order to fix this problem, we need understand that Spring MVC and Jersey all implemented by Filter. If the Jersey intercept the root path, Spring will not dispatch any url. And, our /prometheus endpoint registered in Spring, not in Jersey, so the 404 Not Found.

In order to fix the problem, simply setting the @ApplicationPath("/somePrefix") rather than using root path will fine (Spring dispatcher will handle all other things). If it is not easy to change the url, we have two options, one is to change the default implementation of Jersey to add Spring’s dispatcher, another is to register prometheus endpoint manually using Jersey:

@Path("/prometheus")  
public class PrometheusResource {   
  @Autowired  
  private PrometheusScrapeMvcEndpoint endpoint;  
  
  @GET  
  @Produces(MediaType.TEXT_PLAIN)  
  public Response getPrometheusData() {  
      Object o = endpoint.invoke();  
      ResponseEntity re = (ResponseEntity) o;  
      return Response.ok(re.getBody()).build();  
  }   
}

Aggregate URL with Path Parameter

In our cases, there exists some path which is like send/123/456, i.e. containing some path parameter. This kind of REST style should be aggregated together, otherwise, the number of different urls will be bloated very fast and will be impossible to view in Proemetheus.

As we have introduced, every metric has some tags attached. The url info is actually a tag for a request timing. So we can customize TagProvider to reduce those url.

private Pattern idMatcher = Pattern.compile("/[0-9]+");  
@Override  
public Iterable<Tag> httpRequestTags(HttpServletRequest request, HttpServletResponse response,  
    Object handler, Throwable ex) {  
  Tag tag = WebMvcTags.uri(request, response);  
  String uri = tag.getValue();  
  Matcher matcher = idMatcher.matcher(uri);  
  if (matcher.find()) {  
    String value = matcher.replaceAll("/{id}");  
    return Arrays.asList(WebMvcTags.method(request), Tag.of(tag.getKey(), value),  
        WebMvcTags.exception(ex), WebMvcTags.status(response));  
  }  
  return super.httpRequestTags(request, response, handler, ex);  
}

Full examples can be found here.

Ref

• Micrometer back port documentation
• Actuator example
• Example code

Written with StackEdit.