Search Icon

Ryan Harrison My blog, portfolio and technology related ramblings

30 Useful Java Libraries

A collection of some interesting Java libraries and frameworks (not including the big ones everyone uses such as Spring or JUnit). Hopefully it contains some lesser known libraries which you might not have heard of before and may find useful.

Name Description
ArchUnit Write unit tests to enforce the architecture of your Java code. E.g. all your service classes follow established conventions for naming and package layout through to finding cyclic dependencies.
AssertJ Forget Hamcrest or those provided by JUnit, AssertJ is the only library you will need for writing assertions. Provides countless rich integrations which are easily discoverable through auto-completion.
Awaitility Testing asynchronous systems can be challenging. Awaitility provides a simple DSL that allows you to define the expectations of async operations within tests - without having to deal with threads, timeouts and other concurrency issues.
Caffeine A high performance, near optimal caching library with a modern lambda-based API. Can be easily integrated into Spring applications through its standard caching abstractions.
Eclipse Collections Perhaps not needed in all cases, but can give much improved GC times and overall performance over the standard Java collections in some high throughput pipelines. Also includes immutable versions for better safety in parallel workloads.
Failsafe A lightweight, zero-dependency library for handling all kinds of application failures. Provides a concise API that allows you to wrap executable logic within a number of resilience policies such as Retry, Timeout, Fallback and CircuitBreaker.
FastExcel Apache POI has numerous features for interacting with Excel files, but the API is cumbersome and resource consumption can be a problem for larger files. FastExcel provides a much simpler API to generate and read big Excel files quickly.
Guava Numerous core Java libraries from Google including additional collection types and utilities for concurrency, I/O, hashing, caching and strings. Perhaps becoming less necessary now, but still a great extension over the Java standard library.
Handlebars Simple, logic-less templating engine when you want something a bit more lightweight than Freemarker or Thymeleaf.
HikariCP The best JDBC connection pool out there - fast, simple, reliable and with “zero-overhead”. The default in newer Spring Boot based applications.
Immutables For those that prefer not to use Lombok, but still like all the benefits of immutable classes. Immutables is an annotation processor that generates simple, fast and consistent value objects (data classes) without all the traditional boilerplate.
Faker Generate all kinds of fake data (names, addresses, emails and many more). Really useful when you need to create some throwaway data as part of tests or quick proof-of-concepts/demos.
Jimfs An in-memory file system for Java from Google. Great for unit testing when you don’t want to/can’t mock the underlying filesystem (usually due to static methods from java.nio.Files etc).
Jib Build optimised Docker images from your Java applications directly from Maven/Gradle, without the need for CLI dependencies or Dockerfiles. Jib separates your app into multiple layers (splitting dependencies from classes) for much quicker rebuilds.
jOOR The Java Reflection API is powerful, but often cumbersome to use. jOOR provides a simple, fluent wrapper that gives much more intuitive access to the standard meta information and class structures.
Lombok A set of annotations that helps remove all the typical boilerplate and verbosity in Java applications. Easily auto-generate data classes with constructors, getters/setters, toString, equals/hashcode, builders, loggers and more.
Micrometer A vendor-neutral application metrics facade (think SLF4J, but for metrics). Support for dimensional metrics and Spring Boot Actuator offers a number of useful out-of-the-box integrations.
Mug A small set of utilities over Java 8 Streams and Optionals. Also includes a functional style Maybe implementation.
Picocli A user-friendly framework for building command-line based apps with the JVM. Supports autocompletion, colours, subcommands and is GraalVM compatible.
Resilience4J A lightweight fault tolerance library for Java 8+ and a successor for Netflix Hystrix. Provides decorators for Circuit Breakers, Rate Limiters, Retries and Bulkheads. Can also be used as a circuit breaker implementation within Spring Cloud Circuit Breaker.
Rest Assured A powerful DSL for writing tests over your RESTful API’s.
Retrofit A lightweight and type-safe HTTP client for the JVM. A common go-to within Android applications to interact with external services.
ShedLock A distributed lock that makes sure your scheduled tasks are executed at most once at the same time. Uses an external store like a Redis, JDBC database or Zookeeper for coordination.
StreamEx Many enhancements over the standard Java Stream API that make common tasks shorter and more convenient. Fully compatible with the built-in Java 8 Stream classes, but provides many additional useful methods.
TestContainers Write proper integration tests that use Docker to spin-up throwaway instances of databases, queues, browsers and more. Integrates with JUnit and doesn’t require any extra complex configuration. No longer feel the need to mock out external services!
ThreeTen Extra Additional date-time classes that complement those already in Java 8 (MutableClock, LocalDateRange etc). Curated by the primary author of the new java.time API.
Vavr An essential for all the functional programmers out there. Provides all the essential data types and functional control structures alongside a number of extra modules for integrations with other libraries and frameworks.
Verbal Expressions A library that helps to construct difficult regular expressions with a fluent and human-readable API.
WireMock A library for stubbing and mocking HTTP services. Allows you to construct a standalone local web server that can be used within integration tests to verify expected behaviour against external API’s. Also integrates with Spring Cloud Contract.
Yavi A lambda-based, type-safe validation library with no reflection, annotations or dependencies. For those who are not fans of using the Bean Validation frameworks.

For even more Java libraries check out https://github.com/akullpp/awesome-java

Read More

Gathering Metrics with Micrometer and Spring Boot Actuator

Note: Also check out this follow-up post which covers how to query and create reusable dashboards from your metrics: Aggregating and Visualizing Spring Boot Metrics with Prometheus and Grafana

Why Metrics?

Metrics, alongside tracing and logging, form the concept of observability - one of the key cornerstones of DevOps - so hopefully it should be of no surprise to anyone of its importance. As we build larger and more distributed platforms, maintaining sufficient visibility into what the system is doing, when it’s doing it and how well it’s performing becomes more difficult, but also more vital. We can likely no longer just directly query the application for these measurements anymore. Instead, we require aggregations across all our service instances, as well as the ability to drill-down into particular aspects of a metric to gain most insight.

The usefulness of metrics also goes well past just capturing overall system health and resource usage. We can still observe common things like memory and CPU usage, garbage collection and thread utilization metrics - from the SRE perspective these can still go a long way to monitor and manage a production system - but we should also be looking to go much further than just machine level statistics. Well-defined business level metrics, collected automatically by our applications in real-time, are a prerequisite to both our ability to define and measure against SLA’s/SLO’s, alongside our ability to make well informed, data-driven decisions as part of the wider Agile process.

Micrometer

Libraries and frameworks for gathering metrics are nothing new in the Java world. Things like Dropwizard have been around for many years now, but more recently Micrometer has taken a more prominent position - and for good reason. In a lot of ways it supersedes the alternatives.

Micrometer provides a simple facade over the instrumentation clients for the most popular monitoring systems, allowing you to instrument your JVM-based application code without vendor lock-in. Think SLF4J, but for metrics.

The above description is straight from the Micrometer website and I think the SLF4J analogy sums it up quite well. Just like for logging where we use the core API from SLF4J and then pick and choose whichever concrete logging library we want (Logback, Log4J), the same applies here, but for metrics. The core Micrometer library provides the API’s and interfaces for defining timers, gauges, counters (and more), but out-of-the-box doesn’t do too much more than just store them in memory. After you have defined your metrics, you can then plug-in connectors for a number of external monitoring systems or time-series databases - Azure Monitor, Netflix Atlas, Datadog, Graphite, InfluxDB, Prometheus or even just simple JMX. The main facade stays intact, which then quite rightly decouples the collection of metrics from any/all external consumers.

A nice and modern API is all well and good, but perhaps the larger points to mention here are its support for dimensional metrics (see below) and the first-party support from Spring. From Spring Boot 2.0 (1.x used Dropwizard) each component is integrated with Micrometer and will automatically publish metrics out-of-the-box (again, more on this below).

Hierarchical vs. Dimensional Metrics

Traditionally, metrics have really been not much more than just simple key/value pairs - where the key is usually some dot-separated string namespaced into a particular application area and the value being a simple numerical measurement. This is what’s known as a hierarchical metric, where as the key gets longer, the measurement gets more specific. This used to be the way that most libraries captured metrics and indeed is how Dropwizard and Spring Boot 1.x function. Hierarchical metrics do however have a large limitation - consider the below keys for capturing metrics for HTTP endpoints:

http.requests
http.requests.getwidgets
http.requests.getwidgets.region
http.requests.getwidgets.region.200
http.requests.getwidgets.region.200.username
http.requests.getwidgets.region.200.username.duration
...

From the top metric we measure the most general aggregate - total number of HTTP requests served - and then get gradually more specific by introducing additional fields. These tend to make the metric more insightful, as we are likely going to need to drill-down to a specific region/datacentre or endpoint name and then find the total number of 200 or 500 responses for example. Hopefully you can see the issue here - as we add more fields, the number of metrics we gather starts to grow and grow. The above example is just for a single endpoint, username and status code, they can quickly start to multiply as your system is used. This is what’s known as a cardinality explosion and something to be careful of in applications using hierarchical metrics. It can increase resource consumption and cause pressure on the external systems trying to consume and report on them.

The case for hierarchical metrics gets worse though as the final measurement is tied to the full key. This is fine if it represents exactly what you’re looking for - in the above example, perhaps reporting on the total number of 200 responses for the getwidgets endpoint in a particular region is helpful - but what if you need a different combination?

  • total number of 200 responses served (not just getwidgets)
  • total number of requests made by a particular username (independent of endpoint or status code)
  • overall duration spent handling getwidgets requests (independent of status code or region)

With this set of metrics the above questions are not trivial to answer. You would either be forced to create even more metrics specific to these cases (making the aforementioned cardinality explosion worse), or be forced to perform complex extra aggregation steps in your reporting tool of choice

The idea of dimensional metrics aims to solve these issues and overall make your metrics considerably more powerful. Instead of including fields directly in the key, these are instead added as tags to a more general base metric. The equivalent for the above hierarchical metrics becomes:

http.requests
	endpoint: getwidgets
	method: GET
	status: 200
	username: ryan
	region: rr

This kind of structure is supported out-of-the-box by Micrometer, whereby each tag (or dimension) is added as a parameter in each measurement. The connectors also understand how to send these metrics down to your monitoring system of choice, be it Prometheus which supports dimensional metrics, or just JMX (in which case the dimensional metrics will be flattened out).

The real benefit of dimensional metrics comes into play when partnered with a time series database (such as Prometheus). This allows for the ability to drill-down across dimensions at will, without introducing any of the downsides of hierarchical metrics. Going back to the example above, you can add specific predicates on any number of dimensions and get out the expected aggregation directly. Answering a complex query like “give me all the 500 responses from the getwidgets endpoint, across all regions and from users X,Y and Z from the last 24hrs and compare against the same data from yesterday” becomes trivial.

Spring Integration

As I mentioned before, Micrometer is integrated with Spring Boot Actuator and so requires just the single dependency to get running:

implementation 'org.springframework.boot:spring-boot-starter-actuator'

The Spring Boot Actuator starter dependency does a number of useful things which I’ll cover in future posts, but for now we’ll focus just on the metrics support. By default, Spring configures bindings to begin automatically publishing core metrics across many areas:

  • JVM - memory, buffer pools, thread utilization, classes loaded
  • CPU and File Descriptor usage
  • Logback - number of events logged at each level
  • Spring MVC - HTTP requests/responses and errors
  • HTTP Clients - instrumentation on RestTemplate or WebClient instances
  • Kafka - native metrics on all configured producers/consumers
  • Data Sources - HikariCP bindings for connection pool size and usage
  • Caches - Caffeine/JCache bindings for hit/miss counts, sizes

Hopefully you will agree that this is a fairly comprehensive list already for out-of-the-box behaviour (just dependant on which beans you have created in your app), and this is before you have created any of your own custom metrics.

Publishing Metrics

To try these features out we’ll create a quick contrived demo app to test some of the integration areas. First of all the main Spring app config:

@EnableCaching
@Configuration
public class AppConfig {

    @Value("${spring.application.name}")
    private String applicationName;

    @Bean
    public MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() {
        return registry -> registry.config().commonTags("region", "us-east-1", "application", applicationName);
    }

    @Bean
    public CacheManager cacheManager() {
        CaffeineCacheManager cacheManager = new CaffeineCacheManager();
        cacheManager.setCaffeine(Caffeine.newBuilder().recordStats());
        cacheManager.setCacheNames(Collections.singletonList("widgetCache"));
        return cacheManager;
    }

    @Bean
    public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder) {
        return restTemplateBuilder.build();
    }

}

There are a few things going on here:

  • we use a Spring MeterRegistryCustomizer bean to add a common set of tags to all metrics that get published. This is useful to attach instance level details, such as application name and region/datacentre that the service is running in.
  • we create a CacheManager bean using Caffeine as the underlying caching mechanism. The recordStats() portion instructs Caffeine to record metrics that Micrometer can then bind to (also make sure to @EnableCaching).
  • we create a standard RestTemplate bean, making sure to to use the RestTemplateBuilder so that the metrics instrumentation gets added.

Next is a simple @Service class to give something for our endpoints to call:

@Service
@RequiredArgsConstructor
public class WidgetService {

    private static final Logger log = LoggerFactory.getLogger(WidgetService.class);
    private final RestTemplate restTemplate;

    @Cacheable("widgetCache")
    public String getWidgets(String user) {
        log.info("Finding widgets for {}", user);
        if(user.equals("bill")) return "widget 1";
        if(user.equals("bob")) return "widget 2";
        else {
            log.error("Could not find widgets for {}", user);
            throw new IllegalArgumentException("Unknown user");
        }
    }

    public String saveWidget(String input) {
        ParameterizedTypeReference<Map<String, String>> responseType = new ParameterizedTypeReference<>() { };
        log.info("Saving widget with input: {}", input);
        ResponseEntity<Map<String, String>> responseEntity =
                restTemplate.exchange("https://httpbin.org/uuid?param={input}", HttpMethod.GET, null, responseType, input);
        log.info("Successfully saved widget!");
        return responseEntity.getBody().get("uuid");
    }
}

Nothing too interesting here. There is a simple getWidgets method to return some static data based on our user input (which also uses our new cache manager). There is a path to generate an exception so we can ensure the corresponding metrics are created. We also make use of the RestTemplate to call a test endpoint from httpbin.org to generate a a basic UUID (simulating some system latency).

Finally, there are two very simple endpoints that just call our service class (and generate some HTTP request metrics):

@RestController
@RequiredArgsConstructor
public class WidgetEndpoint {

    private final WidgetService widgetService;

    @GetMapping(value = "/widgets", produces = "text/plain")
    public ResponseEntity<String> getWidgets(@RequestParam("user") String user) {
        return ResponseEntity.ok(widgetService.getWidgets(user));
    }

    @PostMapping(value = "/widgets", produces = "text/plain")
    public ResponseEntity<String> saveWidget(@RequestParam("input") String input) {
        return ResponseEntity.ok(widgetService.saveWidget(input));
    }
}

If you build, run the app and hit the endpoints, they will work as expected, but we don’t yet have a way to confirm any metrics actually get created.

Spring Boot Actuator Metrics

The Spring Boot Actuator comes built-in with a number of endpoints we can use to manage our application. One of these allows you to view and query all the metrics generated by your application, but it’s disabled by default. Add the following to application.properties to enable it:

spring.application.name=widgetservice
management.endpoints.web.exposure.include=metrics

If you now visit http://localhost:8080/actuator/metrics in your browser, actuator will give you a list of all the top-level metrics (filtered below for brevity).

{
    "names": [
        "cache.gets",
        "cache.size",
        "http.server.requests",
        "jvm.buffer.memory.used",
        "jvm.classes.loaded",
        "jvm.gc.pause",
        "jvm.memory.max",
        "jvm.memory.used",
        "jvm.threads.live",
        "logback.events",
        "process.cpu.usage",
        "process.uptime",
        "system.cpu.usage",
        "tomcat.sessions.active.current",
        "tomcat.sessions.active.max"
    ]
}

Most of these should look familiar when compared with the list we saw before - they are metrics publishing automatically by our application without having to perform any real work. The actuator metrics endpoint also lets you view specific metrics: http://localhost:8080/actuator/metrics/jvm.memory.used

{
    "name": "jvm.memory.used",
    "description": "The amount of used memory",
    "baseUnit": "bytes",
    "measurements": [
        {
            "statistic": "VALUE",
            "value": 89722280
        }
    ],
    "availableTags": [
        {
            "tag": "area",
            "values": ["heap", "nonheap"]
        },
        {
            "tag": "application",
            "values": ["widgetservice"]
        },
        {
            "tag": "id",
            "values": [
                "G1 Old Gen",
                "CodeHeap 'non-profiled nmethods'",
                "G1 Survivor Space",
                "Compressed Class Space",
                "Metaspace",
                "G1 Eden Space",
                "CodeHeap 'non-nmethods'"
            ]
        },
        {
            "tag": "region",
            "values": ["us-east-1"]
        }
    ]
}

At the top we can see the aggregate measured value for total memory used and then we can also see a number of tags/dimensions associated with it (including the common set we added in the app config). This is where dimensional metrics get interesting, as you can now drill-down specifically into particular areas. For example, to view just heap memory usage: http://localhost:8080/actuator/metrics/jvm.memory.used?tag=area:heap

{
    "name": "jvm.memory.used",
    "description": "The amount of used memory",
    "baseUnit": "bytes",
    "measurements": [
        {
            "statistic": "VALUE",
            "value": 47795248
        }
    ],
    "availableTags": [
        {
            "tag": "id",
            "values": ["G1 Old Gen", "G1 Survivor Space", "G1 Eden Space"]
        }
    ]
}

This gives us a more specific measurement and a different set of tags that we could use to further drill-down into the metric e.g. http://localhost:8080/actuator/metrics/jvm.memory.used?tag=area:heap&tag=id:G1%20Eden%20Space to inspect the total memory used in the heap G1 Eden Space only.

Cache Metrics

Now to actually call some of our own code to generate some other metrics. We can use the getWidgets endpoint to make use of our cache: http://localhost:8080/widgets?user=bill. If we call that multiple times, we should be getting the result straight from the cache instead of calling the method itself. We can consult the cache metrics to confirm: http://localhost:8080/actuator/metrics/cache.gets?tag=cache:widgetCache&tag=result:hit

{
    "name": "cache.gets",
    "description": "The number of times cache lookup methods have returned a cached value.",
    "baseUnit": null,
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 3
        }
    ],
    "availableTags": [
        {
            "tag": "name",
            "values": ["widgetCache"]
        }
    ]
}

Here we are using the the cache.gets metric, with tag predicates on the cache name and result, in order to assess how well our cache is utilized. Similarly, you could also inspect the misses tag to generate the cache hit/miss ratio. You can also use the cache.size metric to observe trends in how many values are in loaded into your caches: http://localhost:8080/actuator/metrics/cache.size?tag=name:widgetCache

HTTP Client Metrics

Next up we can call the saveWidget endpoint to make use of our instrumented RestTemplate to call an external service: http://localhost:8080/widgets?input=something. If you call the /actuator/metrics endpoint again, you should see a new entry http.client.requests:

{
    "name": "http.client.requests",
    "description": "Timer of RestTemplate operation",
    "baseUnit": "seconds",
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 1
        },
        {
            "statistic": "TOTAL_TIME",
            "value": 0.6493909
        },
        {
            "statistic": "MAX",
            "value": 0.6493909
        }
    ],
    "availableTags": [
        {
            "tag": "method",
            "values": ["GET"]
        },
        {
            "tag": "clientName",
            "values": ["httpbin.org"]
        },
        {
            "tag": "uri",
            "values": ["/uuid?param={input}"]
        },
        {
            "tag": "outcome",
            "values": ["SUCCESS"]
        },
        {
            "tag": "status",
            "values": ["200"]
        }
    ]
}

The previous metrics we’ve looked at were simple counters or gauges, but now we have timer which gives us more measurement values. Not only do you get the count, but also the total time taken (allowing you to compute averages) and a max observed value. The tags are also more interesting here - similar to the above you can easily drill-down to answer some interesting questions:

  • how many requests where made to client X in total within a certain time period?
  • how many requests where made to the /uuid endpoint on client X which failed?
  • how long did the application spend waiting for client X to respond to our calls?

HTTP Request Metrics

By now we should have produced enough traffic to generate some good HTTP request metrics. You could also call the getWidgets endpoint with an unknown user to generate an exception: http://localhost:8080/widgets?user=none. The http.server.requests metric is used to capture measurements for our HTTP endpoints:

{
    "name": "http.server.requests",
    "description": null,
    "baseUnit": "seconds",
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 23
        },
        {
            "statistic": "TOTAL_TIME",
            "value": 0.7761524969999999
        },
        {
            "statistic": "MAX",
            "value": 0.0031665
        }
    ],
    "availableTags": [
        {
            "tag": "exception",
            "values": ["None", "IllegalArgumentException", "BadOperationRequestException"]
        },
        {
            "tag": "method",
            "values": ["GET", "POST"]
        },
        {
            "tag": "uri",
            "values": ["/saveWidget", "/getWidgets", "/actuator/metrics", "/**"]
        },
        {
            "tag": "outcome",
            "values": ["CLIENT_ERROR", "SUCCESS", "SERVER_ERROR"]
        },
        {
            "tag": "status",
            "values": ["404", "200", "400", "500"]
        }
    ]
}

Similarly to the http.client.requests metric, we have a timer meter and a number of useful tags to inspect:

  • how many requests resulted in 500 status codes within a certain time period?
  • how many times was a POST request made to the /widget resource?
  • how many requests made to the /widget resource resulted in IllegalArgumentException?
  • how long did it take to respond to all GET requests which resulted in 500 status codes?

Logging Metrics

A perhaps lesser known, but I think very useful metric is logback.events which keeps track of all lines that were logged and at what level. We can then query for example just for ERROR log lines across generated over time: http://localhost:8080/actuator/metrics/logback.events?tag=level:error:

{
    "name": "logback.events",
    "description": "Number of error level events that made it to the logs",
    "baseUnit": "events",
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 4
        }
    ]
}

DataSource Metrics

Finally, we have metrics produced by any JDBC DataSource instances available in our app. This can be done either through the Spring Boot spring.datasource auto-configuration, or creating your own. HikariCP exposes a number of metrics over it’s connection pool, such as hikaricp.connections.usage and hikaricp.connections.idle. This can be helpful if you’re trying to track down connection pool exhaustion issues in your applications.

Creating Custom Metrics

All the built-in Spring metrics are great, but you will also need to create and publish your own custom metrics. Spring creates a Micrometer MeterRegistry instance by default, that you can then inject and use to create custom counters, gauges and timers. As a very simple and contrived example, we can create some counters in our widget service:

private final MeterRegistry meterRegistry;

public String getWidgets(String user) {
    log.info("Finding widgets for {}", user);
    if(user.equals("bill")) {
        meterRegistry.counter("custom.widgets", "user", "bill").increment();
        return "widget 1";
    }
    if(user.equals("bob")) {
        meterRegistry.counter("custom.widgets", "user", "bob").increment();
        return "widget 2";
    }
    else {
        log.error("Could not find widgets for {}", user);
        throw new IllegalArgumentException("Unknown user");
    }
}

Here we use our injected MeterRegistry to create a new counter called widgets (this will be the base name of your metric). We also provide a custom tag for our metric so we can then filter based on username. Note that you generally don’t want to create tags where there can be too many possible values. In the example above, we know there can only ever be two, but with too many it can cause some resource usage issues.

Once the metric is published, you should be able to see it show up in actuator just like any other metric: http://localhost:8080/actuator/metrics/custom.widgets

{
    "name": "custom.widgets",
    "description": null,
    "baseUnit": null,
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 2
        }
    ],
    "availableTags": [
        {
            "tag": "user",
            "values": ["bob", "bill"]
        }
    ]
}

Since it’s a counter meter, we just have the single measurement alongside the custom user tag that we passed in. We could then drill-down as needed into how many widgets a specific user had asked for etc. (note in this case the max is always 2 since we have used the @Cacheable annotation).

Finally, in Spring Boot we can also create a custom timer surrounding a method call using the @Timed annotation. You also need to create an aspect bean for Micrometer to recognise and instrument the methods:

@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
    return new TimedAspect(registry);
}

@Timed
public String saveWidget(String input) {}

We can then view the timing data in the method.timed base metric, which can be filtered down by tags on class and method names: http://localhost:8080/actuator/metrics/method.timed?tag=method:saveWidget

{
    "name": "method.timed",
    "description": null,
    "baseUnit": "seconds",
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 3
        },
        {
            "statistic": "TOTAL_TIME",
            "value": 1.0327454
        },
        {
            "statistic": "MAX",
            "value": 0.600250599
        }
    ],
    "availableTags": [
        {
            "tag": "exception",
            "values": ["none"]
        },
        {
            "tag": "class",
            "values": ["com.example.demo.WidgetService"]
        }
    ]
}

This just scratches the surface of the Micrometer API to define your own metrics. Take a look at the docs page https://micrometer.io/docs/concepts which goes into more detail about each available meter type.

Aggregation and Visualization

See this follow-up post for a complete look at visualizing Spring Boot metrics: Aggregating and Visualizing Spring Boot Metrics with Prometheus and Grafana

Capturing and publishing metrics is one thing, but in a multi-instance environment we also need some way of aggregating all the measurements and allow us to visualize state across the whole platform. The built-in Spring Boot Actuator endpoint is useful as a quick admin tool, but generally we need to use the Micrometer connectors and other external monitoring tools to get the full benefits. There are a lot of options here, but Prometheus tends to be the go-to as a time series database and monitoring tool. For Spring based apps, you just need to add the appropriate Micrometer dependency and the rest is handled for you:

implementation 'io.micrometer:micrometer-registry-prometheus'

Actuator will then expose a dedicated /actuator/prometheus endpoint which can be called by Prometheus to gather the latest measurements (requires extra config not covered here). There are generally two ways to get your metrics into a time series database - the push model or the pull model. In the push model, your application itself has to send all your metrics downstream. This might result in slightly more real-time measurements, but also introduces issues if the monitoring tool becomes unavailable. Your app might be continuously blocking a thread trying to publish historical metrics downstream, even though it may be down or otherwise overwhelmed. Additionally, you also introduce unneeded coupling between your application and the external monitoring tooling. In the push model, which tools like Prometheus favour, your application exposes a simple and optimized endpoint for your metrics and then it’s the responsibility of the external tool to continuously poll for the latest measurements. Crucially, this minimises any negative impact to your application itself and avoids the coupling. It does however require a service discovery mechanism (such as Eureka or Zookeeper) to be configured in order for the tooling to pull metrics from each of the application instances.

Prometheus provides it’s own basic visualization tooling, but instead Grafana can be used as another layer in the stack to offer rich querying and graphing feature sets, alongside alerting and reporting. Prometheus still stores the raw data, but you can put together dashboards in Grafana, such as the one below for a generic Spring Boot app.

Grafana Spring Boot Dashboard

Takeaways (TL;DR)

  • metrics are an essential part of any application, not only in assessing health and stability, but also to make data-informed decisions on the future direction of your application
  • for JVM based apps, Micrometer is the way to go for metrics collection (especially when using Spring)
  • dimensional metrics are extremely powerful - and resolve longstanding issues with conventional hierarchical metrics
  • Spring Boot 2+ and Spring Actuator have great built-in support for metrics with Micrometer and out-of-the-box integration for a number of key areas
  • default application metrics should only form part of the overall picture and must be partnered with business level metrics - which are well-defined against established and agreed SLA’s
  • an external time series database such as Prometheus should be used to aggregate metrics across all application instances and allow tools such as Grafana to provide dynamic visualisations on top of the underlying data.

Useful links:

Read More

The Importance of Integration Testing Part 1 - HTTP Endpoints

Bert is a new joiner within the team. As his first task he’s been assigned to create a simple endpoint to expose some existing data onto one of the front-end screens. His manager says that this should give him a gentle introduction to the area whilst also giving opportunity to gain familiarity with the codebase and SDLC processes. She mentions that he should keep in regular contact with Brenda who will be handling the UI changes. Bert agrees, he’s already quite familiar with Spring Boot from his previous position - creating a simple new endpoint in an existing service should be straightforward he thinks to himself. It should look good if he can get this finished and signed off within a couple days.

Bert clones the repo and notices that the service doesn’t yet expose any API endpoints (Spring @RestController) so he quickly creates a new one, using the existing service pointed out to him by his teammate Ben (simplified below).

@RestController
@RequiredArgsConstructor
public class SomeResultEndpoint {

  private final SomeResultService someResultService;

  @GetMapping(value = "/someResult")
  public SomeResult getSomeResult() {
    return someResultService.getSomeResult();
  }
}

During a quick catch-up, Ben informs Bert about the team’s core shared library, which includes various components that should automatically handle all the other key requirements for him. This includes things like authentication, object mapping and error handling. Bert starts up the service locally, pointing to the dev database, hits the endpoint and can see data returns successfully. Content that everything looks to be working ok, Bert moves on to writing tests for the change. He knows from reading the team’s SDLC documentation that the pull request build will fail if it sees any drop in code coverage.

The Bad

Generally the first thing done in these situations is the trusty unit test cases - and that’s exactly what Bert does initially. In Java, tools like JUnit and Mockito (amongst many others) make this kind of change straightforward to unit test, just mock out the service dependency and ensure the controller behaves as expected. Bert comes up with the following simple test case:

class SomeResultEndpointTest {

  private SomeResultEndpoint someResultEndpoint;

  private SomeResultService someResultService;

  @BeforeEach
  void setUp() {
    someResultService = mock(SomeResultService.class);
    someResultEndpoint = new SomeResultEndpoint(someResultService);
  }

  @Test
  void getSomeResult() {
    SomeResult expected = TestUtils.getSomeResultData();
    when(someResultService.getSomeResult()).thenReturn(expected);
    SomeResult actual = someResultEndpoint.getSomeResult();
    assertThat(actual).isEqualTo(expected);
    verify(someResultService, times(1)).getSomeResult();
  }
}

Initially, Bert tried to construct some test SomeResult instances himself, but quickly realised that the data structure was complex and he was unfamiliar with what a real-world scenario would look like. Thankfully, Ben pointed him towards some existing helper methods, created alongside the original service, that looked to create some realistic data and populate most of the fields.

Bert ran the test suite, and as expected, everything passed successfully. But Bert had some issues - what about the negative scenarios? What about the HTTP status codes etc? All this was being handled by either Spring or the core shared library components. Bert created a simple negative test case, but then began to realise that there wasn’t really much more that he could add here:

@Test
void getSomeResult_propagates_exception() {
  when(someResultService.getSomeResult()).thenThrow(new RuntimeException("error");
  assertThrows(RuntimeException.class, () -> someResultEndpoint.getSomeResult());
}

Bert commits his change, pushes it and creates a pull request for the team to review. The build passes successfully and code coverage is 100% on Bert’s changes - great! The pull request gets approved, merged and deployed into the dev/qa environments. Bert pings Brenda (who is doing the UI changes) that the change is ready to begin integrating with.

The Ugly

The next day Bert gets a message from Brenda - “the service isn’t working properly”, she explains. “Every time I call it in QA I get a 500 error returned”. Bert quickly pulls up the logs and notices many exceptions being thrown from the endpoint - all of which seem to be related to Jackson when the response SomeResult is converted to JSON.

com.fasterxml.jackson.databind.JsonMappingException: Infinite recursion (StackOverflowError) (through reference chain: java.util.concurrent.ConcurrentHashMap[" "]->com.java.sample.OraganisationStructures$$EnhancerBySpringCGLIB$$99c7d84b["$$beanFactory"]->org.springframework.beans.factory.support.DefaultListableBeanFactory["forwardRef"]at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:706)
at at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
at at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
at at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:690)
at at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)

A quick search of the error indicated a circular reference issue when trying to serialize the SomeResult instance. Sure enough, the data structure contained self references. Ben explained that it was actually a linked-list type structure, useful for the existing processing but perhaps not ideal for an endpoint representation. “Why didn’t this issue come up when you were testing?”, asked Ben. Some investigation later, Bert found that the dev database had old data, preventing such instances from ever being created when he ran it locally. The test utility methods did create this scenario, but the tests themselves had failed to pick up the issue. Ben suggests, “I guess we have never tried converting that model to JSON before”. Bert quickly resolved the issue, pushed his changes and informed Brenda about the fix - noting that he didn’t have to change any of the tests as part of his change.

The next day Bert gets another message from Brenda - “I’m now seeing some strange behaviour when unauthorized users call the endpoint. I expect the response to conform to our standard error model, but I just get a wall of text and a 500 error instead of the 401 status code I expect”. Again Bert checks the logs, he sees new AuthorizationException stack traces coming from the shared library component which performs the authorization checks. This looks like expected behaviour, Bert ponders, but why doesn’t the response get mapped correctly? Ben points him towards the AuthExceptionMapper class in the shared library, which converts the exception to the common error model. After some debugging, Bert found that the mapper had not been correctly configured in the service. Ben explained, “that mapper is still quite new, I guess it never got added since that service hasn’t exposed any endpoints before”. Again, Bert quickly fixes the issue, pushes his changes and informs Brenda - again noting that he did not have to change any of his test cases as part of the fix.

The Good

After these fixes, the new endpoint works as expected and Brenda is able to integrate successfully, but Bert is quite rightly less than satisfied. Not only did an otherwise straightforward change take much longer than it should, even now he could not be confident that it works as expected in all scenarios, let alone 6 months down the line. Bert brings up the issue in the sprint retrospective, highlighting a number of areas that are not covered by the current test suites - even though the code coverage metrics might suggest they are:

  • JSON object marshalling - both in request and response bodies
  • URL formatting and HTTP request methods
  • usage of query and/or path parameters
  • any exception scenario requiring use of separate exception mappers or handlers (response format and HTTP status codes)
  • any additional functionality provided in controller advice, filters, converters etc.

The team agrees, clearly there is a need for automated integration tests for these endpoints in addition to the conventional unit tests. Thankfully, Spring comes with a number of built-in solutions - one being MockMvc, which the team end-up using. A typical problem for integration tests like this is the need to spin-up a full Spring container - for most apps that might mean queue listeners get created, calling out to other services during startup etc - not something that is ideal for a short-lived test suite. Bert suggests configuring the app in such a way that allows for a “test” instance to be started (without all the external dependencies etc.) - to make creation of a proper integration test suite much easier. But in the meantime, MockMvc has a nice way around it:

Using this annotation will disable full auto-configuration and instead apply only configuration relevant to MVC tests (i.e. @Controller, @ControllerAdvice, @JsonComponent, Converter/GenericConverter, Filter, WebMvcConfigurer and HandlerMethodArgumentResolver beans but not @Component, @Service or @Repository beans).

Bert explains that this in effect gives you a a cut-down Spring container which just creates the components required to support MVC/REST functionality. No need for the rest of your beans to be created, those can be mocked out as usual. Bert comes up with the following additional test cases for the original change:

@WebMvcTest(SomeResultEndpoint.class)
class SomeResultEndpointIntTest {

  @Autowired
  private MockMvc mockMvc;

  @MockBean
  private SomeResultService someResultService;

  @MockBean
  private AuthorizationService authorizationService;

  @Test
  void getSomeResult_succeeds() throws Exception {
    when(authorizationService.isAuthorized(anyString())).thenReturn(true);
    SomeResult expected = TestUtils.getSomeResultData();
    when(someResultService.getSomeResult()).thenReturn(expected);

    this.mockMvc.perform(get("/someResult"))
      .andExpect(status().isOk())
      .andExpect(content().string(equalTo(marshalObject(expected))));
  }

  @Test
  void getSomeResult_notfound() throws Exception {
    when(authorizationService.isAuthorized(anyString())).thenReturn(true);
    when(someResultService.getSomeResult()).thenReturn(null);

    mockMvc.perform(get("/someResult"))
      .andExpect(status().isNotFound());
  }

  @Test
  void getSomeResult_unauthorized() throws Exception {
    when(authorizationService.isAuthorized(anyString())).thenReturn(false);
    SomeResult expected = TestUtils.getSomeResultData();
    when(someResultService.getSomeResult()).thenReturn(expected);

    mockMvc.perform(get("/someResult"))
      .andExpect(status().isUnauthorized());
  }
}

The above are three very simple test cases, but crucially provide coverage in a number of key areas:

  • we have the correct URL and are able to respond to GET requests over HTTP
  • the response can be successfully serialized to the required response format and matches the expected output (JSON in this case, but it could be anything)
  • exceptions are handled as expected, the HTTP response status codes are correct

Bert highlights that Spring has a number of other methods of handling the above - @SpringBootTest if you want to really startup the full application (combine with @ActiveProfiles) alongside utils like TestRestTemplate, but the team agrees that even just the above is a vast improvement.

Takeaways (TL;DR)

The example above is somewhat contrived, but really these scenarios are not unrealistic at all. There can easily be large areas of your application that your tests don’t actually cover - likely parts that are deeply reliant on ‘magic’ from your framework of choice and/or rely on (at least part of) your application to be running in order to test. How does your code integrate with the framework or other libraries? Is your configuration correct? You need something more than unit tests for this kind of thing and you want to know about such issues as early as possible.

  • Unit tests are great, but not enough on their own
  • Code coverage metrics will lie to you
  • Making any change/fix that doesn’t require you to also modify a test is a red flag
  • We rely more and more on the ‘magic from the framework’, but how do we test it? How do we know we’ve configured it correctly?
  • Above the core business logic (unit cases), every HTTP endpoint should have automated tests covering URL descriptors, methods, object marshalling, exception handling etc.
  • Spring MockMvc is a neat way of producing integration tests for your endpoints, but is not the only solution, see
Read More

Automatically Update Dependencies with GitHub Dependabot

With the introduction of Actions, GitHub is quickly becoming the one-stop shop for all things CI. But one, perhaps less well-known feature, is dependabot which allows you to automatically keep all your dependencies up to date. Depending on which language/framework you are using, making sure all your libraries are on the latest versions can be tricky. Maven/Gradle have plugins which will notify you of new versions, but this is a decidedly manual process. Plus if you are unlucky enough to develop in JS land, then good luck attempting to keep your 400 npm dependencies updated at any reasonable recurrence.

Instead, GitHub Dependabot will automatically read your project build files (build.gradle, package.json, requirements.txt etc) and create new pull requests to update libraries to newer versions. If you have a GitHub Action configured to run your build on all PR’s, then you can also gain some reasonable level of confidence that such newer versions won’t break your project (of course depending on quality of your tests).

Updating Java Dependencies

Configuring Dependabot is as simple as adding a dependabot.yml file in the .github directory at the root level of your project (the same place any action workflow config files are also placed).

version: 2
updates:
    # Enable version updates for Gradle
    - package-ecosystem: "gradle"
      # Look for `build.gradle` in the `root` directory
      directory: "/"
      # Check for updates once daily
      schedule:
          interval: "daily"

The above example sets up the most simple of use cases, which will use the gradle package manager to search for a build.gradle file in the / directory of your project and attempt to update any libraries on a daily basis.

When a new library version is released, a new Pull Request will be opened on your project - in the below example for Kotlin Ktor:

Dependabot Pull Request

The great thing about Dependabot though is that these pull requests aren’t just notifications - it’s clever enough to actually modify the build files (build.gradle in this case to the newer version):

Dependabot Changes

If all looks good (and hopefully your build still passes), all you have to do is merge the PR and you are good to go.

Updating Python Dependencies

The config for Python is very much the same (and for a variety of other languages). In this case, the scheduled interval is set to weekly instead of daily to avoid too much PR noise for fast moving dependencies. It is also set to ignore all updates to flask libraries - useful if you are required to fix at an older level for some reason:

- package-ecosystem: "pip"
  # Look for `build.gradle` in the `root` directory
  directory: "/"
  # Check for updates once weekly
  schedule:
      interval: "weekly"
      ignore:
          # Ignore updates to packages that start 'aws'
          # Wildcards match zero or more arbitrary characters
          - dependency-name: "flask*"

Make sure that you have your build workflow configured to run on all pull requests to the master branch in order to run your build automatically. See this previous post on how to setup build actions.

name: Build

on:
    push:
        branches: [master]
    pull_request:
        branches: [master]

You will then see the standard “All checks have passed” message on the PR if your build still passes on the newer dependency version. Make sure that you have adequate tests before you hit merge without trying it yourself though - ideally decent integration tests which actually start up your application. Unit tests are generally not enough to verify this kind of thing.

Dependabot Build

Additional Commands

You can interact with Dependabot by leaving comments on the pull requests that it opens - if for example you want to rebase against newer changes you’ve committed since it was run, or if you want to ignore a certain major/minor release version:

  • @dependabot rebase will rebase this PR
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

More Info

GitHub Dependabot Docs

Above Examples

dependabot.yml

Read More

Kotlin & Java CI with Github Actions

If you have a Kotlin/Java project of any reasonable size, you probably want some kind of CI (Continuous Integration) process running alongside the main development workflow. Commonly this takes the form of something like:

  • running a small build on every pull request before merging
  • running a complete build on every change pushed to master (or any other branch) - including integration tests etc
  • automatically run deployment steps e.g to Heroku, AWS or Github Pages
  • ensure that your project builds and runs on a wide variety of devices e.g different JDK versions/OS’ - or really that it can build on a machine that isn’t your local box
  • in general your main branch contains a fully working version of your project
  • run static code analysis tools or linters
  • anything else that can be automated..

Previously, the most widespread tool for this is is probably TravisCI (which is free for open source usage). Now however, there is an alternative that’s built into Github itself - Github Actions. You can think of it as pretty much the same as other CI tools out there, but you get the added benefit of full integration with Github, so now everything can be in the same place!

Creating a a Gradle Build Action

Your repository should have a new tab called Actions which is your new portal for anything CI related. Once you click on the tab you will be able to create your first Action. By default, Github will suggest some common workflows relevant to your project (e.g if it’s a Node project run npm run build and npm test). These take the form of open source packages hosted within other repositories, but you can of course create your own custom actions taking the best bits from each.

Github Actions tab

Actions take the form of simple .yml files which describes the workflow and steps to execute. In our case, we want to build and test our Kotlin or Java project. This example will use Gradle, but Maven will also work just as well. The below configuration is all we need to build our repo:

name: Build

on:
    push:
        branches: [master]
    pull_request:
        branches: [master]

jobs:
    build:
        runs-on: ubuntu-latest

        steps:
            - uses: actions/[email protected]
            - name: Set up JDK 11
              uses: actions/[email protected]
              with:
                  java-version: 11
            - name: Grant execute permission for gradlew
              run: chmod +x gradlew
            - name: Build with Gradle
              run: ./gradlew build

Thankfully the YAML markup is pretty readable. In the above action we perform the following steps:

  • Instruct Github to execute this Action on any push to the master branch, or pull requests targeting master
  • Create a single job called build (you can have as many as you want within a single Action) which runs on an Ubuntu container. There are plenty of other options for which OS image you want to target (runs-on: windows-latest or runs-on: macos-latest). This is great to make sure your project will build and run on a range of different machines.
  • Perform a Git checkout of your repo in the new virtual environment. This step makes use of the uses statement which allows you to reference other packaged actions - in this case actions/checkout. This is where things start to get a lot more powerful as you can begin to publish and reuse workflows from the community
  • Setup a JDK using another action provided by Github. In this case we just use JDK 11, but you could run these steps with a range e.g 8 to 14 to ensure compatibility
  • Run a simple shell script to give permissions on the Gradle wrapper. Similarly you could run pretty much any shell scripts you need
  • Execute the Gradle wrapper script to perform a complete build and test of our project. Note that this is exactly what we would do if we were to do the same locally - nothing needs to change just because we need to run this in a CI environment.

That’s it to run a simple Gradle build for our Kotlin or Java project. Github will instruct you to commit the .yml file into the .gitub/workflows directory in the root of your repo so that it can be picked up properly.

Github Actions sample file

Running the CI Workflow

Because we just set up our Action to be run automatically on any PR or push to master, there is nothing else we need to do to start utilising our new continuous integration process. In the Actions tab you will see all builds of your project alongside all log output. You will get notified in the event that your build process fails by email.

Github Actions output

Caching Build Dependencies

If you run the above Action you will probably notice that it takes some time to execute. This is because it has to go out and download all of your JAR dependencies every time it runs. To speed this up, you can use a caching mechanism. After your workflow is executed successfully, the local Gradle package cache will be stored in Github to allow it to be restored on other subsequent runs.

steps:
    - uses: actions/[email protected]
    - name: Set up JDK 1.8
      uses: actions/[email protected]
      with:
          java-version: 1.8
    - name: Cache Gradle packages
      uses: actions/[email protected]
      with:
          path: ~/.gradle/caches
          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
          restore-keys: ${{ runner.os }}-gradle
    - name: Build with Gradle
      run: ./gradlew build

More information

This just touches the surface of what you can do with Github Actions (it is a CI solution after all), focusing specifically on Kotlin or Java projects using Gradle. There are of course an ever increasing number of other supported languages/tools being added (Node, Python, Go, .NET, Ruby), alongside a number of other nice use cases integrating into other aspects of Github:

  • Create Github releases automatically after successful builds
  • Mark issues and pull requests as stale if not updated recently
  • Automatically label new pull requests based upon predefined criteria
  • Run within Docker containers, Kubernates and AWS uploads
  • Static analysis and linting
  • Automatically publish build artifacts to Github Pages

See the below links for more info and how to find some of the more popular packages created by the community. There is probably already something covering your use case:

Read More