Ryan Harrison My blog, portfolio and technology related ramblings

Aggregating and Visualizing Spring Boot Metrics with Prometheus and Grafana

Note: this is a follow-up post covering the collection and visualization of Spring Boot metrics within distributed environments. Make sure to take a look at Gathering Metrics with Micrometer and Spring Boot Actuator which outlines using Micrometer to instrument your application with some of the built-in Spring Boot integrations and how to start defining and capturing custom metrics.

From the previous part we should now have a Spring Boot application that is capable of capturing a variety of dimensional metrics, but is of limited use since it only stores these values locally within it’s own Micrometer MetricsRegistry. We can use the built-in Spring actuator endpoints to perform simple queries on these metrics, but this alone is not meant as a complete monitoring tool. We don’t have any access to historical data and we need to query the instances directly - not something which is viable when running many (perhaps ephemeral) instances.

Prometheus and Grafana (also sometimes known as Promstack) is a popular and open source stack which aims to solve these problems and provide a complete platform for observing metrics in widely distributed systems. This includes tracking metrics over time, creating complex queries based on their dimensions/tags, aggregating across many instances (or even across the whole platform), raising alerts based on predefined criteria and thresholds, alongside the creation of complex visualizations and dashboards with Grafana.

Basic Architecture

Below is a quick diagram taken from a talk given by one of the Prometheus founders and gives a good summary view of how Prometheus/Grafana work together and integrate with your systems. I would definitely recommend watching for some more background: https://www.youtube.com/watch?v=5O1djJ13gRU

Prometheus Architecture

Prometheus

At the centre is Prometheus which provides the core backbone for collection and querying. It’s based on it’s own internal times series database and is optimized specially for consumption and reporting of metrics. It can be used to monitor your hosts, applications or really anything that can serialize metrics data into the format that it can then pull from.

Crucially, Prometheus supports the dimensional data model that attaches labels/tags to each captured metric. For more details see Part 1 which covers some of the advantages over conventional hierarchical metrics, but in short for example if you were capturing the total number of HTTP requests, you would attach labels to each time series value allowing you to then query and aggregate based on source process, status code, URL etc. The underlying time series database is very well optimized around labelled data sets, so is able to efficiently handle frequent changes in the series’ dimensions over time (e.g for pod names or other ephemeral labels).

Prometheus operates in a pull model, so unlike other solutions whereby your application has to push all of its metrics to a separate collector process itself, Prometheus is instead configured to periodically scrape each of your service instances for the latest meter snapshot. These are then saved into the time series database and become immediately available for querying. Depending on the scrape interval and underlying observability requirements, this can potentially give you close to real-time visibility into your apps.

Rather than having to hardcode host names, Prometheus can integrate directly into your service discovery mechanism of choice (somewhat limited at the moment focusing on cloud offerings, but there are ways around it) in order to maintain an updated view of your platform and which which instances to scrape from. At the application level the only requirement is an endpoint exposing your metrics in the textual serialization format understood by Prometheus. With this approach your app need not know about whatever monitoring tools are pulling metrics - helping to decouple it from your observability stack. You also don’t need to worry about applying collection backpressure and handling errors like may need to in the push model, if for example the monitoring tool starts to become overwhelmed.

Prometheus provides its own dynamic query language called PromQL which understands each major metric type: Counters, Gauges, Summaries and Histograms. This is the main entry point into the underlying time series data and allows you to perform a wide variety of operations on your metrics:

  • binary operations between time series
  • rates over counters - taking into account service restarts
  • filtering the time series by any provided dimension/label
  • aggregating summary stats to create cumulative histograms

These queries can be used to support graphs and dashboards, or also to create alerts in conjunction with the AlertManager component (if for example certain thresholds are breached by an aggregated query across instances).

Grafana

Prometheus is bundled with it’s own simple UI for querying and charting its time series data, but doesn’t come close to the flexibility offered by Grafana which is probably the most well known visualization and dashboarding tool out there. Grafana has deep integrations with Prometheus, allowing you to configure it as a data source like you would any other upstream store (the default is even Prometheus now) and then write your own PromQL queries to generate great looking charts, graphs and dashboards.

Following on with the config-as-code approach, all of your Grafana dashboards can also be exported as JSON files which makes sharing considerably easier. For JVM/Spring based applications there are many community dashboards already available that you can begin utilizing immediately (or otherwise as a decent starting point for your own visualizations).

Micrometer / Prometheus Exporter

Since Prometheus operates in a pull model, we need our Spring Boot application to expose an HTTP endpoint serializing each of our service metrics. As discussed in the previous part, this is where Micrometer comes into it’s own. Since it acts as a metrics facade, we can simply plug in any number of exporters that provide all the necessary integrations - with zero changes needed to our actual application code. Micrometer will take care of serializing it’s local store (and maybe pushing) into whatever formats required - be it Prometheus, Azure Monitor, Datadog, Graphite, Wavefront etc.

For our basic Spring Boot application, the actuator endpoint does something similar to this, but it doesn’t offer the correct format for Prometheus to understand. Instead, we can add the below library provided by Micrometer to enable the required exporter functionality:

implementation("io.micrometer:micrometer-registry-prometheus")

In other apps you would need to manually setup an endpoint integrating with your local MetricsRegistry, but in this case Spring Boot sees this library on the classpath and will automatically setup a new actuator endpoint for Prometheus. These are all disabled by default so we also need to expose it through a management property:

management.endpoints.web.exposure.include=metrics,prometheus

If you start the application, now you should be able to visit http://localhost:8080/actuator/prometheus to see the serialized snapshot:

Prometheus Exporter Snapshot

You should see each of the built-in metrics provided by Spring Boot (JVM memory usage, HTTP requests, connection pools, caches etc) alongside all of your custom metrics and tags. Taking one of these lines to examine further gives some insight into how Prometheus stores these metrics:

http_server_requests_seconds_count{application="nextservice",exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/info",} 1.0

In this case it’s the total number of HTTP requests serviced by our app - for now just one after I called one of the actuator endpoints. The serialized form of the counter consists of the metric name, a series of tags (representing the dimensions) and a 64-bit floating point value - all of which will be stored in the time series dataset after capture. Since Prometheus understands the dimensional nature of our metrics, we can use PromQL to perform filtering, aggregations and other calculations specific to these attributes and not the metric as a whole. The serialized form also includes metadata on the underlying meters, including a description which will be available later on as tooltips.

Setting Up Prometheus & Grafana

Prometheus is bundled as a single Go binary, so is very easy to deploy even as a standalone process, but for the purposes of this quick demo we will instead make use of the official Docker image. Refer to this post for more details about setting up and running Prometheus.

docker run -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

The above command will run a Prometheus instance exposed on port 9090, binding the configuration YAML file from our own working directory (change as needed).

prometheus.yml

scrape_configs:
    # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
    - job_name: "springboot"
      metrics_path: "/next/actuator/prometheus"
      scrape_interval: 5s
      static_configs:
          - targets: ["localhost:8080"] # wherever our Spring Boot app is running

The Prometheus configuration file has many options, more information at https://prometheus.io/docs/prometheus/latest/configuration/configuration/. For now though we just setup a single scrape target which tells Prometheus where to pull our metrics from - in this case a single static host, but could also be your service discovery mechanism. We also point the path to our new actuator URL and set the scrape interval.

For Grafana we can do something very similar with Docker. Refer to this article for more details about installation. Here we run a basic Grafana instance exposed on port 3000:

docker run -p 3000:3000 grafana/grafana

Using Prometheus

Visit localhost:9090 in the browser to view the Prometheus dashboard. On the Status->Targets page you should see the single static host - indicating that Prometheus is able to connect successfully and is continuously pulling metrics from our Spring Boot app:

Prometheus Targets

The built-in dashboard also lets you test and run PromQL expressions, the results of which can be in tabular form or as a simple graph. To test this out we can use the HTTP request counter from before to produce a close to real-time view into the traffic being handled by our app (the below returns the per second average over the last 5 minutes):

rate(http_server_requests_count[5m])

Prometheus Rate Graph

If we plot this using the graph tab and generate some load on the application, you should see the rate start to increase over time. At the bottom you should also be able to see each of the tags attached to the time series - in this case the app name, instance, URI, status code, method etc - any of these can be used to further refine the PromQL query as needed. If for example we were only interested in the count of successful requests we could instead run:

rate(http_server_requests_count{outcome="SUCCESS"}[5m])

Note that the Prometheus charts don’t update themselves automatically even though the underlying data has been updated, so you need to search again periodically (Grafana does however do this). Although basic, the built-in editor is useful to explore the underlying dataset and build queries before creating dashboards. Another popular example is mapping memory usage - below we can clearly see garbage collection happening, with memory usage broken down by area and pool type (any of which could be filtered on within the PromQL query):

Prometheus JVM Usage Graph

See this post on PromQL for a more in depth look at what it’s capable of: PromQL and Recording Rules

Using Grafana

The Prometheus graphs are useful for adhoc queries, but Grafana is significantly more powerful in terms of its capabilities for visualization and creation of summary dashboards. Visit localhost:3000 to access the instance we started alongside Prometheus. We first need to create a new datasource pointing to our Prometheus instance on the same host.

We could begin creating our own custom dashboards immediately, but one of the great things about Grafana is that there are many open source community dashboards that we can reuse in order to get going quickly. For example the below screenshot shows a general JVM application dashboard which displays many key indicators out-of-the-box. All of these are default JVM metrics that get measured and exported automatically by default within Spring Boot apps:

  • I/O rates, duration and errors
  • Memory usage broken down by each pool
  • Garbage collection and pause times
  • Thread count and states
  • Open file descriptors etc.

Grafana JVM Dashboard

Grafana dashboards can also be set to automatically refresh every few seconds, so if required you can get a close to real-time summary view of you application (depending on your scrape interval).

General JVM level measurements can be useful in some cases, but we can get significantly more value by inspecting some of the Spring Boot specific meters which integrate deeply into various aspects of our application:

  • HTTP request counts - by URL, method, exception etc
  • HTTP response times - latest measured or averaged over a wider time period, can also be split by URL, response codes
  • Database connection pool statistics - open connections, create/usage/acquire times
  • Tomcat statistics - active sessions, error count
  • Logback events - number of info/warn/error lines over time

Grafana Spring Dashboard

Finally, we can of course also create our own panels and dashboards based on the custom metrics added specifically within our business processes:

  • Custom timers with our own tags
  • Cache utilization taken from our instrumented Spring Boot cache manager - calculating using the hit/miss counts
  • Client request latency/counts - exported from all outbound calls made using RestTemplate/WebClient
  • Response time distribution and percentiles - uses a great feature of Prometheus/Grafana allowing as to display aggregated cumulative timing histograms

Since we have easy access to a variety of timing and exception data, we can also record breaches against predefined SLA's - in the example below visualizing all requests which have missed a 100ms threshold value. We could easily do the same for exceptions/errors, or even better utilize our custom metrics integrated into the functional areas:

Grafana Custom Dashboard

Bonus: Node Exporter and Elasticsearch

I mentioned above how an additional part of the Promstack is the Node Exporter This is a simple daemon process which exposes a variety of Prometheus metrics about the underlying host it runs on. By default this runs on port 9100, to begin scraping we just need an additional section in the Prometheus config:

- job_name: "node-exporter"
  scrape_interval: 10s
  static_configs:
      - targets: ["localhost:9100"]

Again, there are a variety of community dashboards available which give an idea of some the metrics made available by the exporter:

Grafana Node Exporter Dashboard

If you are running an Elasticsearch cluster then you can also make use of the community driven exporter to expose a variety of Prometheus metrics. In much the same way we can add this to our Prometheus instance and create a monitoring dashboard:

Grafana Elasticsearch Dashboard

Takeaways (TL;DR)

  • Prometheus and Grafana offer a very powerful platform for consumption and visualization of Spring Boot metrics
  • Prometheus runs in a pull model meaning that it needs to maintain a view of the current running instances - this can be done through static targets or by integrating directly into service discovery. Each app instance must expose an HTTP endpoint serializing a snapshot view of all metrics into the Prometheus textual format
  • We can easily spin up Prometheus and Grafana with the official Docker images
  • Spring Boot applications using Micrometer can easily integrate with Prometheus by adding the appropriate exporter dependency - no direct code changes required
  • Prometheus provides it’s own dynamic query language PromQL which is built specifically for dimensional metrics. It allows you to perform aggregation and mathematical operations on time series datasets, alongside the ability to apply filters to metric dimensions for both high level and more focused queries
  • The AlertManager component can be used to systematically generate alerts based on PromQL queries and predefined thresholds
  • Grafana can sit on top of Prometheus to offer rich visualization and dashboarding capabilities - utilising the underlying time series database and PromQL to generate graphs/charts
Read More

WSL2 - Better Managing System Resources

WSL2 is great, but unfortunately after moderate usage it’s easy to get in a situation where it will eat up all of your disk space and use up to 50% of your total system memory. We’ll go over how to address these issues below.

Setting a WSL2 Memory Limit

By default the WSL2 will consume up to 50% of your total system memory (or 8GB whichever is lower). You can configure an upper limit for the WSL2 VM by creating a .wslconfig file in your home directory (C:\Users\<user>\.wslconfig).

[wsl2]
memory=6GB
swap=0

Note that in this case the Linux VM will consume the entire amount regardless of actual usage by your apps, but it will at least prevent it growing beyond this limit.

Free Unused Memory

As described in this post the Linux kernel often uses available memory for it’s page cache unless its otherwise needed by a program running on the system. This is good for performance, but for WSL2 it can often mean the VM uses a lot more memory than it really needs (especially for file-intensive operations).

Whilst the WSL2 VM is running can you also run a simple command to drop the memory caches and free up some memory:

sudo sh -c \"echo 3 >'/proc/sys/vm/drop_caches' && swapoff -a && swapon -a && printf '\n%s\n' 'Ram-cache and Swap Cleared'\"

Compact the WSL2 Virtual Disk

If you copy some large files into WSL2 and then delete them, they will disappear from the filesystem but the underlying virtual disk may have still grown in size and the extra space will not be re-used. We can run a command to optimize/vacuum the virtual disk file to reclaim some space.

## Must be run in PowerShell as Administrator user
# Distro Examples:
#   CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc
#   CanonicalGroupLimited.Ubuntu20.04onWindows_79rhkp1fndgsc

cd C:\Users\user\AppData\Local\Packages<Replace-Eg-CanonicalGroupLimited\LocalState
wsl --shutdown
optimize-vhd -Path .\ext4.vhdx -Mode full

https://github.com/microsoft/WSL/issues/4699

Docker

If you run a lot of Docker containers within WSL2m a lot of disk space may be used unnesessarily by old images. We can run the standard Docker command to remove any old/dangling images, networks and volumes:

docker system prune

Along the same lines as above as we can also compact the VM file that Docker Desktop creates to get some disk space back. You’ll want to open PowerShell as an admin and then run these commands:

# Close all WSL terminals and run this to fully shut down WSL.
wsl.exe --shutdown

# Replace <user> with your Windows user name. This is where Docker stores its VM file.
cd C:\Users\user\AppData\Local\Docker\wsl\data

# Compact the Docker Desktop WSL VM file
optimize-vhd -Path .\ext4.vhdx -Mode full

The above optimize-vhd command will only work on Windows 10 Pro. For folks on the Home edition there are some scripts with workarounds:

Read More

Prometheus Monitoring Guide Part 2 - PromQL and Recording Rules

PromQL

Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets the user select and aggregate time series data in real time. The result of an expression can either be shown as a graph, viewed as tabular data in Prometheus’s expression browser, or consumed by external systems via the HTTP API.

Data Types

An expression or sub-expression can evaluate to one of four types:

  • Instant vector - a set of time series containing a single sample for each time series, all sharing the same timestamp (prometheus_http_requests_total)
  • Range vector - a set of time series containing a range of data points over time for each time series (prometheus_http_requests_total[5m])
  • Scalar - a simple numeric floating point value

Depending on the use-case (e.g. when graphing vs. displaying the output of an expression), only some of these types are legal as the result from a user-specified expression. For example, an expression that returns an instant vector is the only type that can be directly graphed.

Selectors and Matchers

In the simplest form, only a metric name is specified. This results in an instant vector containing elements for all time series that have this metric name:

http_requests_total

It is possible to filter these time series further by appending a comma separated list of label matchers in curly braces ({}).

This example selects only those time series with the http_requests_total metric name that also have the job label set to prometheus and their group label set to canary:

http_requests_total{job="prometheus",group="canary"}
  • = - select labels that are exactly equal to the provided string
  • != - select labels that are not equal to the provided string
  • =~ - select labels that regex-match the provided string
  • !~ - select labels that do not regex-match the provided string

Range vector literals work like instant vector literals, except that they select a range of samples back from the current instant. A time duration is appended in square brackets ([]) at the end of a vector selector to specify how far back in time values should be fetched for each resulting range vector element.

In this example, we select all the values we have recorded within the last 5 minutes for all time series that have the metric name http_requests_total and a job label set to prometheus:

http_requests_total{job="prometheus"}[5m]

Operators

Prometheus’s query language supports basic logical and arithmetic operators. For operations between two instant vectors, the matching behavior can be modified.

  • binary arithmetic operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs. (+, -, *, /, %, ^)
  • comparison operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs. By default they filter. Their behaviour can be modified by providing bool after the operator, which will return 0 or 1 for the value rather than filtering (==, !=, >, >=)
  • operations between vectors attempt to find a matching element in the right-hand side vector for each entry in the left-hand side.
    • when applying operators Prometheus attempts to find a matching element in both vectors by labels. Can ignore labels to get matches
    • method_code:http_errors:rate5m{code="500"} / ignoring(code) method:http_requests:rate5m
  • aggregation operators can be used to aggregate the elements of a single instant vector, resulting in a new vector of fewer elements with aggregated values: (sum, min, max, avg, count, topk, quantile)
    • if the metric http_requests_total had time series that fan out by application, instance, and group labels, we could calculate the total number of seen HTTP requests per application and group over all instances via: sum without (instance) (http_requests_total)
  • rate calculates per second increment over a time-period (takes in a range vector and outputs an instant vector)
  • https://prometheus.io/docs/prometheus/latest/querying/functions

Examples

Return all time series with the metric http_requests_total:

http_requests_total

Return all time series with the metric http_requests_total and the given job and handler labels:

http_requests_total{job="apiserver", handler="/api/comments"}

Return a whole range of time (in this case 5 minutes) for the same vector, making it a range vector (not graphable):

http_requests_total{job="apiserver", handler="/api/comments"}[5m]

Return the 5-minute rate of the http_requests_total metric for the past 30 minutes, with a resolution of 1 minute:

rate(http_requests_total[5m])[30m:1m]

Return sum of 5-minute rate over all instances by job name:

sum by (job) (
  rate(http_requests_total[5m])
)

Return the unused memory in MiB for every instance:

If we have two different metrics with the same dimensional labels, we can apply binary operators to them and elements on both sides with the same label set will get matched and propagated to the output:

(instance_memory_limit_bytes - instance_memory_usage_bytes) / 1024 / 1024

The same expression, but summed by application, could be written like this:

sum by (app, proc) (
  instance_memory_limit_bytes - instance_memory_usage_bytes
) / 1024 / 1024

Return the top 3 CPU users grouped by application (app) and process type (proc):

topk(3, sum by (app, proc) (rate(instance_cpu_time_ns[5m])))

Return the count of the total number of running instances per application:

count by (app) (instance_cpu_time_ns)

Recording Rules

Prometheus supports two types of rules which may be configured and then evaluated at regular intervals: recording rules and alerting rules. To include rules in Prometheus, create a file containing the necessary rule statements and have Prometheus load the file via the rule_files field in the config.

  • recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series
  • querying the precomputed result will then often be much faster than executing the original expression every time it is needed
  • this is especially useful for dashboards which need to query the same expression repeatedly every time they refresh

Recording and alerting rules exist in a rule group. Rules within a group are run sequentially at a regular interval, with the same evaluation time. The names of recording rules must be valid metric names. The names of alerting rules must be valid label values.

Rule Definitions

  • Recording rules should be of the general form level:metric:operation
    • level = the aggregation level of the metric and labels of the rule output
    • metric = the metric name under evaluation
    • operation = list of operations applied to the metric under evaluation
# rules/myrules.yml
groups:
    - name: example # The name of the group. Must be unique within a file.
      rules:
          - record: job:http_inprogress_requests:sum # The name of the time series to output to. Must be a valid metric name.
            # The PromQL expression to evaluate. Every evaluation cycle this is
            # evaluated at the current time, and the result recorded as a new set of
            # time series with the metric name as given by 'record'.
            expr: sum by (job) (http_inprogress_requests)

The rule file paths need to be added into the main Prometheus config to be executed periodically as defined by evaluation_interval

rule_files:
    - "rules/myrules.yml"

Checking Rule Syntax

To quickly check whether a rule file is syntactically correct without starting a Prometheus server, you can use Prometheus’s promtool command-line utility tool:

promtool check rules /path/to/example.rules.yml

HTTP API

Allows direct endpoints for querying instant/range queries, viewing targets, configuration etc

  • localhost:9090/api/v1/query?query=up
  • localhost:9090/api/v1/query?query=http_requests_total[1m]
  • localhost:9090/api/v1/targets?state=active / localhost:9090/api/v1/rules?type=alert

https://prometheus.io/docs/prometheus/latest/querying/api/

Read More

Prometheus Monitoring Guide Part 1 - Installation and Instrumentation

An open-source systems monitoring and alerting toolkit. Now a standalone open source project and maintained independently of any company. Analyse how your applications and infrastructure is performing based on the metrics they publish. Particularly suitable for large distributed systems with ephemeral instances.

  • metrics are stored in a multi-dimensional data model instead of hierarchical, where each measurement consists of a name and a number of key/value pairs
    • http.server.requests.count(uri="/endpoint", method="GET") 12 instead of http.server.requests.GET.endpoint=12
    • backed by it’s own custom time-series database built specifically for metrics
    • provides it’s own query language PromQL as a read-only and flexible query language for aggregation on time series data
  • no reliance on distributed storage; single server nodes are autonomous
  • time series collection happens via a pull model over HTTP
  • targets are discovered via service discovery or static configuration
  • multiple modes of graphing and dashboarding support
  • Alternative to Graphite, InfluxDB
  • Ecosystem provides a number of pre-built exporters that expose metrics ready for Prometheus to scrape

Architecture

Prometheus Architecture

  • main Prometheus server consists of a time series database storing each of the captured measurements
    • alongside a scraper which pulls samples from external applications, hosts, platforms
    • and an HTTP server which allows operations to be performed on the tsdb (e.g querying by PromQL)
  • Prometheus is a single-instance component; all data is stored on local node storage
    • if you need to scale, recommendation is to spin up multiple separate Prometheus instances with different/replicated targets
  • operates in a pull model, whereby Prometheus is setup to periodically scrape the metrics from all target application instances
    • therefore has to know about the location of all active instances via service discovery
    • more easily tell if a target is down, can manually go to a target and inspect its health with a web browser
    • application itself has no knowledge of Prometheus apart from an endpoint exposing the latest metrics snapshot (/metrics)
  • the pull model can be problematic for short-lived/batch operations which may not be alive long enough to be scraped
    • Pushgateway component can be used as a middle-man - gets pushed metrics from jobs and forwards them to Prometheus
  • service discovery integration into Kubernetes, AWS, Azure to understand current landscape of all target nodes
  • after metrics are scraped and stored in tsdb, can be made available to users through web UI/Grafana/API
  • AlertManager component can be used with a set of rules querying the metrics to generate system alerts
    • performs de-duplication of alerts, throttling, silences etc
    • forwards to email, PagerDuty, Slack etc

Installation

  • provided as a single Go binary from https://prometheus.io/download/ so can be executed directly
    • ./prometheus
    • by default looks for prometheus.yml config file in the same directory (by default will only scrape Prometheus itself)
  • Prometheus Web UI available at http://localhost:9090
    • allows you to view current targets, configuration and run queries. For more complex visualizations use Grafana
  • or can be executed through Docker
    • docker run -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Configuration

  • all Prometheus config provided in prometheus.yml file
  • Prometheus can reload its configuration at runtime. If the new configuration is not well-formed, the changes will not be applied
    • reload is triggered by sending a SIGHUP to the Prometheus process (kill -HUP <pid>)
    • or sending an HTTP POST request to the /-/reload endpoint when the --web.enable-lifecycle flag is enabled
global: # applied to all targets unless overridden
    scrape_interval: 15s # how often to scrape each target
    evaluation_interval: 15s # how often to evaluate predefined rules

scrape_configs: # a set of jobs denoting where to scrape metrics from
    - job_name: "prometheus" # a group of targets, also added as a label to each measurement
      metrics_path: "/metrics" # default
      static_configs: # a hardcoded host/port or could be service discovery
          - targets: ["localhost:9090"] # uses metrics path to pull metrics, by default http

Service Discovery

  • static_configs does not scale to more dynamic environments where instances are added/removed frequently
  • Prometheus can integrate with service discovery mechanisms to automatically update it’s view of of running instances
    • when new instances are added Prometheus will begin scraping, when lost from discovery the time series will also be removed
    • built-in integrations with Consul, Azure, AWS or file based if custom mechanism required
  • JSON/YAML file can be published by the platform specifying all targets to scrape from. Prometheus uses it to automatically update targets

Prometheus Service Discovery

- job_name: 'myapp'
  file_sd_configs:
    refresh_interval: 5m # default
    - files:
      - file_sd.yml

Where the published file file_sd.json contains all the current live targets and any attached labels:

[
    {
        "targets": ["<host>"],
        "labels": {
            "<labelname>": "<labelvalue>"
        }
    }
]

Relabelling

Prometheus needs to know what to scrape, and that’s where service discovery and relabel_configs come in. Relabel configs allow you to select which targets you want scraped, and what the target labels will be. So if you want to say scrape this type of machine but not that one, use relabel_configs.

metric_relabel_configs by contrast are applied after the scrape has happened, but before the data is ingested by the storage system. So if there are some expensive metrics you want to drop, or labels coming from the scrape itself (e.g. from the /metrics page) that you want to manipulate that’s where metric_relabel_configs applies.

So as a simple rule of thumb: relabel_config happens before the scrape, metric_relabel_configs happens after the scrape

relabel_configs:
    - source_labels: [__meta_kubernetes_service_label_app]
      regex: nginx
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      regex: web
      action: keep
    - source_labels: [__meta_ec2_public_ip] # rewrite private ip to public from service discovery
      regex: "(.*)"
      replacement: "${1}:9100"
      target_label: __address__

You can perform the following action operations:

  • keep: Keep a matched target or series, drop all others
  • drop: Drop a matched target or series, keep all others
  • replace: Replace or rename a matched label with a new one defined by the target_label and replacement parameters
  • labelkeep: Match the regex against all label names, drop all labels that don’t match (ignores source_labels and applies to all label names)
  • labeldrop: Match the regex against all label names, drop all labels that match (ignores source_labels and applies to all label names)

metric_relabel_configs can be used to drop unnecessary time-series before ingestion:

- job_name: cadvisor
  metric_relabel_configs:
      - source_labels: [container_label_JenkinsId]
        regex: ".+"
        action: drop
      - source_labels: [__name__]
        regex: "(container_tasks_state|container_memory_failures_total)"
        action: drop

https://grafana.com/docs/grafana-cloud/billing-and-usage/prometheus/usage-reduction/

Instrumentation

  • there are two ways in which application metrics can be exposed for Prometheus
    • use a client library directly in the application to create and expose a Prometheus endpoint (usually /metrics)
    • use an intermediate proxy exporter instance instrumenting the target application and converting to the Prometheus metrics format
# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"}    3 1395066363000

# Minimalistic line:
metric_without_timestamp_and_labels 12.47

# A histogram, which has a pretty complex representation in the text format:
# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="0.1"} 33444
http_request_duration_seconds_bucket{le="0.2"} 100392
http_request_duration_seconds_bucket{le="0.5"} 129389
http_request_duration_seconds_bucket{le="1"} 133988
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320

# Finally a summary, which has a complex representation, too:
# HELP rpc_duration_seconds A summary of the RPC duration in seconds.
# TYPE rpc_duration_seconds summary
rpc_duration_seconds{quantile="0.01"} 3102
rpc_duration_seconds{quantile="0.05"} 3272
rpc_duration_seconds{quantile="0.5"} 4773
rpc_duration_seconds{quantile="0.9"} 9001
rpc_duration_seconds{quantile="0.99"} 76656
rpc_duration_seconds_sum 1.7560473e+07
rpc_duration_seconds_count 2693

Conventions and Practices

  • metrics names should start with a letter can be followed with any number of letters, numbers and underscores
  • metrics must have unique names, client libraries should report an error if you try to register the same one twice
  • should have a suffix describing the unit, in plural form (e.g _bytes or _total)
  • should represent the same logical thing-being-measured across all label dimensions
  • every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values

Exporters

There are a number of libraries and servers which help in exporting existing metrics from third-party systems as Prometheus metrics. This is useful for cases where it is not feasible to instrument a given system with Prometheus metrics directly (Linux kernel) as cannot modify source etc

  • https://prometheus.io/docs/instrumenting/exporters/
  • an exporter is a separate process dedicated entirely to pulling metrics from a target system and exposing them as Prometheus metrics
    • “proxy service” converting the target interface into one that can be scraped by Prometheus
  • Common exporters (some official):

    • Node Exporter - hardware and OS metrics exposed by Unix kernels, CPU load, memory, I/O, network
    • MySQL Expoter - database metrics, queries ran, timings, pool sizes
    • Blackbox Exporter - probing of endpoints over HTTP, DNS, TCP, ICMP
    • Kafka, Kafka Lag, Nginx, Postgres, Jenkins, AWS, Graphite, JMX
  • for cases which you have access to modify the application code, instrumentation must be added in order to add Prometheus metrics
    • can use existing application frameworks to expose default common metrics (Spring Actuator)
    • use the client libraries to add custom metrics to be exposed (Go, Java, Python, Ruby)
  • other metrics libraries offer a facade over the definition of metrics and allow pluggable Prometheus exporter to be added instead (Micrometer)
    • don’t have to use Prometheus client library directly for increased flexibility in overall monitoring solution

Metric Types

The client libraries offer four core metric types. These are currently only differentiated in the client libraries (to enable APIs tailored to the usage of the specific types) and in the wire protocol. The Prometheus server does not yet make use of the type information and flattens all data into untyped time series.

Counter

  • a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart
  • for example, you can use a counter to represent the number of requests served, tasks completed, or errors
  • do not use a counter to expose a value that can decrease. For example, do not use a counter for the number of currently running processes; instead use a gauge

Gauge

  • a metric that represents a single numerical value that can arbitrarily go up and down
  • gauges are typically used for measured values like temperatures or current memory usage, but also “counts” that can go up and down, like the number of concurrent requests

Histogram

  • samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values
  • a histogram with a base metric name of <basename> exposes multiple time series during a scrape:

    • cumulative counters for the observation buckets, exposed as <basename>_bucket{le="<upper inclusive bound>"}
    • the total sum of all observed values, exposed as <basename>_sum
    • the count of events that have been observed, exposed as <basename>_count (identical to <basename>_bucket{le="+Inf"} above)
  • use the histogram_quantile() function to calculate quantiles from histograms or even aggregations of histograms across instances
  • when operating on buckets, remember that the histogram is cumulative

Summary

  • similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window
  • a summary with a base metric name of <basename> exposes multiple time series during a scrape:
    • streaming φ-quantiles (0 ≤ φ ≤ 1) of observed events, exposed as <basename>{quantile="<φ>"}
    • the total sum of all observed values, exposed as <basename>_sum
    • the count of events that have been observed, exposed as <basename>_count
  • https://prometheus.io/docs/practices/histograms/
    • if you need to aggregate, choose histograms.
    • otherwise, choose a histogram if you have an idea of the range and distribution of values that will be observed. Choose a summary if you need an accurate quantile, no matter what the range and distribution of the values is.
Read More

How to Dynamically Change Log Levels at Runtime with Spring Boot

As we can all probably agree, logging is a vital part of any application. It provides visibility into what our component is doing at any point in time and is an invaluable tool when trying to debug issues. When faced with a problem in production, likely one of the first things that will be done is to check the logs for any errors and to hopefully locate the source of the issue. It’s therefore vital that our logs are not only relevant and useful (perhaps a topic for another day), but are also visible and accessible when we need them the most. Hopefully we will be lucky and the logs will include all the necessary details to pinpoint the issue - or maybe they won’t. This may be decided by the level at which the loggers are configured.

By default, most if not all of our loggers will probably be set to INFO level. This is fine when everything is working correctly - including only the key operations/tasks and not creating too much noise as to overload the logging tools - but perhaps not so great when you need to work an issue. Although the developers may have included some basic INFO level log output within the problematic area, this may not include enough data to properly trace through an erroneous transaction. The same developer may have also added some additional DEBUG log lines (hopefully, if not they should do) that give us some additional details into what was being done, but these may not be visible to us when the application is deployed within an environment.

I bet most people reading this can remember that instance in which they were debugging an issue, were able to focus down on a particular segment of code, only to find that the log line that gave them that critical piece of information was set to DEBUG level and so was unavailable to them. We may have 6 different log levels available, but without finer control over their configuration at runtime, we may as well only have half that.

Spring Boot Logging Properties

Log configuration within Spring Boot applications takes a number of different forms these days, but in this post we’ll focus just on the main ones (excluding specific logback.xml modifications etc). A lot of the same configuration changes you would previously have made in those dedicated files can now also be set as simple application properties. For example, to change the global root log level to DEBUG, we can just add the following to application.properties (or YAML equivalent):

logging.level.root=DEBUG

This however has similar problems to the conventional logback.xml approach - to update these configuration settings you would have to rebuild and redeploy your entire application - not something that’s plausible in production. The difference here is that these all just standard Spring Boot properties, so in much the same way as you would elsewhere, you can set them as environment variables or pass them in as JVM options:

-Dlogging.level.root=DEBUG

This gets us somewhat closer to where we want to be - we can now control the log level without redeploying - but it also introduces another problem. This enables DEBUG log output for the entire application. If you’ve ever done this before, even in a basic Spring application, you will know that the output is extremely noisy:

  • At default INFO level a simple Spring Boot app generates 14 lines of log output at startup
  • At DEBUG level the same application generates over 2440 log lines - just at startup!
DEBUG 6852 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'org.springframework.boot.autoconfigure.web.servlet.MultipartAutoConfiguration'
DEBUG 6852 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Creating shared instance of singleton bean 'spring.servlet.multipart-org.springframework.boot.autoconfigure.web.servlet.MultipartProperties'
2021-01-30 17:36:25.708 DEBUG 6852 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Autowiring by type from bean name 'org.springframework.boot.autoconfigure.web.servlet.MultipartAutoConfiguration' via constructor to bean named 'spring.servlet.multipart-org.springframework.boot.autoconfigure.web.servlet.MultipartProperties'
DEBUG 6852 --- [1)-192.168.1.22] sun.rmi.transport.tcp                    : RMI TCP Connection(1)-192.168.1.22: (port 65235) op = 80
DEBUG 6852 --- [1)-192.168.1.22] javax.management.remote.rmi              : [javax.management.remote.rmi.RMIConnectionImpl@3440316d: connectionId=rmi://192.168.1.22  1] closing.
DEBUG 6852 --- [1)-192.168.1.22] javax.management.remote.rmi              : [javax.management.remote.rmi.RMIConnectionImpl@3440316d: connectionId=rmi://192.168.1.22  1] closed.
DEBUG 6852 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Autowiring by type from bean name 'errorPageCustomizer' via factory method to bean named 'dispatcherServletRegistration'
DEBUG 6852 --- [           main] o.apache.tomcat.util.IntrospectionUtils  : IntrospectionUtils: setProperty(class org.apache.coyote.http11.Http11NioProtocol port=8080)

If you look at the some of the output above you will notice that most of it is just noise and won’t help you resolve any real issues. In fact, it may hinder you when trying to search through all the volume just to find those few useful lines.

In reality setting the root log level to DEBUG is not something you really want to do unless you feel some need to closely inspect the Spring startup sequence. It belongs at the default INFO level where the frameworks and other libraries won’t overwhelm you with data. What you actually probably wanted was to only set the new log level for certain targeted areas of your application - be it a set of packages or even a particular class in much the way you can in logback.xml. Spring Boot also allows you to use the same property syntax for this. For example, to enable DEBUG output only in our service packages or specifically within our WidgetService class, we can instead add the following to application.properties:

logging.level.com.example.demo.service=DEBUG
logging.level.com.example.demo.service.WidgetService=DEBUG

Checking the log output again, you should see most of the noise from before disappear - leaving only our targeted package at the lower log level. Much more useful! You might notice that this really just has the same effect as updating the logback.xml file directly, and you would be correct. The big difference here however is that when combined with the VM argument trick from before, you now have much fine-grained control over your loggers after your application is deployed.

INFO 8768  --- [nio-8080-exec-1] com.example.demo.service.WidgetService   : Finding widgets for bob
DEBUG 8768 --- [nio-8080-exec-1] com.example.demo.service.WidgetService   : Background working happening to find widgets..
DEBUG 8768 --- [nio-8080-exec-1] com.example.demo.service.WidgetService   : Found user bill, returning widget 2

This is another improvement, but still has one big problem - you need to restart your application for the changes to be applied. This might be ok for some cases, but for example if the underlying issue was due some particular bad state of a local cache, restarting would reset the component, hide the underlying cause and make it much harder to reproduce further. Depending on the issue it may be possible to get your application back into the same state again after rehydrating caches etc, but sometimes “turn it off and on again” just hides the larger underlying problem.

As a last improvement, it would be great if you could dynamically change the log levels for particular targeted areas at runtime - without having to rebuild, redeploy or restart.

Spring Boot Actuator

Actuator, amongst other things, allows you to do just this through one of it’s administration endpoints. To enable this functionality we first need to add the starter dependency to the build.gradle file:

implementation 'org.springframework.boot:spring-boot-starter-actuator'

By default actuator doesn’t expose any of it’s admin endpoints, so we also need to explicitly enable the loggers feature(and others if you need to). Add the following into application.properties:

management.endpoints.web.exposure.include=loggers

If we now visit http://localhost:8080/actuator in the browser (or as GET request from elsewhere) you should see just the logger endpoints are now enabled:

{
    "_links": {
        "self": {
            "href": "http://localhost:8080/actuator",
            "templated": false
        },
        "loggers": {
            "href": "http://localhost:8080/actuator/loggers",
            "templated": false
        },
        "loggers-name": {
            "href": "http://localhost:8080/actuator/loggers/{name}",
            "templated": true
        }
    }
}

Actuator hooks into the log system used within your app (be it Logback or Log4j) and allows you to interact with these endpoints as an API to query and modify the underlying log levels.

Viewing Current Log Levels

First of all visit http://localhost:8080/actuator/loggers (via GET request) to see all the loggers configured within your application and their current levels (it will likely be quite a large list):

{
    "levels": ["OFF", "ERROR", "WARN", "INFO", "DEBUG", "TRACE"],
    "loggers": {
        "ROOT": {
            "configuredLevel": "INFO",
            "effectiveLevel": "INFO"
        },
        "com.example.demo.DemoApplication": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        },
        "com.example.demo.service": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        },
        "com.example.demo.service.WidgetService": {
            "configuredLevel": null,
            "effectiveLevel": "INFO"
        }
    }
}

In the above extract of the output we can see all the available levels, the current level for each area of our application and finally whether or not a custom level has been applied. These are all null for now since we are yet to override anything. This is not too helpful, but does help in identifying the logger names that we can later customize.

If you want to explicitly target a specific logger, you can add the name to the path (as described in the first actuator output). For example http://localhost:8080/actuator/loggers/com.example.demo.service will return:

{
    "configuredLevel": null,
    "effectiveLevel": "INFO"
}

Modifying Log Levels

In the above examples we have been using simple GET requests to query the current log configuration. The /actuator/loggers/{name} endpoint however also lets you send a POST request that allows you to update the configured level for a particular logger. For example, to change our service loggers to DEBUG level, send a POST request to http://localhost:8080/actuator/loggers/com.example.demo.service with the JSON body:

{
    "configuredLevel": "DEBUG"
}

The corresponding cURL command would be (note the Content-Type header needs to be set as the payload is a JSON object):

curl -i -X POST -H 'Content-Type: application/json' -d '{"configuredLevel": "DEBUG"}'
  http://localhost:8080/actuator/loggers/com.example.demo.service
  HTTP/1.1 204

If successful, the API will return a 204 No Content response. Checking the application logs again after some additional calls were made to the service class, you should see the same DEBUG log output as before, whilst all other output remains at the default INFO level:

INFO 10848  --- [nio-8080-exec-2] com.example.demo.service.WidgetService   : Finding widgets for bob
DEBUG 10848 --- [nio-8080-exec-2] com.example.demo.service.WidgetService   : Background working happening to find widgets..
DEBUG 10848 --- [nio-8080-exec-2] com.example.demo.service.WidgetService   : Found user bill, returning widget 2

To confirm the update, we can also try querying actuator again with the same logger name to view the updated configuration - GET http://localhost:8080/actuator/loggers/com.example.demo.service:

{
    "configuredLevel": "DEBUG",
    "effectiveLevel": "DEBUG"
}

Pretty cool! This gives you a lot of flexibility at runtime to better utilize your logs at different levels to debug and resolve issues. If you so wanted to, you can also target the the ROOT logger at http://localhost:8080/actuator/loggers/ROOT, but of course be aware of the potential noise.

Takeaways (TL;DR)

  • Logs are a vital tool when debugging issues, but only if you can see the right lines when you need them. These might not be at INFO level.
  • Developers should be using the various log levels TRACE, DEBUG, INFO, ERROR accordingly to add additional data points to your log output. In general volume should increase as the level decreases, but more detailed data points will be included.
  • The root logger should be kept at INFO level. Turning on DEBUG logs for our entire application will generate too much noise and will overwhelm both us and our log tooling.
  • Use Spring Boot properties to set specific log levels for particular packages/classes. Pass these in as runtime JVM options for greater flexibility. Note that you will have to restart the app for them to take effect.
  • Spring Boot Actuator gives the most fine-grained control - allowing you both query and update log levels at runtime through it’s admin endpoints.

Useful links:

Read More