Ryan Harrison My blog, portfolio and technology related ramblings

Prometheus Monitoring Guide Part 3 - Alerting


Alerting rules allow you to define alert conditions based on Prometheus expression language expressions and to send notifications about firing alerts to an external service. Whenever the alert expression results in one or more vector elements at a given point in time, the alert counts as active for these elements’ label sets.

Alerting rules are configured in Prometheus in the same way as recording rules:

# rules/alert_rules.yml
    - name: example
          # Alert for any instance that has a median request latency >1s.
          - alert: APIHighRequestLatency
            expr: api_http_request_latencies_second{quantile="0.5"} > 1
            for: 10m
                severity: page
                summary: "High request latency on {{ $labels.instance }}"
                description: "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)"
  • for - wait for a certain duration between first encountering a new expression output vector element and counting an alert as firing for this element
  • labels - specifying a set of additional labels to be attached to the alert
  • annotations = informational labels that can be used to store longer additional information such as alert descriptions or runbook links
    - "rules/alert_rules.yml"

Alerts can be monitored through the “Alerts” tab in the Prometheus dashboard (which ones are active, pending, firing etc)


  • another layer is needed to add summarization, notification rate limiting, silencing and alert dependencies on top of the simple alert definitions
  • Prometheus is configured to periodically send information about alert states to an Alertmanager instance, which then takes care of dispatching the right notifications
    • takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty
  • provided as a single Go binary from https://prometheus.io/download/ so can be executed directly
    • ./alertmanager - by default runs on port 9093
    • or with Docker docker run --name alertmanager -d -p 9093:9093 quay.io/prometheus/alertmanager
    • takes configuration from alertmanager.yml file in same directory
        - static_configs:
              - targets:
                    - "localhost:9093"

Sending Email Notifications

  • alertmanager.yml file defines routing tree defining how an alert should be managed. If no labels are matching, default root is used
    receiver: admin

    - name: admin
          - to: "example@gmail.com"
            from: "example@gmail.com"
            smarthost: smtp.gmail.com:587
            auth_username: "example@gmail.com"
            auth_password: "abcdefghijklmnop"

Routing Tree

  • Grouping categorizes alerts of similar nature into a single notification. This is especially useful during larger outages when many systems fail at once and hundreds to thousands of alerts may be firing simultaneously.
    • configure Alertmanager to group alerts by their cluster and alertname so it sends a single compact notification.
  • Inhibition is the concept of suppressing notifications for certain alerts if certain other alerts are already firing
  • Silences are a way to simply mute alerts for a given time. A silence is configured based on matchers, just like the routing tree. Incoming alerts are checked whether they match all the equality or regular expression matchers of an active silence. If they do, no notifications will be sent out for that alert. Configured through the UI. If time based, add condition to the underlying rule instead
  • By default each alert running through the routing tree will halt after matching against the first receiver at the same level - can use continue clause
    receiver: admin # root fallback
    group_wait: 2m # how long to wait for other alerts in a group to fire before notifying (after initial)
    group_interval: 10s # how long to wait before sending a notification about new alerts added to an already firing group
    repeat_interval: 30m # how long to wait before sending a notification again if it has already been sent
        - match_re:
              app_type: (linux|windows) # custom label specified in the rule definition file
          receiver: ss-admin # fallback receiver
          group_by: [severity] # group all alerts on a label to send compact notification
              - match:
                    app_type: linux # match on more specific label
                receiver: linux-teamlead # target more specific receiver
                routes: # nested routes on different labels
                    - match:
                          severity: critical
                      receiver: delivery-manager
                      continue: true
                    - match:
                          severity: warning
                      receiver: linux-teamlead

        - match_re:
              app_type: (python|go)
          receiver: pec-admin # fallback receiver
              - match:
                    app_type: python
                receiver: python-team-admin # fallback receiver
                    - match:
                          severity: critical
                      receiver: python-team-manager
                    - match:
                          severity: warning
                      receiver: python-team-lead

    - source_match:
          severity: "critical"
          severity: "warning" # mute warning alert if critical alert already raised in same app and category
      equal: ["app_type", "category"]

    - name: linux-team-lead
          - to: "example@gmail.com"

Checking Tree Syntax

To quickly check whether an alerting route treefile is syntactically correct without starting the AlertManager instance, you can use the amtool utility:

amtool check-config alertmanager.yml

Or https://prometheus.io/webtools/alerting/routing-tree-editor/ can be used to visualize a routing tree

Read More

Aggregating and Visualizing Spring Boot Metrics with Prometheus and Grafana

Note: this is a follow-up post covering the collection and visualization of Spring Boot metrics within distributed environments. Make sure to take a look at Gathering Metrics with Micrometer and Spring Boot Actuator which outlines using Micrometer to instrument your application with some of the built-in Spring Boot integrations and how to start defining and capturing custom metrics.

From the previous part we should now have a Spring Boot application that is capable of capturing a variety of dimensional metrics, but is of limited use since it only stores these values locally within it’s own Micrometer MetricsRegistry. We can use the built-in Spring actuator endpoints to perform simple queries on these metrics, but this alone is not meant as a complete monitoring tool. We don’t have any access to historical data and we need to query the instances directly - not something which is viable when running many (perhaps ephemeral) instances.

Prometheus and Grafana (also sometimes known as Promstack) is a popular and open source stack which aims to solve these problems and provide a complete platform for observing metrics in widely distributed systems. This includes tracking metrics over time, creating complex queries based on their dimensions/tags, aggregating across many instances (or even across the whole platform), raising alerts based on predefined criteria and thresholds, alongside the creation of complex visualizations and dashboards with Grafana.

Basic Architecture

Below is a quick diagram taken from a talk given by one of the Prometheus founders and gives a good summary view of how Prometheus/Grafana work together and integrate with your systems. I would definitely recommend watching for some more background: https://www.youtube.com/watch?v=5O1djJ13gRU

Prometheus Architecture


At the centre is Prometheus which provides the core backbone for collection and querying. It’s based on it’s own internal times series database and is optimized specially for consumption and reporting of metrics. It can be used to monitor your hosts, applications or really anything that can serialize metrics data into the format that it can then pull from.

Crucially, Prometheus supports the dimensional data model that attaches labels/tags to each captured metric. For more details see Part 1 which covers some of the advantages over conventional hierarchical metrics, but in short for example if you were capturing the total number of HTTP requests, you would attach labels to each time series value allowing you to then query and aggregate based on source process, status code, URL etc. The underlying time series database is very well optimized around labelled data sets, so is able to efficiently handle frequent changes in the series’ dimensions over time (e.g for pod names or other ephemeral labels).

Prometheus operates in a pull model, so unlike other solutions whereby your application has to push all of its metrics to a separate collector process itself, Prometheus is instead configured to periodically scrape each of your service instances for the latest meter snapshot. These are then saved into the time series database and become immediately available for querying. Depending on the scrape interval and underlying observability requirements, this can potentially give you close to real-time visibility into your apps.

Rather than having to hardcode host names, Prometheus can integrate directly into your service discovery mechanism of choice (somewhat limited at the moment focusing on cloud offerings, but there are ways around it) in order to maintain an updated view of your platform and which which instances to scrape from. At the application level the only requirement is an endpoint exposing your metrics in the textual serialization format understood by Prometheus. With this approach your app need not know about whatever monitoring tools are pulling metrics - helping to decouple it from your observability stack. You also don’t need to worry about applying collection backpressure and handling errors like may need to in the push model, if for example the monitoring tool starts to become overwhelmed.

Prometheus provides its own dynamic query language called PromQL which understands each major metric type: Counters, Gauges, Summaries and Histograms. This is the main entry point into the underlying time series data and allows you to perform a wide variety of operations on your metrics:

  • binary operations between time series
  • rates over counters - taking into account service restarts
  • filtering the time series by any provided dimension/label
  • aggregating summary stats to create cumulative histograms

These queries can be used to support graphs and dashboards, or also to create alerts in conjunction with the AlertManager component (if for example certain thresholds are breached by an aggregated query across instances).


Prometheus is bundled with it’s own simple UI for querying and charting its time series data, but doesn’t come close to the flexibility offered by Grafana which is probably the most well known visualization and dashboarding tool out there. Grafana has deep integrations with Prometheus, allowing you to configure it as a data source like you would any other upstream store (the default is even Prometheus now) and then write your own PromQL queries to generate great looking charts, graphs and dashboards.

Following on with the config-as-code approach, all of your Grafana dashboards can also be exported as JSON files which makes sharing considerably easier. For JVM/Spring based applications there are many community dashboards already available that you can begin utilizing immediately (or otherwise as a decent starting point for your own visualizations).

Micrometer / Prometheus Exporter

Since Prometheus operates in a pull model, we need our Spring Boot application to expose an HTTP endpoint serializing each of our service metrics. As discussed in the previous part, this is where Micrometer comes into it’s own. Since it acts as a metrics facade, we can simply plug in any number of exporters that provide all the necessary integrations - with zero changes needed to our actual application code. Micrometer will take care of serializing it’s local store (and maybe pushing) into whatever formats required - be it Prometheus, Azure Monitor, Datadog, Graphite, Wavefront etc.

For our basic Spring Boot application, the actuator endpoint does something similar to this, but it doesn’t offer the correct format for Prometheus to understand. Instead, we can add the below library provided by Micrometer to enable the required exporter functionality:


In other apps you would need to manually setup an endpoint integrating with your local MetricsRegistry, but in this case Spring Boot sees this library on the classpath and will automatically setup a new actuator endpoint for Prometheus. These are all disabled by default so we also need to expose it through a management property:


If you start the application, now you should be able to visit http://localhost:8080/actuator/prometheus to see the serialized snapshot:

Prometheus Exporter Snapshot

You should see each of the built-in metrics provided by Spring Boot (JVM memory usage, HTTP requests, connection pools, caches etc) alongside all of your custom metrics and tags. Taking one of these lines to examine further gives some insight into how Prometheus stores these metrics:

http_server_requests_seconds_count{application="nextservice",exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/info",} 1.0

In this case it’s the total number of HTTP requests serviced by our app - for now just one after I called one of the actuator endpoints. The serialized form of the counter consists of the metric name, a series of tags (representing the dimensions) and a 64-bit floating point value - all of which will be stored in the time series dataset after capture. Since Prometheus understands the dimensional nature of our metrics, we can use PromQL to perform filtering, aggregations and other calculations specific to these attributes and not the metric as a whole. The serialized form also includes metadata on the underlying meters, including a description which will be available later on as tooltips.

Setting Up Prometheus & Grafana

Prometheus is bundled as a single Go binary, so is very easy to deploy even as a standalone process, but for the purposes of this quick demo we will instead make use of the official Docker image. Refer to this post for more details about setting up and running Prometheus.

docker run -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

The above command will run a Prometheus instance exposed on port 9090, binding the configuration YAML file from our own working directory (change as needed).


    # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
    - job_name: "springboot"
      metrics_path: "/next/actuator/prometheus"
      scrape_interval: 5s
          - targets: ["localhost:8080"] # wherever our Spring Boot app is running

The Prometheus configuration file has many options, more information at https://prometheus.io/docs/prometheus/latest/configuration/configuration/. For now though we just setup a single scrape target which tells Prometheus where to pull our metrics from - in this case a single static host, but could also be your service discovery mechanism. We also point the path to our new actuator URL and set the scrape interval.

For Grafana we can do something very similar with Docker. Refer to this article for more details about installation. Here we run a basic Grafana instance exposed on port 3000:

docker run -p 3000:3000 grafana/grafana

Using Prometheus

Visit localhost:9090 in the browser to view the Prometheus dashboard. On the Status->Targets page you should see the single static host - indicating that Prometheus is able to connect successfully and is continuously pulling metrics from our Spring Boot app:

Prometheus Targets

The built-in dashboard also lets you test and run PromQL expressions, the results of which can be in tabular form or as a simple graph. To test this out we can use the HTTP request counter from before to produce a close to real-time view into the traffic being handled by our app (the below returns the per second average over the last 5 minutes):


Prometheus Rate Graph

If we plot this using the graph tab and generate some load on the application, you should see the rate start to increase over time. At the bottom you should also be able to see each of the tags attached to the time series - in this case the app name, instance, URI, status code, method etc - any of these can be used to further refine the PromQL query as needed. If for example we were only interested in the count of successful requests we could instead run:


Note that the Prometheus charts don’t update themselves automatically even though the underlying data has been updated, so you need to search again periodically (Grafana does however do this). Although basic, the built-in editor is useful to explore the underlying dataset and build queries before creating dashboards. Another popular example is mapping memory usage - below we can clearly see garbage collection happening, with memory usage broken down by area and pool type (any of which could be filtered on within the PromQL query):

Prometheus JVM Usage Graph

See this post on PromQL for a more in depth look at what it’s capable of: PromQL and Recording Rules

Using Grafana

The Prometheus graphs are useful for adhoc queries, but Grafana is significantly more powerful in terms of its capabilities for visualization and creation of summary dashboards. Visit localhost:3000 to access the instance we started alongside Prometheus. We first need to create a new datasource pointing to our Prometheus instance on the same host.

We could begin creating our own custom dashboards immediately, but one of the great things about Grafana is that there are many open source community dashboards that we can reuse in order to get going quickly. For example the below screenshot shows a general JVM application dashboard which displays many key indicators out-of-the-box. All of these are default JVM metrics that get measured and exported automatically by default within Spring Boot apps:

  • I/O rates, duration and errors
  • Memory usage broken down by each pool
  • Garbage collection and pause times
  • Thread count and states
  • Open file descriptors etc.

Grafana JVM Dashboard

Grafana dashboards can also be set to automatically refresh every few seconds, so if required you can get a close to real-time summary view of you application (depending on your scrape interval).

General JVM level measurements can be useful in some cases, but we can get significantly more value by inspecting some of the Spring Boot specific meters which integrate deeply into various aspects of our application:

  • HTTP request counts - by URL, method, exception etc
  • HTTP response times - latest measured or averaged over a wider time period, can also be split by URL, response codes
  • Database connection pool statistics - open connections, create/usage/acquire times
  • Tomcat statistics - active sessions, error count
  • Logback events - number of info/warn/error lines over time

Grafana Spring Dashboard

Finally, we can of course also create our own panels and dashboards based on the custom metrics added specifically within our business processes:

  • Custom timers with our own tags
  • Cache utilization taken from our instrumented Spring Boot cache manager - calculating using the hit/miss counts
  • Client request latency/counts - exported from all outbound calls made using RestTemplate/WebClient
  • Response time distribution and percentiles - uses a great feature of Prometheus/Grafana allowing as to display aggregated cumulative timing histograms

Since we have easy access to a variety of timing and exception data, we can also record breaches against predefined SLA's - in the example below visualizing all requests which have missed a 100ms threshold value. We could easily do the same for exceptions/errors, or even better utilize our custom metrics integrated into the functional areas:

Grafana Custom Dashboard

Bonus: Node Exporter and Elasticsearch

I mentioned above how an additional part of the Promstack is the Node Exporter This is a simple daemon process which exposes a variety of Prometheus metrics about the underlying host it runs on. By default this runs on port 9100, to begin scraping we just need an additional section in the Prometheus config:

- job_name: "node-exporter"
  scrape_interval: 10s
      - targets: ["localhost:9100"]

Again, there are a variety of community dashboards available which give an idea of some the metrics made available by the exporter:

Grafana Node Exporter Dashboard

If you are running an Elasticsearch cluster then you can also make use of the community driven exporter to expose a variety of Prometheus metrics. In much the same way we can add this to our Prometheus instance and create a monitoring dashboard:

Grafana Elasticsearch Dashboard

Takeaways (TL;DR)

  • Prometheus and Grafana offer a very powerful platform for consumption and visualization of Spring Boot metrics
  • Prometheus runs in a pull model meaning that it needs to maintain a view of the current running instances - this can be done through static targets or by integrating directly into service discovery. Each app instance must expose an HTTP endpoint serializing a snapshot view of all metrics into the Prometheus textual format
  • We can easily spin up Prometheus and Grafana with the official Docker images
  • Spring Boot applications using Micrometer can easily integrate with Prometheus by adding the appropriate exporter dependency - no direct code changes required
  • Prometheus provides it’s own dynamic query language PromQL which is built specifically for dimensional metrics. It allows you to perform aggregation and mathematical operations on time series datasets, alongside the ability to apply filters to metric dimensions for both high level and more focused queries
  • The AlertManager component can be used to systematically generate alerts based on PromQL queries and predefined thresholds
  • Grafana can sit on top of Prometheus to offer rich visualization and dashboarding capabilities - utilising the underlying time series database and PromQL to generate graphs/charts
Read More

WSL2 - Better Managing System Resources

WSL2 is great, but unfortunately after moderate usage it’s easy to get in a situation where it will eat up all of your disk space and use up to 50% of your total system memory. We’ll go over how to address these issues below.

Setting a WSL2 Memory Limit

By default the WSL2 will consume up to 50% of your total system memory (or 8GB whichever is lower). You can configure an upper limit for the WSL2 VM by creating a .wslconfig file in your home directory (C:\Users\<user>\.wslconfig).


Note that in this case the Linux VM will consume the entire amount regardless of actual usage by your apps, but it will at least prevent it growing beyond this limit.

Free Unused Memory

As described in this post the Linux kernel often uses available memory for it’s page cache unless its otherwise needed by a program running on the system. This is good for performance, but for WSL2 it can often mean the VM uses a lot more memory than it really needs (especially for file-intensive operations).

Whilst the WSL2 VM is running can you also run a simple command to drop the memory caches and free up some memory:

sudo sh -c \"echo 3 >'/proc/sys/vm/drop_caches' && swapoff -a && swapon -a && printf '\n%s\n' 'Ram-cache and Swap Cleared'\"

Compact the WSL2 Virtual Disk

If you copy some large files into WSL2 and then delete them, they will disappear from the filesystem but the underlying virtual disk may have still grown in size and the extra space will not be re-used. We can run a command to optimize/vacuum the virtual disk file to reclaim some space.

## Must be run in PowerShell as Administrator user
# Distro Examples:
#   CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc
#   CanonicalGroupLimited.Ubuntu20.04onWindows_79rhkp1fndgsc

cd C:\Users\user\AppData\Local\Packages<Replace-Eg-CanonicalGroupLimited\LocalState
wsl --shutdown
optimize-vhd -Path .\ext4.vhdx -Mode full



If you run a lot of Docker containers within WSL2m a lot of disk space may be used unnesessarily by old images. We can run the standard Docker command to remove any old/dangling images, networks and volumes:

docker system prune

Along the same lines as above as we can also compact the VM file that Docker Desktop creates to get some disk space back. You’ll want to open PowerShell as an admin and then run these commands:

# Close all WSL terminals and run this to fully shut down WSL.
wsl.exe --shutdown

# Replace <user> with your Windows user name. This is where Docker stores its VM file.
cd C:\Users\user\AppData\Local\Docker\wsl\data

# Compact the Docker Desktop WSL VM file
optimize-vhd -Path .\ext4.vhdx -Mode full

The above optimize-vhd command will only work on Windows 10 Pro. For folks on the Home edition there are some scripts with workarounds:

Read More

Prometheus Monitoring Guide Part 2 - PromQL and Recording Rules


Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets the user select and aggregate time series data in real time. The result of an expression can either be shown as a graph, viewed as tabular data in Prometheus’s expression browser, or consumed by external systems via the HTTP API.

Data Types

An expression or sub-expression can evaluate to one of four types:

  • Instant vector - a set of time series containing a single sample for each time series, all sharing the same timestamp (prometheus_http_requests_total)
  • Range vector - a set of time series containing a range of data points over time for each time series (prometheus_http_requests_total[5m])
  • Scalar - a simple numeric floating point value

Depending on the use-case (e.g. when graphing vs. displaying the output of an expression), only some of these types are legal as the result from a user-specified expression. For example, an expression that returns an instant vector is the only type that can be directly graphed.

Selectors and Matchers

In the simplest form, only a metric name is specified. This results in an instant vector containing elements for all time series that have this metric name:


It is possible to filter these time series further by appending a comma separated list of label matchers in curly braces ({}).

This example selects only those time series with the http_requests_total metric name that also have the job label set to prometheus and their group label set to canary:

  • = - select labels that are exactly equal to the provided string
  • != - select labels that are not equal to the provided string
  • =~ - select labels that regex-match the provided string
  • !~ - select labels that do not regex-match the provided string

Range vector literals work like instant vector literals, except that they select a range of samples back from the current instant. A time duration is appended in square brackets ([]) at the end of a vector selector to specify how far back in time values should be fetched for each resulting range vector element.

In this example, we select all the values we have recorded within the last 5 minutes for all time series that have the metric name http_requests_total and a job label set to prometheus:



Prometheus’s query language supports basic logical and arithmetic operators. For operations between two instant vectors, the matching behavior can be modified.

  • binary arithmetic operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs. (+, -, *, /, %, ^)
  • comparison operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs. By default they filter. Their behaviour can be modified by providing bool after the operator, which will return 0 or 1 for the value rather than filtering (==, !=, >, >=)
  • operations between vectors attempt to find a matching element in the right-hand side vector for each entry in the left-hand side.
    • when applying operators Prometheus attempts to find a matching element in both vectors by labels. Can ignore labels to get matches
    • method_code:http_errors:rate5m{code="500"} / ignoring(code) method:http_requests:rate5m
  • aggregation operators can be used to aggregate the elements of a single instant vector, resulting in a new vector of fewer elements with aggregated values: (sum, min, max, avg, count, topk, quantile)
    • if the metric http_requests_total had time series that fan out by application, instance, and group labels, we could calculate the total number of seen HTTP requests per application and group over all instances via: sum without (instance) (http_requests_total)
  • rate calculates per second increment over a time-period (takes in a range vector and outputs an instant vector)
  • https://prometheus.io/docs/prometheus/latest/querying/functions


Return all time series with the metric http_requests_total:


Return all time series with the metric http_requests_total and the given job and handler labels:

http_requests_total{job="apiserver", handler="/api/comments"}

Return a whole range of time (in this case 5 minutes) for the same vector, making it a range vector (not graphable):

http_requests_total{job="apiserver", handler="/api/comments"}[5m]

Return the 5-minute rate of the http_requests_total metric for the past 30 minutes, with a resolution of 1 minute:


Return sum of 5-minute rate over all instances by job name:

sum by (job) (

Return the unused memory in MiB for every instance:

If we have two different metrics with the same dimensional labels, we can apply binary operators to them and elements on both sides with the same label set will get matched and propagated to the output:

(instance_memory_limit_bytes - instance_memory_usage_bytes) / 1024 / 1024

The same expression, but summed by application, could be written like this:

sum by (app, proc) (
  instance_memory_limit_bytes - instance_memory_usage_bytes
) / 1024 / 1024

Return the top 3 CPU users grouped by application (app) and process type (proc):

topk(3, sum by (app, proc) (rate(instance_cpu_time_ns[5m])))

Return the count of the total number of running instances per application:

count by (app) (instance_cpu_time_ns)

Recording Rules

Prometheus supports two types of rules which may be configured and then evaluated at regular intervals: recording rules and alerting rules. To include rules in Prometheus, create a file containing the necessary rule statements and have Prometheus load the file via the rule_files field in the config.

  • recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series
  • querying the precomputed result will then often be much faster than executing the original expression every time it is needed
  • this is especially useful for dashboards which need to query the same expression repeatedly every time they refresh

Recording and alerting rules exist in a rule group. Rules within a group are run sequentially at a regular interval, with the same evaluation time. The names of recording rules must be valid metric names. The names of alerting rules must be valid label values.

Rule Definitions

  • Recording rules should be of the general form level:metric:operation
    • level = the aggregation level of the metric and labels of the rule output
    • metric = the metric name under evaluation
    • operation = list of operations applied to the metric under evaluation
# rules/myrules.yml
    - name: example # The name of the group. Must be unique within a file.
          - record: job:http_inprogress_requests:sum # The name of the time series to output to. Must be a valid metric name.
            # The PromQL expression to evaluate. Every evaluation cycle this is
            # evaluated at the current time, and the result recorded as a new set of
            # time series with the metric name as given by 'record'.
            expr: sum by (job) (http_inprogress_requests)

The rule file paths need to be added into the main Prometheus config to be executed periodically as defined by evaluation_interval

    - "rules/myrules.yml"

Checking Rule Syntax

To quickly check whether a rule file is syntactically correct without starting a Prometheus server, you can use Prometheus’s promtool command-line utility tool:

promtool check rules /path/to/example.rules.yml


Allows direct endpoints for querying instant/range queries, viewing targets, configuration etc

  • localhost:9090/api/v1/query?query=up
  • localhost:9090/api/v1/query?query=http_requests_total[1m]
  • localhost:9090/api/v1/targets?state=active / localhost:9090/api/v1/rules?type=alert


Read More

Prometheus Monitoring Guide Part 1 - Installation and Instrumentation

An open-source systems monitoring and alerting toolkit. Now a standalone open source project and maintained independently of any company. Analyse how your applications and infrastructure is performing based on the metrics they publish. Particularly suitable for large distributed systems with ephemeral instances.

  • metrics are stored in a multi-dimensional data model instead of hierarchical, where each measurement consists of a name and a number of key/value pairs
    • http.server.requests.count(uri="/endpoint", method="GET") 12 instead of http.server.requests.GET.endpoint=12
    • backed by it’s own custom time-series database built specifically for metrics
    • provides it’s own query language PromQL as a read-only and flexible query language for aggregation on time series data
  • no reliance on distributed storage; single server nodes are autonomous
  • time series collection happens via a pull model over HTTP
  • targets are discovered via service discovery or static configuration
  • multiple modes of graphing and dashboarding support
  • Alternative to Graphite, InfluxDB
  • Ecosystem provides a number of pre-built exporters that expose metrics ready for Prometheus to scrape


Prometheus Architecture

  • main Prometheus server consists of a time series database storing each of the captured measurements
    • alongside a scraper which pulls samples from external applications, hosts, platforms
    • and an HTTP server which allows operations to be performed on the tsdb (e.g querying by PromQL)
  • Prometheus is a single-instance component; all data is stored on local node storage
    • if you need to scale, recommendation is to spin up multiple separate Prometheus instances with different/replicated targets
  • operates in a pull model, whereby Prometheus is setup to periodically scrape the metrics from all target application instances
    • therefore has to know about the location of all active instances via service discovery
    • more easily tell if a target is down, can manually go to a target and inspect its health with a web browser
    • application itself has no knowledge of Prometheus apart from an endpoint exposing the latest metrics snapshot (/metrics)
  • the pull model can be problematic for short-lived/batch operations which may not be alive long enough to be scraped
    • Pushgateway component can be used as a middle-man - gets pushed metrics from jobs and forwards them to Prometheus
  • service discovery integration into Kubernetes, AWS, Azure to understand current landscape of all target nodes
  • after metrics are scraped and stored in tsdb, can be made available to users through web UI/Grafana/API
  • AlertManager component can be used with a set of rules querying the metrics to generate system alerts
    • performs de-duplication of alerts, throttling, silences etc
    • forwards to email, PagerDuty, Slack etc


  • provided as a single Go binary from https://prometheus.io/download/ so can be executed directly
    • ./prometheus
    • by default looks for prometheus.yml config file in the same directory (by default will only scrape Prometheus itself)
  • Prometheus Web UI available at http://localhost:9090
    • allows you to view current targets, configuration and run queries. For more complex visualizations use Grafana
  • or can be executed through Docker
    • docker run -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus


  • all Prometheus config provided in prometheus.yml file
  • Prometheus can reload its configuration at runtime. If the new configuration is not well-formed, the changes will not be applied
    • reload is triggered by sending a SIGHUP to the Prometheus process (kill -HUP <pid>)
    • or sending an HTTP POST request to the /-/reload endpoint when the --web.enable-lifecycle flag is enabled
global: # applied to all targets unless overridden
    scrape_interval: 15s # how often to scrape each target
    evaluation_interval: 15s # how often to evaluate predefined rules

scrape_configs: # a set of jobs denoting where to scrape metrics from
    - job_name: "prometheus" # a group of targets, also added as a label to each measurement
      metrics_path: "/metrics" # default
      static_configs: # a hardcoded host/port or could be service discovery
          - targets: ["localhost:9090"] # uses metrics path to pull metrics, by default http

Service Discovery

  • static_configs does not scale to more dynamic environments where instances are added/removed frequently
  • Prometheus can integrate with service discovery mechanisms to automatically update it’s view of of running instances
    • when new instances are added Prometheus will begin scraping, when lost from discovery the time series will also be removed
    • built-in integrations with Consul, Azure, AWS or file based if custom mechanism required
  • JSON/YAML file can be published by the platform specifying all targets to scrape from. Prometheus uses it to automatically update targets

Prometheus Service Discovery

- job_name: 'myapp'
    refresh_interval: 5m # default
    - files:
      - file_sd.yml

Where the published file file_sd.json contains all the current live targets and any attached labels:

        "targets": ["<host>"],
        "labels": {
            "<labelname>": "<labelvalue>"


Prometheus needs to know what to scrape, and that’s where service discovery and relabel_configs come in. Relabel configs allow you to select which targets you want scraped, and what the target labels will be. So if you want to say scrape this type of machine but not that one, use relabel_configs.

metric_relabel_configs by contrast are applied after the scrape has happened, but before the data is ingested by the storage system. So if there are some expensive metrics you want to drop, or labels coming from the scrape itself (e.g. from the /metrics page) that you want to manipulate that’s where metric_relabel_configs applies.

So as a simple rule of thumb: relabel_config happens before the scrape, metric_relabel_configs happens after the scrape

    - source_labels: [__meta_kubernetes_service_label_app]
      regex: nginx
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      regex: web
      action: keep
    - source_labels: [__meta_ec2_public_ip] # rewrite private ip to public from service discovery
      regex: "(.*)"
      replacement: "${1}:9100"
      target_label: __address__

You can perform the following action operations:

  • keep: Keep a matched target or series, drop all others
  • drop: Drop a matched target or series, keep all others
  • replace: Replace or rename a matched label with a new one defined by the target_label and replacement parameters
  • labelkeep: Match the regex against all label names, drop all labels that don’t match (ignores source_labels and applies to all label names)
  • labeldrop: Match the regex against all label names, drop all labels that match (ignores source_labels and applies to all label names)

metric_relabel_configs can be used to drop unnecessary time-series before ingestion:

- job_name: cadvisor
      - source_labels: [container_label_JenkinsId]
        regex: ".+"
        action: drop
      - source_labels: [__name__]
        regex: "(container_tasks_state|container_memory_failures_total)"
        action: drop



  • there are two ways in which application metrics can be exposed for Prometheus
    • use a client library directly in the application to create and expose a Prometheus endpoint (usually /metrics)
    • use an intermediate proxy exporter instance instrumenting the target application and converting to the Prometheus metrics format
# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"}    3 1395066363000

# Minimalistic line:
metric_without_timestamp_and_labels 12.47

# A histogram, which has a pretty complex representation in the text format:
# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="0.1"} 33444
http_request_duration_seconds_bucket{le="0.2"} 100392
http_request_duration_seconds_bucket{le="0.5"} 129389
http_request_duration_seconds_bucket{le="1"} 133988
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320

# Finally a summary, which has a complex representation, too:
# HELP rpc_duration_seconds A summary of the RPC duration in seconds.
# TYPE rpc_duration_seconds summary
rpc_duration_seconds{quantile="0.01"} 3102
rpc_duration_seconds{quantile="0.05"} 3272
rpc_duration_seconds{quantile="0.5"} 4773
rpc_duration_seconds{quantile="0.9"} 9001
rpc_duration_seconds{quantile="0.99"} 76656
rpc_duration_seconds_sum 1.7560473e+07
rpc_duration_seconds_count 2693

Conventions and Practices

  • metrics names should start with a letter can be followed with any number of letters, numbers and underscores
  • metrics must have unique names, client libraries should report an error if you try to register the same one twice
  • should have a suffix describing the unit, in plural form (e.g _bytes or _total)
  • should represent the same logical thing-being-measured across all label dimensions
  • every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values


There are a number of libraries and servers which help in exporting existing metrics from third-party systems as Prometheus metrics. This is useful for cases where it is not feasible to instrument a given system with Prometheus metrics directly (Linux kernel) as cannot modify source etc

  • https://prometheus.io/docs/instrumenting/exporters/
  • an exporter is a separate process dedicated entirely to pulling metrics from a target system and exposing them as Prometheus metrics
    • “proxy service” converting the target interface into one that can be scraped by Prometheus
  • Common exporters (some official):

    • Node Exporter - hardware and OS metrics exposed by Unix kernels, CPU load, memory, I/O, network
    • MySQL Expoter - database metrics, queries ran, timings, pool sizes
    • Blackbox Exporter - probing of endpoints over HTTP, DNS, TCP, ICMP
    • Kafka, Kafka Lag, Nginx, Postgres, Jenkins, AWS, Graphite, JMX
  • for cases which you have access to modify the application code, instrumentation must be added in order to add Prometheus metrics
    • can use existing application frameworks to expose default common metrics (Spring Actuator)
    • use the client libraries to add custom metrics to be exposed (Go, Java, Python, Ruby)
  • other metrics libraries offer a facade over the definition of metrics and allow pluggable Prometheus exporter to be added instead (Micrometer)
    • don’t have to use Prometheus client library directly for increased flexibility in overall monitoring solution

Metric Types

The client libraries offer four core metric types. These are currently only differentiated in the client libraries (to enable APIs tailored to the usage of the specific types) and in the wire protocol. The Prometheus server does not yet make use of the type information and flattens all data into untyped time series.


  • a cumulative metric that represents a single monotonically increasing counter whose value can only increase or be reset to zero on restart
  • for example, you can use a counter to represent the number of requests served, tasks completed, or errors
  • do not use a counter to expose a value that can decrease. For example, do not use a counter for the number of currently running processes; instead use a gauge


  • a metric that represents a single numerical value that can arbitrarily go up and down
  • gauges are typically used for measured values like temperatures or current memory usage, but also “counts” that can go up and down, like the number of concurrent requests


  • samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values
  • a histogram with a base metric name of <basename> exposes multiple time series during a scrape:

    • cumulative counters for the observation buckets, exposed as <basename>_bucket{le="<upper inclusive bound>"}
    • the total sum of all observed values, exposed as <basename>_sum
    • the count of events that have been observed, exposed as <basename>_count (identical to <basename>_bucket{le="+Inf"} above)
  • use the histogram_quantile() function to calculate quantiles from histograms or even aggregations of histograms across instances
  • when operating on buckets, remember that the histogram is cumulative


  • similar to a histogram, a summary samples observations (usually things like request durations and response sizes). While it also provides a total count of observations and a sum of all observed values, it calculates configurable quantiles over a sliding time window
  • a summary with a base metric name of <basename> exposes multiple time series during a scrape:
    • streaming φ-quantiles (0 ≤ φ ≤ 1) of observed events, exposed as <basename>{quantile="<φ>"}
    • the total sum of all observed values, exposed as <basename>_sum
    • the count of events that have been observed, exposed as <basename>_count
  • https://prometheus.io/docs/practices/histograms/
    • if you need to aggregate, choose histograms.
    • otherwise, choose a histogram if you have an idea of the range and distribution of values that will be observed. Choose a summary if you need an accurate quantile, no matter what the range and distribution of the values is.
Read More