Search Icon

Ryan Harrison My blog, portfolio and technology related ramblings

Prometheus Monitoring Guide Part 4 - Pushgateway and Blackbox Exporter


The pull approach doesn’t work for ephemeral jobs which don’t run for long enough for Prometheus to scrape them. Pushgateway is a metrics cache for service-level batch jobs. Used to handle the exposition of metrics that have ben pushed from batch/cron jobs. If a Pushgateway instance collecting metrics from many targets goes down, all metrics will be lost.

Prometheus Pushgateway

  • provided as a single Go binary from so can be executed directly
    • ./pushgateway - by default runs on port 9091
    • or with Docker docker run -d -p 9091:9091 prom/pushgateway
  • need to add Pushgateway as scrape target in Prometheus
- job_name: "pushgateway"
  honor_labels: true # instrumentation labels to override target labels
      - targets: ["localhost:9091"]

Blackbox Exporter

A probing exporter, allowing you to monitor network endpoints - upon probing it returns detailed metrics about the underlying requests.

  • for use when you have no knowledge of system internals, to measure response times, DNS resolution timing, check availability of endpoints etc

Prometheus Blackbox Exporter

  • provided as a single Go binary from so can be executed directly
    • ./blackbox_exporter - by default runs on port 9115
    • or with Docker docker run --rm -d -p 9115:9115 -v pwd:/config prom/blackbox-exporter:master --config.file=/config/blackbox.yml
  • to retrieve metrics in Prometheus, target the probe endpoint directly (which performs and measures the request)
  • modules used to perform the network request (as defined in the probe URL) are defined in the blackbox.yml config file (HTTP, DNS, SSH)

Perform HTTP Request and Find Content in Response Body

# blackbox.yml
    prober: http
        preferred_ip_protocol: ip4 # by default ipv6
            - "monitoring"

http://localhost:9115/probe? = 1

Perform TCP Probe


Perform DNS Probe

    prober: dns
        transport_protocol: "tcp"
        preferred_ip_protocol: ip4
        query_name: ""


Scraping to Prometheus

# prometheus.yml
    - job_name: "blackbox"
      metrics_path: /probe
          module: [http_2xx] # Look for a HTTP 200 response.
          - targets:
                - # Target to probe with http
          - source_labels: [__address__]
            target_label: __param_target # save current target address into temp param
          - source_labels: [__param_target]
            target_label: instance # move current address to instance label
          - target_label: __address__
            replacement: # redirect address