Containers metrics with Prometheus and Grafana

5 min readMay 7, 2020

When we use micro service architecture, we use lots of docker container. So we need to monitor all the containers metrics such as memory, I/O, cpu and etc.

1. What is Prometheus?

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud.
The main Prometheus server which scrapes and stores time series data.
Prometheus is designed for reliability, to be the system you go to during an outage to allow you to quickly diagnose problems.
Prometheus is using its own query language called PromQL.

2. What is Grafana?

Grafana is a multi-platform open source analytics and interactive visualization software.
It provides charts, graphs, and alerts for the web when connected to supported data sources.

3. Prometheus + Grafana

Grafana supports querying Prometheus.
The Grafana data source for Prometheus is included since Grafana 2.5.0.

4. Exporters

Prometheus exporter exposing container metrics as prometheus metrics.

Some of these exporters are maintained as part of the official Prometheus GitHub organization, those are marked as official, others are externally contributed and maintained.

5. CAdvisor

cAdvisor (Container Advisor) provides the resource usage and performance characteristics of their running containers such as memory, cpu and etc.

It is a running daemon that collects, aggregates, processes, and exports information about running containers.

The container-exporter requests a list of containers running on the host by talking to a container manager.

Earlier, We used prom/container-exporter to get container metrics. This is deprecated.

6. Setup the Environment

Here I am using Prometheus, Grafana and CAdvisor.

I am using rest api and postgres docker containers which I used earlier. If you need to use those containers please check this link.

Here, I am using docker compose to run all the docker containers. docker compose is a tool for defining and running multi-container Docker applications.

You can run a single cAdvisor to monitor the whole machine. So according to the below image, we must run one cadvisor container on every machine.

1. cadvisor

This is the docker compose file for cadvisor. 8080 is the default port of cadvisor.

docker-compose up

cAdvisor has exposed a web UI at its port(8080).

link- http://localhost:8082/

Important- Here I am using http://localhost:8082/ because I have added port forwarding for 8080.

when we got to http://<MACHINE_IP>:8082/metrics , we can see, cAdvisor is exposing some metrics.

2. Prometheus

In here, I am using prometheus.yml to add my own custom configurations. Using this config file, I have added targets of scrape_configs for cadvisor and default prometheus metrics.
scrape_interval is 5 second. So every 5 second, prometheus is going to scrape metrics.
According to below config, prometheus knows which metrics it is going to scrape.

i) cAdvisor target- <MACHINE_IP>:8082 (default port is 8080, port forwarding 8082)

ii) prometheus target- <MACHINE_IP>:9090

This is the docker compose file for prometheus. 9090 is the prometheus default port.

docker-compose up

This is the prometheus UI.

link- http://localhost:9090/

3. Grafana

This is the docker compose file for grafana. Here I have overridden admin password using GF_SECURITY_ADMIN_PASSWORD= pass. 3000 is the default grafana port.

link- http://localhost:3000/