Install prometheus under Ubuntu

Install from repository

sudo apt-get install -y prometheus
# optional
sudo apt-get install -y prometheus-node-exporter
sudo apt-get install -y prometheus-alertmanager
sudo apt-get install -y prometheus-pushgateway

Ansible installation

https://github.com/panticz/ansible/tree/master/roles/prometheus
- hosts: localhost
  roles:
    - prometheus

Login
http://SERVER_IP:9090/

Commands

avg_over_time(node_memory_MemAvailable[5m])/1024/1024

Repositroy
https://github.com/cloudalchemy/ansible-prometheus

Docker

docker run --name prometheus -d -p 127.0.0.1:9090:9090 quay.io/prometheus/prometheus

Ports
https://github.com/prometheus/prometheus/wiki/Default-port-allocations

API
http://SERVER_IP:9090/metrics

Query

node_disk_io_now{device="sda"}

Deploy node_exporter without installation

# https://prometheus.io/download/#node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz -qO /tmp/node_exporter.tar.gz
tar xzf /tmp/node_exporter.tar.gz -C /tmp/
mv /tmp/node_exporter-0.16.0.linux-amd64/node_exporter /usr/bin/prometheus-node-exporter
nohup /usr/bin/prometheus-node-exporter &
prometheus-node-exporter --version

node_exporter Ansible installation

https://github.com/panticz/ansible/tree/master/roles/prometheus-node-exporter
- hosts: localhost
  roles:
    - prometheus-node-exporter

Use SSH tunnel to access node_exporter metrics
https://mkezz.wordpress.com/2017/08/04/prometheus-in-docker-ssh-tunnel-to-target-nodes/

Alerting
http://prometheus.example.com:9093/

# /etc/prometheus/prometheus.yml
...
rule_files:
  - alert.rules.yml

# cat /etc/prometheus/alert.rules.yml
groups:
- name: node.rules
  rules:
  - alert: InstanceDown
    expr: up{job="node"} == 0
    for: 1m

# generate test alert
curl -H "Content-Type: application/json" -d '[{"labels":{"alertname":"TestAlert_1"}}]' localhost:9093/api/v1/alert

Exporters
https://prometheus.io/docs/instrumenting/exporters/

https://github.com/prometheus/alertmanager
https://www.robustperception.io/alerting-on-down-instances
https://pierrevincent.github.io/2017/12/prometheus-blog-series-part-5-alerting-rules/
https://en.fabernovel.com/insights/tech-en/alerting-in-prometheus-or-how-i-can-sleep-well-at-night
https://www.digitalocean.com/community/tutorials/how-to-use-alertmanager-and-blackbox-exporter-to-monitor-your-web-server-on-ubuntu-16-04

unsee
https://github.com/cloudflare/unsee

wget https://github.com/cloudflare/unsee/releases/download/v0.9.2/unsee-linux-amd64.tar.gz
tar xzf unsee-linux-amd64.tar.gz unsee-linux-amd64 
./unsee-linux-amd64  --alertmanager.uri http://localhost:9093

http://prometheus.example.com:8080/

Cleanup cache

rm -rf /var/lib/prometheus/metrics/01*

Get metrics output

curl ctl1-dev:9100/metrics | grep mem | grep -v '#'

Ports

9090
9100

Links
https://github.com/prometheus/pushgateway
https://prometheus.io/docs/introduction/overview/
https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04