monitoring

Telegraf

Agent for collecting, processing and sending metrics.

nginx

Add an endpoint on localhost to get nginx metrics

nginx/status.conf

server {
    listen 127.0.0.1:8080;  # localhost only
    server_name _;

    location /status {
        stub_status on;
        allow <IP>;
        deny all;
    }
}

# nginx -t                  # check config
# systemctl reload nginx

uwsgi

For emperor (emperor is very secretive, very few info):

/etc/uwsgi-emperor/emperor.ini

stats = 127.0.0.1:9000
stats-http = false      # prefer tcp socket than http

For each app:

/etc/uwsgi-emperor/vassals/.ini

stats = 127.0.0.1:900n  # n being incremented for each app
stats-http = false      # prefer tcp socket than http

Test that socket is exposing stats:

# apt install netcat-openbsd
$ echo | nc 127.0.0.1 9000 | jq .

install & config

# wget -q -O - https://repos.influxdata.com/influxdata-archive_compat.key | gpg --dearmor > /etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg
# echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg] https://repos.influxdata.com/debian stable main' > /etc/apt/sources.list.d/influxdata.list

/etc/telegraf/telegraf.conf

[[inputs.nginx]]
    urls = ["http://localhost:8080/status"]
    response_timeout = "5s"
[[inputs.tail]]
    # add telegraf to adm group:
    # usermod -aG adm telegraf
    name_override = "nginxlog"
    files = ["/var/log/nginx/access.log"]
    from_beginning = true
    pipe = false
    data_format = "grok"
    grok_patterns = ["%{COMBINED_LOG_FORMAT}"]

[[inputs.uwsgi]]
    servers = ["http://localhost:9000"]
    tags = { app = "<APP_NAME" }
[[outputs.prometheus_client]]
    # default port is 9273
    listen = ":9234"
    metric_version = 2

By default, telegraf adds the following tags: host, instance, source

check

Prometheus

Timeseries database and monitoring (alerts).

# apt install prometheus
# systemctl status prometheus
# systemctl status prometheus-node-exporter

Note Exporter is installed by default with Prometheus (see exporters)

exporters

To make data available for pulling, one solution is to install exporters for each application.
By default, prometheus-node-exporter is installed and exporting system (hardware + kernel) metrics on localhost:9100.
For applications, custom exporters might be available; e.g. nginx would export its metrics with:

$ wget https://github.com/nginxinc/nginx-prometheus-exporter/releases/download/

Preferred option is to use Telegraf to gather metrics directly from applications to monitor, and then make it available on a unique endpoint for pulling. With Telegraf, even if Note Exporter is a lightweight solution for OS metrics (only), and more specialized on OS metrics than Telegraf, it is usually not needed.

check

Prometheus exposes a web console http://localhost:9090.
Status | Target shows datasources status.

Grafana

Data visualization and dashboarding platform.

install & config

# wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor > /etc/apt/trusted.gpg.d/grafana.gpg
# echo 'deb [signed-by=/etc/apt/trusted.gpg.d/grafana.gpg] https://apt.grafana.com stable main' > /etc/apt/sources.list.d/grafana.list
# apt update
# apt install grafana
# systemctl start grafana-server
# systemctl enable grafana-server # autostart at boot

/etc/grafana/grafana.ini

# default port for grafana http server is 3888
http_port = 3000

misc

reset password

# grafana-cli admin reset-admin-password admin