Blog
A side dish is still a dish
- 31 December 2022
- en
- Blog
While writing together some thoughts on 2022 I wanted to elaborate on one of my side projects in 2022. Initially this started as putting together a simple monitoring stack for a NGO I’m involved in. Instead of simply installing the components I chose to make everything easily deployable via ansible.
The core components involved are:
- prometheus - alongside with alertmanager and various exporters
- karma - an alert dashboard for alertmanager
- grafana - visualisation for everything else
I’ve built the setup around docker. At some point I’ll rework it to work with podman, for now its based on docker. For that it comes with an ansible role to prep the host for running docker containers. Currently I’m working on getting log aggregation up and running with loki - there will be future posts on that.
Aside from alerting via e-mails I wanted to have the possibility to send out alerts via matrix. For this I wrote a role to deploy goneb, which is a matrix bot.
All of the components are running behind a reverse proxy that is built with nginx. The ansible role for nginx takes care of proper SSL-termination. Either via certificates created via let’s encrypt or via self-signed certificates. The ansible role supports both.
The core is a prometheus role that I forked from OME. The original role has not been touched in more than two years and it had some drawbacks that I wanted to solve. Aside from the prometheus role, the role to bootstrap the docker host is forked from elsewhere. For easier upstream tracking, I kept the roles that I forked on the original platform (github.com for the prometheus role, gitlab for the docker host role).
- prometheus as well as alertmanager and blackbox exporter role
- grafana dockerized setup
- alert dashboard karma dockerized
- matrix bot to send alerts to a matrix room
The roles to support the environment:
These roles can be tied together in a simple playbook. Since there are a few variables that hold sensitive data, you’ll need to come up with a way to handle your secrets. I do that via an integration with gopass (which leaves the scope of this intro).
Especially the users to access prometheus as well as for the matrix bot should be kept secret.
goneb_access_token: ""
goneb_device_id: ""
goneb_room: ""
nginx_user: "username:password"
grafana_admin_password: "grafana-password"
In my setup I bundle everything up via git submodules, so that the playbook only holds the glue code to pull everything together.
The main.yaml in the playbook could look like this:
---
- hosts: all
vars:
domain_name: "monitoring.example.net"
prometheus_alertmanager_webhook: "https://goneb-bot.example.net/services/hooks/gaa6Oshah4ieJie6neih"
prometheus_targets: [
{
hosts: [ "hostA.example.net", "hostB.example.net" ],
jobname: node_exporter,
port: 9100,
scheme: https,
insecure_skip_verify: true
}
]
prometheus_http_2xx_external_targets: [ "https://www.example.net" ]
prometheus_icmp_targets: [ "hostA.example.net" ]
prometheus_tcp_targets: [ "imap.example.net:993" ]
prometheus_smtp_targets: [ "smtp.example.net" ]
prometheus_alertmanager_additional_command_args: "--web.external-url=https://"
prometheus_internal_ip: 127.0.0.1
prometheus_alertmanager_internal_ip: 127.0.0.1
prometheus_blackboxexporter_internal_ip: 127.0.0.1
prometheus_alertmanager_domain_name: "alertmanager.example.net"
prometheus_blackboxexporter_domain_name: "blackboxexporter.example.net"
prometheus_blackboxexporter_smtp_test_from: "monitor@example.net"
prometheus_blackboxexporter_smtp_test_rcpt: "monitor@example.net"
prometheus_static_targets: [ "prometheus:9090" ]
goneb_userid: "@my-bot-name:matrix.org"
goneb_display_name: "monitoring bot"
goneb_homeserver_url: "https://matrix-client.matrix.org"
goneb_base_url: "https://goneb-bot.example.net/"
grafana_domain_name: "monitoring.example.net"
karma_alertmanager_uri: "https://alertmanager.example.net"
karma_internal_ip: 127.0.0.1
nginx_letsencrypt_email: "info@example.net"
nginx_letsencrypt_enabled: true
nginx_hosts: [
{
default: true,
domain_name: "monitoring.example.net",
users: [ "" ],
nginx_satisfy: "any",
nginx_allow: [ "127.0.0.1", "", "", "", "", "::1" ],
reverse_hosts: [
{
protocol: "http",
path: "/",
host: "",
port: ""
},
{
protocol: "http",
path: "/grafana",
host: "",
port: "",
extra: "proxy_set_header Host $http_host;\n
# grafan has its own authentication enabled\n
auth_basic off;"
},
],
extra: "location /grafana/api/live {\n
rewrite ^/(.*) /$1 break;\n
proxy_http_version 1.1;\n
proxy_set_header Host $http_host;\n
proxy_pass http://:/;\n
# grafan has its own authentication enabled\n
auth_basic off;\n
}"
},
{
domain_name: "blackboxexporter.example.net",
users: [ "" ],
nginx_satisfy: "any",
nginx_allow: [ "127.0.0.1", "", "", "", "::1" ],
reverse_hosts: [
{
protocol: "http",
path: "/",
host: "",
port: " "
}
]
},
{
domain_name: "alertmanager.example.net",
users: [ "" ],
nginx_satisfy: "any",
nginx_allow: [ "127.0.0.1", "", "", "", "", "::1" ],
reverse_hosts: [
{
protocol: "http",
path: "/",
host: "",
port: ""
}
]
},
{
domain_name: "karma.example.net",
users: [ "" ],
reverse_hosts: [
{
protocol: "http",
path: "/",
host: "",
port: ""
}
]
},
{
domain_name: "goneb-bot.example.net,
reverse_hosts: [
{
protocol: "http",
path: "/",
host: "",
port: ""
}
]
}
]
roles:
- ansible-docker-host
- ansible-role-prometheus
- ansible-goneb-docker
- ansible-grafana-docker
- ansible-karma-docker
- ansible-nginx-letsencrypt
Feel free to ping me with any questions ;)