Grafana alerts in NTFY

I’ve used Grafana for years at this point. Mainly for monitoring my own servers, VPS’es, laptops, and desktops.

Apart from building fancy dashboards, I wanted to get alerted if something was wrong (like running out of disk space), but the built-in options for receiving alerts didn’t really suit me. So I decided to build something.

First I integrated it with Matrix using a small Go program wrote - Grafana Forwarder. It was quite simple, but I added a few features like reacting to the original alert message when the alert was resolved. It was a great way to learn more about Grafana and the Matrix protocol.

Some time later I came across NTFY and started using it to get alerted for other things, so it made sense for my Grafana alerts to come through here too. NTFY was perfect for this!

ntfy (pronounced notify) is a simple HTTP-based pub-sub notification service. It allows you to send notifications to your phone or desktop via scripts from any computer, and/or using a REST API. It’s infinitely flexible, and 100% free software.

— ntfy website

So I followed the same pattern and made a very simple “forwarder” to send alerts through to NTFY: another simple Go project! (this one never released)

I’ve been using this system for over a year and it has been working well. But last week, as I was playing around in he Cloudflare dashboard, I started thinking that I could probably implement this as a Cloudflare worker.

So I did! I wrote a completely new project to handle this that runs as a Cloudflare worker (for free!)

Here is the project page: Grafana to NTFY

And here is what the alerts look like:

Alert firing:

Screenshot of NTFY alerts for the battery level of my laptop. Alert for the laptop battery being low.

And an alert being resolved:

Screenshot of NTFY alerts for the battery level of my laptop. Alert resolving the alert when the battery level has gone back above the minimum level.

The concept remains very simple: take a HTTP request from Grafana (a webhook contact point), convert it into a format NTFY can understand, and sends it along. There really isn’t much more to it.

The biggest changes to previous versions are the language (Typescript instead Go) and how/where it gets deployed.

I also took the opportunity to add a couple of features the previous iteration didn’t have. In particular allowing the alert rule in Grafana to set the priority and emoji sent in the NTFY alert. This allows more customisation in the alerts I receive, all driven directly from Grafana.

The code is open-source, so anyone can run their own “forwarder”, all you need is a free Cloudflare account.

Once you have it running, all you need to do is set up a contact point in Grafana of type Webhook, point it at your worker, and that’s about it!

This was a fun little project. I’m happy with the result and I think I built something I’ll be using for many months to come!

Now I’m wondering what I’ll build next with Cloudflare workers 🤔