How to silence nodes on Sensu
Sensu is an open source monitoring framework, written purely in Ruby, that we use heavily at Cloudways to monitor not only the servers of our customers but our own as well. Sensu is very extensive. If combined with Graphite, the pair can be used to cover a wide variety of monitoring and reporting/graphing scenarios.
While Sensu has a helpful API, what’s not very helpful is its documentation. For a lack of a better word, its documentation is sparse. While the v0.12 of its documentation attempts to fill in the gaps, it continues to fall short.
Recently, a need arose which required Sensu clients, which are servers that are running the Sensu agent, to be silenced when the servers on which they were running were stopped. Looking through the clients API, I could only find a way to delete a client, which theoretically could work, only, there wasn’t an API to re-register a client again. The documentation vaguely said that the clients registered themselves the first time they came alive.
The Sensu CLI, which isn’t an official part of Sensu, is a wonderful command line Sensu client. This client, interestingly, provides a means of silencing a client and un-silencing it. Playing with its source, I was finally able to figure out how it was doing it.
The way Senus makes possible silencing and un-silencing of clients is through the Stashes API. The important bit is the format of the path parameter. If it takes the form of /silence/{{ client_name }}, it will silence the client identified by client_name. Here’s a Python snippet to make it clear how to use the Stashes API to that effect:
The code should be self-explanatory. The silence() method makes a POST request to the Stashes API. The timestamp value identifies the current time in epoch. The unsilence method, in contrast, makes a DELETE request to the same Stashes API.
Note that by default the silence operation will silence the given client indefinitely. If you wish to silence it for a defined period, an expires parameter can be provided as part of the payload.
