Adding a custom tag to data coming in from an input plugin for telegraf
— 2 min read
For a recent project, I wanted to add a custom tag to data coming in from a built-in input plugin for telegraf.
The input plugin was the procstat plugin, and the custom data was information from pacemaker (a clustering solution for linux). I wanted to add a tag indicating if the current host was the "active" host in my active/passive setup.
The execd processor plugin runs an external program as a separate process and pipes metrics in to the process's STDIN and reads processed metrics from its STDOUT.
Telegraf's filtering parameters can be used to select or limit data from which input plugins will go to this processor.
The external program
The external program I wrote does the following:
Get pacemaker status and cache it for 10 seconds
Read a line from stdin
Append the required information as a tag in the data
Write it to stdout
The caching is just an optimization - it was more to do with decreasing polluting the logs than actual speed improvements.
Also, I've done the Influxdb lineprotocol parsing in my code directly (because my usecase is simple), but this can be substituted by an actual library meant for handling lineprotocol.
Here's a sample telegraf configuration that routes data from "system" plugin to execd processor plugin, and finally outputs to influxdb.
Other types of dynamic tags
In this example, we wanted to get the value of the tag from an external program. If the tag can be calculated from the incoming data itself, then things are much simpler. There are a lot of processor plugins, and many things can be achieved using just those.
This post will go through my experience with setting up some advanced monitoring
for PostgreSQL database using Telegraf, InfluxDB, and Grafana (also known as the
TIG stack), the problems I faced, and what I ended up doing at the end.
What do I mean by advanced? I liked this