Zabbix: Use Low-Level Discovery for Software Update Notifications

Assume you have an already running Zabbix instance that is able to send notifications – via e-Mail or Signal for example. The goal of this post is, to use such an instance to get notified whenever there is a new release for a software you may have installed on some of your machines. Just to manage expectations: this is not about a full-blow "asset management" for software installed in your infrastructure. The goal is not to automatically detect installed software applications together with their version. Or then to detect discrepancies between the installed version and the most recent version. Or even enrich this information with feeds of vulnerabilities to prioritize updates. This all would be cool and probably sell-able, so we are simply going to use Zabbix and a feature called "Low-Level Discovery" to send out notifications based on the response of a REST-like API. Also, I wrote this post to document the feature for myself, hoping that next time I use Low-Level Discovery, I'll not re-read the entire documentation. ## Rest-like API Because I needed the most recent version of some software applications in multiple places, I decided to go API-first here and implement a simple REST-like API available at: https://software-versions.wallenborn.net/v1/most_recent. It will respond with something like:

{
  "wordpress": "5.7",
  "nextcloud": "21.0.0",
  "signal-cli": "0.8.1"
}

The code for the REST-like API is available on GitHub. The API is very simple, it tries to guess based on RSS feeds, custom APIs or github releases, what the current version is and emits that in the response. Think of this API as a "normalizer" to retrieve the most recent version of software. ## Zabbix I have a machine which is responsible for all kinds of automation tasks that also happens to run a Zabbix agent. The plan is, to let this machine query the above mentioned API and generate item data from that. In case you forgot: an "item" is something like "column" in the Zabbix context. The data in those items will then be used via a trigger to send out notifications whenever the item value changes. I didn't want to change the Zabbix configuration every time I add a new software to the above monitoring, so Zabbix needs to somehow automatically generate items based on the API response: enter "Low-Level Discovery". Create a new template and within that template a new "Discovery Rule". We will be using a Python script to call zabbix_sender so the type of the Discovery Rule will be "Zabbix trapper" and let's agree on the name software_versions.discovery. After that, add a new "Item Prototype" – this is a rule Zabbix will use to generate new items whenever it receives new values for the software_versions.discovery Discovery rule. Zabbix accepts JSON for Discovery Items and we will be sending data like the following to software_versions.discovery:

{'data': [
    {'{#SOFTWARENAME}': 'wordpress'},
    {'{#SOFTWARENAME}': 'nextcloud'},
    {'{#SOFTWARENAME}': 'signal-cli'}
]}

(in more recent Zabbix version, you don't seem to need the dictionary with a single data anymore) Note the key name {#SOFTWARENAME}: this string will be replaced by the corresponding value for every prototype you create in Zabbix. So if you create an Item Prototype with the following parameters, Zabbix will create one item for every list entry under the data key. * *Name:* {#SOFTWARENAME}: Most Recent Version * *Type:* Zabbix trapper * *Key:* software_versions.most_recent_version[{#SOFTWARENAME}] * *Type of information:* text So in this case, you will end up with three items with keys like software_versions.most_recent_version[wordpress]. Let's also create two Trigger Prototypes to get notified when the cronjob populating this data didn't run for too long or - obviously - when a new version was released. New Version Information: * *Name:* "[Software {#SOFTWARENAME}] new version available" * *Severity:* Information * *Expression:* {Template Software:software_versions.most_recent_version[{#SOFTWARENAME}].diff(0)}=1 Trigger to warn if cronjob didn't run for two days: * *Name:* "[Software {#SOFTWARENAME}] no new version information" * *Severity:* High * *Expression:* {Template Software:software_versions.most_recent_version[{#SOFTWARENAME}].nodata(2d)}=1 ## Cronjob We will now use zabbix_sender on the automation host – don't be me and forget to assign the created template to this host – to first discover all monitored software products and then send the most recent version for each of those. In the following script, ZabbixSender abstracts away calling zabbix_sender:

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger()
sender = ZabbixSender(logger, '/usr/bin/zabbix_sender', '/etc/zabbix/zabbix_agentd.conf')

response = requests.get(os.environ.get('API_URL', 'https://software-versions.wallenborn.net/v1/most_recent'))
response.raise_for_status()
response_data = response.json()

discovery = []
for software_name in sorted(response_data.keys()):
    discovery.append({'{#SOFTWARENAME}': software_name})
sender.send_item('software_versions.discovery', json.dumps({'data': discovery}))
logger.info(F'Discovered {len(discovery)} software names.')

for software_name, current_version in response_data.items():
    try:
        sender.send_item(F'software_versions.most_recent_version[{software_name}]', current_version)
    except ZabbixSenderException as e:
        logger.exception(e)
        logger.error('Cannot send item, maybe a new software was discovered, just re-run the script in a minute.')

I call this script daily and Zabbix monitors both that the script is called and emitted some data as well as if there was a new version released for any of the monitored software products. A full version of this script can also be found on GitHub. ## Conclusion This blag posts covers three steps: 1. Stand up an API that retrieves most recent version information 2. Configure Zabbix to automatically create items and triggers based on "discovered" software 3. Populate these newly created items with data retrieved from the API There are obvious improvements like using some sort of caching in the API or ensuring that the version didn't only change but also that it increased. But I think this is over-engineerd enough already.