(cache)ZFS health check script for Monit

When setting up your own home server then proper monitoring system is the one of most important things you want to do. Monit is a monitoring tool I decided to use. It's easy to configure but still very powerful. I also decided to use ZFS filesystem on my server. It's a bit more advanced than ext4 or ntfs thus there is more things to check regarding your pools health.

Thankfully I found very cool ZFS health check script on Calomel.org that sends email if something is wrong with any of the pools. I decided to adjust it a bit to work well with Monit.

Script, to be able to be run by Monit, needs to return exit code instead of sending an email. Next difference is that in bellows script lines 126 and 127 are uncommented (and lines 130 and 131 commented). This is to support Ubuntu date format. If you want to use this script on FreeBSD then you want these lines to look like in the original script. Next thing is that, in the beginning of the script, I added support for input parameters for max capacity and scrub expiration of your zpools. This is in order to keep configuration in one place (i.e. Monit's configuration file). Last but not least is the "Output for monit user interface" section. As its title says it outputs zpool status to the console so it could be recorded by Monit and displayed in its user interface.

#! /bin/sh
#
## ZFS health check script for monit.
## v0.9
#
## Should be compatible with FreeBSD and Linux. Tested on Ubuntu.
## If you want to use it on FreeBSD then go to Scrub Expired section
## and comment two Ubuntu date lines and uncomment two FreeBSD lines
#
## Assumed usage in monitrc (where 80 is max capacity in percentages
## and 691200 is scrub expiration in seconds):
## check program zfs_health with path "/path/to/this/script 80 691200"
##     if status != 0 then alert
#
## Original script from:
## Calomel.org
##     https://calomel.org/zfs_health_check_script.html
##     FreeBSD ZFS Health Check script
##     zfs_health.sh @ Version 0.17
#
## Main difference from the original script is that this one exits
## with a return code instead of sending an e-mail

# Parameters

maxCapacity=$1 # in percentages
scrubExpire=$2 # in seconds (691200 = 8 days)

usage="Usage: zfs_health_check.sh maxCapacityInPercentages scrubExpireInSeconds\n"

if [ ! "${maxCapacity}" ]; then
  printf "Missing arguments\n"
  printf "${usage}"
  exit 1
fi

if [ ! "${scrubExpire}" ]; then
  printf "Missing second argument\n"
  printf "${usage}"
  exit 1
fi


# Output for monit user interface

printf "==== ZPOOL STATUS ====\n"
printf "$(/sbin/zpool status)"
printf "\n\n==== ZPOOL LIST ====\n"
printf "%s\n" "$(/sbin/zpool list)"


# Health - Check if all zfs volumes are in good condition. We are looking for
# any keyword signifying a degraded or broken array.

condition=$(/sbin/zpool status | egrep -i '(DEGRADED|FAULTED|OFFLINE|UNAVAIL|REMOVED|FAIL|DESTROYED|corrupt|cannot|unrecover)')

if [ "${condition}" ]; then
  exit 1
fi


# Capacity - Make sure the pool capacity is below 80% for best performance. The
# percentage really depends on how large your volume is. If you have a 128GB
# SSD then 80% is reasonable. If you have a 60TB raid-z2 array then you can
# probably set the warning closer to 95%.
#
# ZFS uses a copy-on-write scheme. The file system writes new data to
# sequential free blocks first and when the uberblock has been updated the new
# inode pointers become valid. This method is true only when the pool has
# enough free sequential blocks. If the pool is at capacity and space limited,
# ZFS will be have to randomly write blocks. This means ZFS can not create an
# optimal set of sequential writes and write performance is severely impacted.

capacity=$(/sbin/zpool list -H -o capacity | cut -d'%' -f1)

for line in ${capacity}
  do
    if [ $line -ge $maxCapacity ]; then
      exit 1
    fi
  done


# Errors - Check the columns for READ, WRITE and CKSUM (checksum) drive errors
# on all volumes and all drives using "zpool status". If any non-zero errors
# are reported an email will be sent out. You should then look to replace the
# faulty drive and run "zpool scrub" on the affected volume after resilvering.

errors=$(/sbin/zpool status | grep ONLINE | grep -v state | awk '{print $3 $4 $5}' | grep -v 000)

if [ "${errors}" ]; then
  exit 1
fi


# Scrub Expired - Check if all volumes have been scrubbed in at least the last
# 8 days. The general guide is to scrub volumes on desktop quality drives once
# a week and volumes on enterprise class drives once a month. You can always
# use cron to schedule "zpool scrub" in off hours. We scrub our volumes every
# Sunday morning for example.
#
# Scrubbing traverses all the data in the pool once and verifies all blocks can
# be read. Scrubbing proceeds as fast as the devices allows, though the
# priority of any I/O remains below that of normal calls. This operation might
# negatively impact performance, but the file system will remain usable and
# responsive while scrubbing occurs. To initiate an explicit scrub, use the
# "zpool scrub" command.
#
# The scrubExpire variable is in seconds.

currentDate=$(date +%s)
zfsVolumes=$(/sbin/zpool list -H -o name)

for volume in ${zfsVolumes}
  do
    if [ $(/sbin/zpool status $volume | egrep -c "none requested") -ge 1 ]; then
      printf "ERROR: You need to run \"zpool scrub $volume\" before this script can monitor the scrub expiration time."
      break
    fi

    if [ $(/sbin/zpool status $volume | egrep -c "scrub in progress|resilver") -ge 1 ]; then
      break
    fi

    ### Ubuntu with GNU supported date format
    scrubRawDate=$(/sbin/zpool status $volume | grep scrub | awk '{print $11" "$12" " $13" " $14" "$15}')
    scrubDate=$(date -d "$scrubRawDate" +%s)

    ### FreeBSD with *nix supported date format
    #scrubRawDate=$(/sbin/zpool status $volume | grep scrub | awk '{print $15 $12 $13}')
    #scrubDate=$(date -j -f '%Y%b%e-%H%M%S' $scrubRawDate'-000000' +%s)

    if [ $(($currentDate - $scrubDate)) -ge $scrubExpire ]; then
      exit 1
    fi
  done


# Finish - If we made it here then everything is fine

exit 0


### EOF ###

To use the script go to your monitrc file and add following lines:

check program zfs_health with path "/path/to/this/script 80 691200"
  if status != 0 then alert

Where 80 is the max capacity in percentages and 691200 is the scrub expiration in seconds. This will make Monit notify you everytime something is wrong with your zpool.

As a bonus you will get nice status with last script run output on the web user interface:

**Figure 1.** ZFS health check status on the web user interface

If you don't see the whole output for your zpools then you probably need to set higher PROGRAMOUTPUT limit in monitrc file. The default one is 512 bytes.

The script is also available as Github Gist.

Thanks (1)

Back to articles list

Comments

evgenymagata

14-12-2018 08:59:47 UTC

I tried your script, works perfectly from command line. But apparently Monit does not support passing arguments to the scripts, i wonder how you got this running? this is my status output:
´Missing arguments
Usage: zfs_health_check.sh maxCapacityInPercentages scrubExpireInSeconds´

when running like this

´check program zfs_health with path "/bin/bash -c /root/monit_scripts/zfs_check.sh 80 691200"
if status != 0 then alert´

Rychu

14-12-2018 17:39:49 UTC

@evgenymagata skip "/bin/bash -c" in "with path" statement. Take a look at the example in the article. In your case it should be like:

check program zfs_health with path "/root/monit_scripts/zfs_check.sh 80 691200"
    if status != 0 then alert

Let know if that helps.

Comments

evgenymagata

Rychu

Add comment