Having a NAS with mechanical disks and a door open to your bedroom because of the heatwave may be problematic during the night if the disks spin up and start crunching.
In my case, the crunching noise happens every day at 8:00. Ok, maybe I like to sleep in and not hear crrrrr-crrrr-clack-clack as a wake-up call.
Time to find the culprit. Since the NAS is running a vanilla Ubuntu and the disks are used only for cold storage, I kind-of know why they spin up each time they do. Most likely some cron process is running at 8:00 and interrogating the disks.
Checking the crontab, all user's crontab, /etc/crontab, /etc/cron.d/* doesn't show anything obvious starting at 8:00. There could be some long-term running process which decides to index something at 8 o'clock, or there could be some external process on some other server that uses NFS or SSH to access some data on the NAS.
I decided to learn something new and use auditd to monitor file access to my /media/wdc mountpoint at 8:00 and see if I can figure out what's going on.
So - Step 0: Install and start auditd:
$ sudo apt-get install auditd
$ sudo service auditd start
Step 1: Set up a trigger to log read-write-execute access to files and directories under /media/wdc and add a tag of "test-hdd" for the logs
$ sudo /usr/sbin/auditctl -w /media/wdc/ -p warx -k test-hdd
But I'm not running it all the time, just enable it at 8:00 via cron:
0 8 * * * /usr/sbin/auditctl -w /media/wdc/ -p warx -k test-hdd
What gets enabled automatically should also get disabled, so at 8:03 I'm deleting the rules:
3 8 * * * /usr/sbin/auditctl -D -k test-hdd
Step 2: Wait for morning to come. This time I'm prepared!
Step 3: Once 8:00 happened, I can search the logs collected with:
# ausearch -k test-hdd --interpret | grep proctitle | cut -d ':' -f 5 | sort -u
Let's see what's going on:
proctitle=find /media/wdc/storage3TB/backup/ -type f -name README
proctitle=/usr/bin/python3 /usr/local/bin/borg info /media/wdc/storage3TB/backup/borg_aldebaran/borg_aldebaran
proctitle=/usr/bin/python3 /usr/local/bin/borg info /media/wdc/storage3TB/backup/borg_hc4/borg_hc4
proctitle=/usr/bin/python3 /usr/local/bin/borg info /media/wdc/storage3TB/backup/borg_stingray/borg_stingray
proctitle=/usr/bin/python3 /usr/local/bin/borg list --last 1 /media/wdc/storage3TB/backup/borg_aldebaran/borg_aldebaran
proctitle=/usr/bin/python3 /usr/local/bin/borg list --last 1 /media/wdc/storage3TB/backup/borg_hc4/borg_hc4
proctitle=/usr/bin/python3 /usr/local/bin/borg list --last 1 /media/wdc/storage3TB/backup/borg_stingray/borg_stingray
proctitle=/usr/bin/python3 /usr/local/bin/borg list /media/wdc/storage3TB/backup/borg_aldebaran/borg_aldebaran
proctitle=/usr/bin/python3 /usr/local/bin/borg list /media/wdc/storage3TB/backup/borg_hc4/borg_hc4
proctitle=/usr/bin/python3 /usr/local/bin/borg list /media/wdc/storage3TB/backup/borg_stingray/borg_stingray
Aha! So that file access is generated by a script that checks the health of my backups. I remember I created that script, but it's not run by cron. Perhaps it's a systemd service that does this.
# systemctl | grep borg
prometheus-borg-exporter.timer loaded active waiting Prometheus Borg Exporter Timer
So it's a timer! Systemd's equivalent for cron:
# cat /etc/systemd/system/prometheus-borg-exporter.timer
[Unit]
Description=Prometheus Borg Exporter Timer
[Timer]
OnCalendar= *-*-* 8:00:00
[Install]
WantedBy=timers.target
Got you! I had completely forgot I played with timers at some point. I had to remember the hard way.
Changing the timer here allowed me to wake up gently in the screams of my hungry kids, just like God intended...
Comments