Skip to main content

Which process wakes up my HDD at 8 AM?!

 Having a NAS with mechanical disks and a door open to your bedroom because of the heatwave may be problematic during the night if the disks spin up and start crunching.

In my case, the crunching noise happens every day at 8:00. Ok, maybe I like to sleep in and not hear crrrrr-crrrr-clack-clack as a wake-up call.

Time to find the culprit. Since the NAS is running a vanilla Ubuntu and the disks are used only for cold storage, I kind-of know why they spin up each time they do. Most likely some cron process is running at 8:00 and interrogating the disks.

Checking the crontab, all user's crontab, /etc/crontab, /etc/cron.d/* doesn't show anything obvious starting at 8:00. There could be some long-term running process which decides to index something at 8 o'clock, or there could be some external process on some other server that uses NFS or SSH to access some data on the NAS. 

I decided to learn something new and use auditd to monitor file access to my /media/wdc mountpoint at 8:00 and see if I can figure out what's going on.

So - Step 0: Install and start auditd:

$ sudo apt-get install auditd

$ sudo service auditd start

Step 1: Set up a trigger to log read-write-execute access to files and directories under /media/wdc and add a tag of "test-hdd" for the logs

$ sudo /usr/sbin/auditctl -w /media/wdc/ -p warx -k test-hdd

But I'm not running it all the time, just enable it at 8:00 via cron:

0 8 * * * /usr/sbin/auditctl -w /media/wdc/ -p warx -k test-hdd

What gets enabled automatically should also get disabled, so at 8:03 I'm deleting the rules:

3 8 * * * /usr/sbin/auditctl -D -k test-hdd

Step 2: Wait for morning to come. This time I'm prepared!

Step 3: Once 8:00 happened, I can search the logs collected with:

# ausearch -k test-hdd --interpret | grep proctitle  | cut -d ':' -f 5 | sort -u

Let's see what's going on:

proctitle=find /media/wdc/storage3TB/backup/ -type f -name README
 proctitle=/usr/bin/python3 /usr/local/bin/borg info /media/wdc/storage3TB/backup/borg_aldebaran/borg_aldebaran
 proctitle=/usr/bin/python3 /usr/local/bin/borg info /media/wdc/storage3TB/backup/borg_hc4/borg_hc4
 proctitle=/usr/bin/python3 /usr/local/bin/borg info /media/wdc/storage3TB/backup/borg_stingray/borg_stingray
 proctitle=/usr/bin/python3 /usr/local/bin/borg list --last 1 /media/wdc/storage3TB/backup/borg_aldebaran/borg_aldebaran
 proctitle=/usr/bin/python3 /usr/local/bin/borg list --last 1 /media/wdc/storage3TB/backup/borg_hc4/borg_hc4
 proctitle=/usr/bin/python3 /usr/local/bin/borg list --last 1 /media/wdc/storage3TB/backup/borg_stingray/borg_stingray
 proctitle=/usr/bin/python3 /usr/local/bin/borg list /media/wdc/storage3TB/backup/borg_aldebaran/borg_aldebaran
 proctitle=/usr/bin/python3 /usr/local/bin/borg list /media/wdc/storage3TB/backup/borg_hc4/borg_hc4
 proctitle=/usr/bin/python3 /usr/local/bin/borg list /media/wdc/storage3TB/backup/borg_stingray/borg_stingray

Aha! So that file access is generated by a script that checks the health of my backups. I remember I created that script, but it's not run by cron. Perhaps it's a systemd service that does this.

# systemctl | grep borg
  prometheus-borg-exporter.timer              loaded active waiting   Prometheus Borg Exporter Timer

So it's a timer! Systemd's equivalent for cron:

# cat /etc/systemd/system/prometheus-borg-exporter.timer
[Unit]
Description=Prometheus Borg Exporter Timer

[Timer]
OnCalendar= *-*-* 8:00:00

[Install]
WantedBy=timers.target

Got you! I had completely forgot I played with timers at some point. I had to remember the hard way.

Changing the timer here allowed me to wake up gently in the screams of my hungry kids, just like God intended...



Comments

Post a Comment

Popular posts from this blog

Home Assistant + Android TV = fun

Here's a quick setup guide for controlling your Android TV from within Home Assistant. I've used it to control a genuine Android TV (Philips 7304) and an Odroid N2 running Android TV. For this to work you need ADB access. It can usually be enabled from within Developer Settings. The great part is - you don't need root access! The most important things are described in the androidtv component for Home Assistant: https://www.home-assistant.io/integrations/androidtv/ Make sure you go through the adb setup. My configuration is simple (inside configuration.yaml): media_player:   - platform: androidtv     name: TV Bedroom ATV     host: 192.168.1.61     device_class: androidtv Once Home Assistant restarts, your TV might require you to accept the connection (adb authentication). This happens only once (or until you reset your ATV to factory settings). Once running the integration will show you the current ATV state (on or off) and al...
6 comments
Read more

SmokePing + InfluxDB export + docker + slaves + Grafana = fun

I've been working for a while on this project - with the purpose of getting SmokePing measurements from different hosts (slaves) into InfluxDB so that we can better graph them with Grafana. The slaves run multiple Smokeping instances inside Docker so that they have separate networking (measure through different uplinks, independently). This will not be a comprehensive configuration guide, but a quick "how to" to handle setup and basic troubleshooting. It assumes you already know how to set up and operate a regular Smokeping install with or without slaves and that you are fluent in Smokeping configuration syntax, know your way around Docker and aren't a stranger from InfluxDB and Grafana (sorry, there's a lot of information to take in). 1. Getting Smokeping with InfluxDB support - you can get it either from the official page (most changes have been merged) - https://github.com/oetiker/SmokePing (PR discussion here: https://github.com/oetiker/SmokePing/issues/...
16 comments
Read more

Android: Adding scp/sftp support to dropbear and mounting with sshfs

I have recently received an android smartphone, and one of the first things I did with it was to root it :). This allows power-usres to get the most out of their hardware. The next thing on my list was to set up a SSH server and to be able to transfer files between my Linux system and my phone (by the way, I'm running Android 4.1 and it seems USB mass storage support has been removed. MTP/PTP modes have either horrible transfer speed or are poorly supported in Ubuntu). With the above in mind, the plan was to: Enable tethering on the phone Run a SSH server to support issuing remote commands and file transfer (FTP might have been an alternative, but I'm a SSH adept). Browsing the market I found  SSHDroid which does all that it advertised. Problem is - the free version conflicts with my add-blocking apps and requests that they are disabled to run. For me, this is a big nuisance, so I kept looking. I found Dropbear SSH Server 2 , which is completely free, but doesn'...
5 comments
Read more