www.voj-tech.net


vt-backup

Synopsis

vt-backup is a set of Bash scripts allowing for setting up a regular automatic performing of backups of specific files and directories using rsync.

Motivation

Originally I wrote vt-backup for myself but I thought it might easily suit the needs of others as well.

Installation

Prerequisites

vt-bash-utils 0.9.0 or later are required to enable desktop notifications informing about the progress of the backups. In particular vt-backup calls the functions vtbu_send_desktop_notification and vtbu_single_instance_running. Please download the library from the project page and install it following the instructions provided.

It is required that your environment features reasonably new versions of:

  • rsync
  • Ncat
  • the find command line utility
  • the flock command line utility
  • the GNU core utilities
  • Bash

It should be noted that the scripts have been tested using rsync 3.1.2, Ncat 7.40, findutils 4.6.0 (provides the find utility), util-linux 2.28.2 (provides the flock utility), coreutils 8.25 and Bash 4.3.43 on Fedora 25.

Installation Steps

It is currently required to perform a manual installation. To do so follow these steps:

  1. Create the directories /etc/vt-backup and /var/local/vt-backup/lock.

    mkdir -p /etc/vt-backup /var/local/vt-backup/lock
    
  2. Download a ZIP file with the required version of the scripts from this page. Most likely you want the latest version.

  3. Unpack the downloaded file.

    unzip vt-backup-0.8.1.zip && cd vt-backup-0.8.1
    
  4. Skip this step if desktop notifications are not desired, e.g. when installing in a server environment.

    If the vt-bash-utils library was installed to a directory other than the /usr/local/share directory modify the vt-backup-notify.sh script on the line where the variable bash_utils_path is assigned a value to specifying the correct directory.

  5. Skip this step if desktop notifications are not desired, e.g. when installing in a server environment.

    If you want to use a port number other than 7654 for sending desktop notification messages to the vt-backup-notify.sh process modify the vt-backup-notify.sh script on the line where the variable port is assigned a value to and the vt-backup.sh script on the line where the variable notify_port is assigned a value to specifying the desired port number in both cases.

  6. If you are planning on backing up to a destination directory which does not match the regular expression '^/run/media(/[^/]+){2,}$' modify the vt-backup.sh script on the line where the variable dest_validation_regular_expression is assigned a value to specifying the desired expression. Before any backup is processed it is checked whether the configured destination directory matches this regular expression as a precaution against data loss caused by an incorrect backup configuration.

  7. If you want to perform backups in an interval other than a one-day interval modify the vt-backup.sh script on the line where the variable threshold is assigned a value to specifying the desired relative time or date of a backup which would be the latest backup considered as an out of date one. Refer to the GNU core utilities documentation in regards to the date input format.

  8. Copy the scripts to a directory which is in your PATH environment variable and which is your preferred location to install third-party software to. This might be the /usr/local/bin directory for example. You might want to copy the vt-backup.sh script to the ~root/bin directory instead as it is always going to be run as the root user. In either case make sure this is done as a user account allowed to write to the destination directory. There is no need to copy the vt-backup-notify.sh script anywhere if desktop notifications are not desired, e.g. when installing in a server environment.

    cp vt-backup-notify.sh /usr/local/bin
    cp vt-backup.sh ~root/bin
    
  9. Add the execute mode bit to the script files' mode bits. Again, make sure that this is done as the appropriate user account. You will only add the execute mode bit to the vt-backup-notify.sh script if desktop notifications are desired.

    chmod +x /usr/local/bin/vt-backup-notify.sh ~root/bin/vt-backup.sh
    
  10. Optionally configure logrotate to deal with vt-backup logs which can be produced by redirecting the standard output of the vt-backup.sh script to a file such as the file /var/log/vt-backup.log. Again, make sure that this is done as the appropriate user account.

    cat > /etc/logrotate.d/vt-backup <<'EOF'
    /var/log/vt-backup.log {
        compress
        dateext
        delaycompress
        missingok
        nocreate
        notifempty
        rotate 4
        weekly
    }
    EOF
    
  11. Skip this step if desktop notifications are not desired, e.g. when installing in a server environment.

    A common way to enable an application to autostart upon login to a desktop environment is to create a desktop entry for it in one of the directories /usr/xdg/autostart and ~/.config/autostart. Do that for the vt-backup-notify.sh script.

    cat > ~/.config/autostart/vt-backup-notify.sh.desktop <<'EOF'
    [Desktop Entry]
    Type=Application
    Exec=vt-backup-notify.sh
    Hidden=false
    Name=vt-backup-notify.sh
    Comment=
    EOF
    
  12. To enable a regular automatic performing of backups create a cron job for the vt-backup.sh script. The script needs to be run as the root user. This is required for obvious reasons when the root user is the only user having read access to files configured to backup. It is however required in any case since the script preserves file owners and groups.

    crontab_file=$(mktemp)
    crontab -l > "$crontab_file"
    cat >> "$crontab_file" <<'EOF'
    SHELL=/bin/bash
    PATH=/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
    * * * * * vt-backup.sh >> /var/log/vt-backup.log 2>&1
    EOF
    crontab "$crontab_file"
    rm "$crontab_file"
    

Reference

Usage

vt-backup-notify.sh

Start the vt-backup-notify.sh script to listen for backup events and send desktop notifications.

The intention is to configure the desktop environment to do this upon login (see above).

vt-backup.sh

Perform backups which are due according to configuration files (as the root user).

The intention is to create a cron job for the root user to do this, redirecting the output to a log file (see above).

Neither of the scripts currently understand any command line options or other arguments.

Description

vt-backup comprises two scripts; a back end script vt-backup.sh which processes the backups and a front end script vt-backup-notify.sh which sends desktop notifications informing the user about the progress of the backups. The scripts communicate through a TCP/IP loopback. It is optional to run the front end script.

The vt-backup-notify.sh script listens for desktop notification messages from the back end script. Such a message is a two-element structure; it contains the backup configuration name and a string describing the status of the backup processing. When a message arrives the script translates it into a desktop notification and sends the notification.

The vt-backup.sh script makes a record of the current time as a reference time to later work out if a backup is due or not. It then reads the configuration files (see below) in sequence while doing the following for each:

  1. It validates the configured destination directory by matching it against the regular expression which is the value of the dest_validation_regular_expression variable (see above). If the directory does not match the expression an error message is emitted and the configuration file is ignored.

  2. It checks whether the configured destination directory exists or not. If it does not, the configuration is skipped. This probably means that the removable media in question is not mounted.

  3. If a lock file corresponding to the configuration exists the script compares the current time recorded at the start with the modification time of the lock file. If the time difference between the two is shorter than the value expressed by the threshold variable (see above) the configuration is skipped.

  4. The script attempts to lock the corresponding lock file creating it first if it does not exist. If the file is already locked the configuration is skipped.

  5. It sends a notification message to the front end script process signalling that the backup is about to start.

  6. It then processes the lines of the configuration file which define files and directories to backup while doing the following for each:

    • It constructs the full path to the destination directory for this particular configuration line as the concatenation of the full path to the destination directory and the full path to the base directory (see below).
    • If the destination directory for this configuration line does not exist the script creates it preserving the file permissions, owners, groups and SELinux security contexts and thus creates a sparse copy of the file system root directory in the backup destination directory.
    • It then constructs an array of full paths to the source files and directories to backup and executes rsync with the options -a, -X, -v and --delete using the constructed array and the full path to the destination directory for this configuration line.
  7. The script sends a notification message to the front end script process signalling that the backup has finished.

  8. It unlocks the lock file.

vt-backup currently does not support creating incremental or differential backups. A full backup is always taken overwriting the previous one.

Currently the vt-backup-notify.sh script prevents running multiple instances of itself at the same time. Therefore it does not fully support a system shared among multiple users. This however does not stop the back end vt-backup.sh script from working correctly.

Exit Status

The vt-backup.sh script exits with the status of the last command which is to remove a temporary file used internally by the script. This is 0 if it succeeds, otherwise it is a value greater than 0.

Files

  • /etc/vt-backup: Directory containing the configuration files. Each file in this directory is parsed by vt-backup as a configuration file and represents a separate backup configuration. The format of a configuration file is as follows:

    • The first line is a full path to a directory which is the backup destination. Typically this is a directory on a removable media. The backup is processed only if this directory exists.
    • All the other lines are definitions of files and directories to backup. These lines contain comma-separated values where the first value is a full path to a directory called the base directory and the rest of the values are file and directory names within the base directory to be backed up.

    In the following example the directories /home/jimmy/Documents/Invoices, /home/jimmy/Documents/Letters and /home/jimmy/Pictures and the file /home/jimmy/.zshenv would be backed up to the directory /run/media/jimmy/my-removable-media/backup.

    /run/media/jimmy/my-removable-media/backup
    /home/jimmy,Pictures,.zshenv
    /home/jimmy/Documents,Invoices,Letters
    

    Please note that the destination directory must match the regular expression which is the value of the dest_validation_regular_expression variable (see above).

  • /var/local/vt-backup/lock: Directory containing lock files. For each backup configuration file there is a lock file with the same name. The lock files serve two purposes. A lock file gets locked as the corresponding backup is processed for synchronisation purposes. Furthermore the modification time of a lock file is used to determine when the corresponding backup is due.

Bugs

Suggestions and bug reports are more than welcome at voj-tech@voj-tech.net.