cPremote backup for cPanel

tl;dr

cPremote is a backup system for cPanel servers that lets you push backups to a remote storage location with minimal resource use on the cPanel server itself. It gives the server admin the ability to restore accounts from remote backup without having to manually copy the backup files to the cPanel server first, and end users can restore their own files and databases themselves, via cPanel. The software is reasonably priced, but it has some obvious quality issues and support is poor.

Background

Our main server backup system is R1Soft Server Backup Manager. It works well, using minimal resources on both the storage server and the servers being backed up. It does however have some limitations, especially when it comes to backing up cPanel servers.

The main problem we have with R1Soft is that restoring databases simply doesn't work. This may be because we are now using MariaDB, or perhaps it has to do with the particulars of the database engine in use. Whatever the cause, attempts to restore individual databases eventually fail after a long wait. This leaves us having to restore a database dump file, extract the database from the dump, and then restore it using the mysql command or similar tools.

Another problem with backing up cPanel servers with R1Soft is that it doesn't have any knowledge of how cPanel accounts are structured, so a cPanel account cannot be easily restored. We work around this by using a cron job to package the accounts using cPanel-provides scripts before they are backed up. An account can then be restored by first restoring the account package from backup, restoring the account via WHM, restoring all the user's files from the R1Soft storage server, and finally correcting file ownerships if necessary. A lot of steps that involve accessing several tools - the R1Soft backup manager, WHM, and the command line on the server.

Finally, while R1Soft does provide a cPanel plugin that allows individual users to restore their own files, this plugin does not work properly with the "Paper Lantern" theme that is now default with cPanel. It is possible to get it to work with a bit of manual setup, but the process is far from smooth.

The cPanel Backup System

cPanel natively provides a backup system that works reasonably well and which solves the problems mentioned above. It has its own set of issues, however:

  • Large use of storage space on the cPanel server
  • High CPU load on cPanel server during backups if compression is used
  • WHM cannot restore directly from remote backup destinations

cPanel's backup system works by looping through all the accounts on the server, packaging each account (including all the account's files and databases), and storing all the account packages in a separate location on the server. The account packages can be compressed to reduce disk space use, but this puts a large load on the server CPU while backups are running. (Compression CPU usage is tunable to some extent via "tweak settings" in WHM, but we still found it to be unacceptably high on our servers).

The cPanel backup system lets you specify retention rules, so you can configure it to keep a number of daily and weekly backups in addition to a monthly backup. However, each of these backups is a separate, complete copy of every account on the server, so the storage space requirements quickly become very high. It is possible to switch to an incremental backup strategy instead, but this does not allow retaining more than one backup copy, so with incremental backups it is only possible to restore files and databases from the most recent backup run.

cPanel's backup system has the ability to automatically transfer completed backups to one or more remote destinations using various transfer methods (FTP, SFTP, WebDAV). Amazon S3 is natively supported, and it is possible to write scripts that can be hooked into the backup system if you need to transfer backups to a remote destination type that is not otherwise supported.

The catch is that the remote transfer process runs separately from the backup process on the server. And since the transfer usually runs much slower than the backup process, you still have to allocate enough space on the cPanel server to store at least one complete copy of all the accounts on the server - even if you configure the backup system to not retain any local copies. And even if transfer to the remote server is fast enough to keep up, you still have to allow for the case where the transfer fails and all backups are left on the server being backed up.

Incremental backups do not work with remote destinations, so the bandwidth use is significant, especially if you do not compress backups. Also note that restoring a backup from a remote destination requires the account package to first be manually copied from the remote destination to the cPanel server.

If you have the disk space to store backups locally (for example on a separate disk in a dedicated server), then the native cPanel backup system may be all you need. Just add at least one remote destination - preferably in a different location, if bandwidth use is not a concern.

In our case, however, we do not have disk space to store backups locally - our cPanel servers are on VPS's where disk space is limited. We need a backup solution that basically does the same as cPanel's backup system, but which does not run backup and transfer as independent processes. We want it to take each account in turn, create a backup package, transfer the package to a remote destination, and then delete the local copy before proceeding with the next account.

Custom backup script

cPanel provides a script that allows you to package accounts from the command line, and it is not very hard to write a script that can transfer them offsite. We have been using s3cmd to transfer account backups to Amazon S3 with a script similar to this:

#!/bin/bash
PATH=/usr/local/bin
IFS="$"
now=$(date +"%Y-%m-%d")
mkdir /backup/s3/$now

cd /var/cpanel/users
find * | while read CPUSER; do
  echo "--- Now processing ${CPUSER} ---"
  nice -n 19 ionice -c3 /scripts/pkgacct --nocompress ${CPUSER} \
    /backup/s3/$now
  /root/s3cmd/s3cmd sync /backup/s3/$now s3://bucket/servername/
  rm /backup/s3/$now/*.tar
  echo
done

rm -r /backup/s3/$now
echo "All done"

The script runs once daily, and we use life cycle rules on the S3 buckets to prune old backup versions that we no longer need. We use standard S3 storage instead of Glacier to avoid having to wait for a backup file to become available for download if we need it.

Each account is transferred to S3 every time in their entirety, so bandwidth usage is high. This is not a concern in our case, but others may have to consider this and modify the script to meet their own needs.

The main drawback is that we need to manually copy an account package from S3 to the server to be able to restore it. It is not difficult to do with s3cmd, but it is a step that we would like to avoid if possible.

cPremote

cPremote is a WHM and cPanel backup plugin developed by an Indian company, Syslint Technologies. The basic idea behind the plugin is to build on the cPanel-provided account packaging system to provide incremental backups that can (optionally) be transferred to a remote location. Once the plugin is installed and configured and backups have run, WHM and cPanel users get some additional capabilities.

WHM users can:

  • Restore entire accounts
  • Restore /home/user, /home/user/public_html, or email folders belonging to any account
  • Enable/disable backups for individual accounts

These operations can be performed directly from remote backup locations, without having to transfer files the the cPanel server first.

Individual databases or files/folders cannot be restored from WHM (but this can be done from the account cPanel, with some limitations)

cPanel users can:

  • Restore own /home/user, /home/user/public_html, or email folders
  • Restore individual files or folders on their account (with limitations)
  • Restore own databases

cPremote places some restrictions on the names of files and folders that can be individually restored, presumably for security reasons. After initially installing the plugin (version 8.0) we quickly discovered that files or folders containing "-" in the file name, e.g. wp-config.php, could not be restored. This has since been corrected with version 8.1 of the software.

Incremental Backups with cPremote

If cPremote is used with local storage (e.g. another disk on the cPanel server), then it works more or less the same as the native cPanel backup system in incremental mode: Initially a full backup is performed, and on the following backup runs only deltas are transferred. Compression is not supported, so the backup consumes the same amount of disk space as the account does.

Unlike the cPanel backup system, cPremote offers two different backup retention schemes for incremental backups:

  • daily / weekly / monthly, where one weekly and one monthly copy is kept in addition to the latest daily backup
  • most recent 1-7 daily backups, i.e you can retain the last 7 daily backups, or you can configure cPremote to keep backups from specific days of the last week (e.g. from Tuesday, Thursday, and Saturday).

The two retention schemes cannot be combined, so if you want to keep several daily backups you cannot also retain weekly and monthly copies.

It is important to be aware that cPremote uses a somewhat unusual definition of "incremental" backups. If you decide to retain a daily, a weekly, and a monthly backup, you will eventually end up with three complete backups that will consume three times the disk space of the accounts on the server. "Incremental" means that each of these copies is maintained incrementally from that point onwards, so when the next daily, weekly or monthly backup is run, only the deltas are transferred to backup storage.

Remote backups

cPremote uses rsync to create and maintain incremental backups, so backups made with cPremote can easily be transferred to a remote location instead of local storage. All you need is a storage server somewhere that supports SSH and rsync. (cPremote does not support s3 or other cloud-based storage).

Just like with local storage, you will need a lot of disk space on the storage server if you plan to retain several backup copies. However, this is less of an issue on a storage server that doesn't need the expensive, low-latency disks you typically want on a web or database server.

cPremote pros and cons

cPremote is an improvement on the native cPanel backup system because backups no longer require significant disk space on the cPanel server itself - there is no longer any risk that local storage will fill up because of a slow or failed transfer to a remote backup location. cPremote also allows restores directly from remote backup locations, without the intermediate step of first transferring the backup copy to the cPanel server.

cPremote does not use compression, so server CPU impact is minimal. After the initial backups are made, only deltas are transferred so bandwidth usage is also kept at a minimum.

Installation is simple, and updates are installed automatically via a weekly cron job. The user interfaces in WHM and cPanel are fairly easy to understand and use, and the ability to restore directly from remote storage is a big plus for this software. The ability for end users to perform their own file and database restores is nice to have, although it is not really necessary in our case.

Restores do not happen immediately - they are scheduled via a cron job that runs every 5 minutes. A notification is sent via email when the restore completes, assuming a valid email address was provided when the restore was requested. cPremote does not allow a default email address to be configured, so this address must be entered every time a restore is requested if a notification is required when the restore completes.

Restoring individual files or folders via cPanel requires the user to write the correct path to the item to be restored, e.g. "/home/user/public_html/wp-content/themes/twentyfifteen/style.css" - there is no graphical interface to help selecting af file or folder for restore. A mistake in a file or folder name will eventually result in an email notification that the restore failed - if a correct email address was provided with the restore request - and the user must then try again.

License pricing is reasonable, USD 5 / month for a single license or USD 3 / month if you buy 10 or more licenses. For comparision, R1Soft license prices are typically USD 5 - 8 per month depending on number of servers and server types.

cPremote has a number of blemishes that are somewhat odd to see in a product that has been around for quite a while. For example, when restoring from WHM or cPanel, the user is presented with a list of backups present on the storage server. The default choice is not the most recent backup, and the select list appears to be sorted alphabetically by day of the week instead of a more reasonable sort by date.

Once a cPanel user clicks on the cPremote icon, there is no link back to the cPanel home page, forcing users to click on the "back" button in the browser or logging in to cPanel again in order to get back.

cPremote backup log files on the server have an odd habit of going missing. It appears there is some cleanup routine running that occasionally deletes the logs, including the most recent one.

Product support can only be described as poor. When we contacted support about the problems we had restoring files with hyphens in the file name that are mentioned above, we just got a cut-and-paste response with content from the website that had nothing whatsoever to do with the issue. When we asked for clarification, we were told that further support could not be provided since this was a trial license (Syslint offers 7x24 support on trials according to their website). The supporter also suggested that we buy Syslint's "cPanel Server Management Plan" if we needed additional assistance.

Upgrading to a paid license did not improve matters. After creating a new ticket under the paid license, we got a response from a "Director - Sales & Marketing" who described this as a "unique issue", i.e. something to do with our server. We were asked to provide root login to the server so they could troubleshoot further, something we were not willing to provide.

The issue was eventually resolved after we posted about it in a thread on a public forum where someone described having a similar issue. But the issue itself and the way Syslint dealt with it does not exactly give us confidence that this is quality software, and we can clearly not count on Syslint support to be competent or helpful.

Finally, it is worth mentioning that cPremote - like the native cPanel backup system - works in a "push" mode where backups are initiated and controlled by the server being backed up. This is a fairly fundamental flaw in a backup system, since something or someone that compromises the server also has the ability to wipe out the backups. This is not an major issue to us since we plan to keep R1Soft (where backups are controlled by the storage server via an agent installed on the cPanel server), but if you are planning to use cPremote or cPanel's native backup system as your main backup, you should consider if this is an acceptable risk for your environment.

We will probably end up keeping cPremote on our servers as a replacement for our S3 backup script, assuming that we do not run into any more issues with the software. It is a good supplement to our R1Soft backups, and we expect that it will make our lives a little easier if we have to restore multiple cPanel accounts from backup quickly. We would not like to rely on it as only backup system, though.