Object storage with restic and rclone

wordpress meta

title: 'Object Storage with Restic and Rclone'
date: '2018-08-03T14:14:04-05:00'
status: publish
permalink: /object-storage-with-restic-and-rclone
author: admin
excerpt: ''
type: post
id: 1235
category:
    - Backups
    - OCI
    - rclone
    - restic
tag: []
post_format: []

I have been playing around with some options to utilize Object Storage for backups. Since I am working on Oracle Cloud Infrastructure (OCI) I am doing my POC using the OCI Object Storage. OCI object storage does have Swift and S3 Compatibility API's to interface with. Of course if you want commercial backups many of them can use object storage as back-ends now so that would be the correct answer. If your needs does not warrant commercial backups solutions you can try several things. A few options I played with.

1. Bareos server/client with the object storage droplet. Not working reliably. Too experimental with droplet?
2. Rclone and using tar to pipe with rclone's rcat feature. This works well but is not a backup solution as in incrementals etc.
3. Duplicati. In my case using rclone as connection since S3 interface on OCI did not work.
4. Dupliciti. Could not get this one to work to S3 interface on OCI.
5. Restic. In my case using rclone as connection since S3 interface on OCI did not work.

So far duplicati was not bad but had some bugs. It is beta software so probably should expect problems. Restic is doing a good job so far and I show a recipe of my POC below:

Out of scope is setting up rclone, rclone.conf. Make sure you test that rclone is accessing your bucket first.

Restic binary

# wget https://github.com/restic/restic/releases/download/v0.9.1/restic_0.9.1_linux_amd64.bz2
2018-08-03 10:25:10 (3.22 MB/s) - ‘restic_0.9.1_linux_amd64.bz2’ saved [3786622/3786622]
# bunzip2 restic_0.9.1_linux_amd64.bz2 
# mv restic_0.9.1_linux_amd64 /usr/local/bin/
# chmod +x /usr/local/bin/restic_0.9.1_linux_amd64 
# mv /usr/local/bin/restic_0.9.1_linux_amd64 /usr/local/bin/restic
# /usr/local/bin/restic version
restic 0.9.1 compiled with go1.10.3 on linux/amd64

Initialize repo

# rclone ls s3_servers_phoenix:oci02a
# export RESTIC_PASSWORD="WRHYEjblahblah0VWq5qM"
# /usr/local/bin/restic -r rclone:s3_servers_phoenix:oci02a init
created restic repository 2bcf4f5864 at rclone:s3_servers_phoenix:oci02a

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

# rclone ls s3_servers_phoenix:oci02a
      155 config
      458 keys/530a67c4674b9abf6dcc9e7b75c6b319187cb8c3ed91e6db992a3e2cb862af63

Run a backup

# time /usr/local/bin/restic -r rclone:s3_servers_phoenix:oci02a backup /opt/applmgr/12.2
repository 2bcf4f58 opened successfully, password is correct

Files:       1200934 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Added:      37.334 GiB

processed 1200934 files, 86.311 GiB in 1:31:40
snapshot af4d5598 saved

real    91m40.824s
user    23m4.072s
sys 7m23.715s

# /usr/local/bin/restic -r rclone:s3_servers_phoenix:oci02a snapshots
repository 2bcf4f58 opened successfully, password is correct
ID        Date                 Host              Tags        Directory
----------------------------------------------------------------------
af4d5598  2018-08-03 10:35:45  oci02a              /opt/applmgr/12.2
----------------------------------------------------------------------
1 snapshots

Run second backup

# /usr/local/bin/restic -r rclone:s3_servers_phoenix:oci02a backup /opt/applmgr/12.2
repository 2bcf4f58 opened successfully, password is correct

Files:           0 new,     0 changed, 1200934 unmodified
Dirs:            0 new,     0 changed,     2 unmodified
Added:      0 B  

processed 1200934 files, 86.311 GiB in 47:46
snapshot a158688a saved

Example cron entry

# crontab -l
05 * * * * /usr/local/bin/restic -r servers_phoenix:oci02a backup -q /usr; /usr/local/bin/restic -r servers_phoenix:oci02a forget -q --prune --keep-hourly 2 --keep-daily 7