Object storage with duplicity and rclone

wordpress meta

title: 'Object Storage with Duplicity and Rclone'
date: '2018-08-07T10:26:26-05:00'
status: publish
permalink: /object-storage-with-duplicity-and-rclone
author: admin
excerpt: ''
type: post
id: 1240
category:
    - Dupliciti
    - OCI
    - rclone
tag: []
post_format: []

At this point I prefer using restic for my object storage backup needs but since I did a POC for duplicity and specifically using rclone with duplicity I am writing down my notes. A good description of duplicity and restic here:

Backing Up Linux to Backblaze B2 with Duplicity and Restic

We’re highlighting Duplicity and Restic because they exemplify two different philosophical approaches to data backup: “Old School” (Duplicity) vs “New School” (Restic).

Since I am doing my tests with Oracle Cloud Infrastructure (OCI) Object Storage and so far it's Amazon S3 Compatibility Interface does not work out of the box with most tools except with rclone, I am using rclone as a backend. With restic using rclone as a back-end worked pretty smooth but duplicity does not have good rclone support so I used a python back-end written by Francesco Magno and hosted here: https://github.com/GilGalaad/duplicity-rclone/blob/master/README.md

I had a couple issues with getting duplicity to work with this back-end so I will show how to get around it.

First:
1. Make sure rclone is working with your rclone config and can at least "ls" your bucket.
2. Setup a gpg key.
3. Copy rclonebackend.py to duplicity backends folder. In my case /usr/lib64/python2.7/site-packages/duplicity/backends

# PASSPHRASE="mypassphrase" duplicity --encrypt-key 094CA414 /tmp rclone://mycompany-POC-phoenix:dr01-duplicity
InvalidBackendURL: Syntax error (port) in: rclone://mycompany-POC-phoenix:dr01-duplicity AFalse BNone Cmycompany-POC-phoenix:dr01-duplicity

## Hack backends.py

# diff /usr/lib64/python2.7/site-packages/duplicity/backend.py /tmp/backend.py 
303c303
< if not (self.scheme in ['rsync'] and re.search('::[^:]*$', self.url_string) or (self.scheme in ['rclone']) ): --- >             if not (self.scheme in ['rsync'] and re.search('::[^:]*$', self.url_string)):

# PASSPHRASE="mypassphrase" duplicity --encrypt-key 094CA414 /tmp rclone://mycompany-POC-phoenix:dr01-duplicity
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1533652997.49 (Tue Aug  7 14:43:17 2018)
EndTime 1533653022.35 (Tue Aug  7 14:43:42 2018)
ElapsedTime 24.86 (24.86 seconds)
SourceFiles 50
SourceFileSize 293736179 (280 MB)
NewFiles 50
NewFileSize 136467418 (130 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 50
RawDeltaSize 293723433 (280 MB)
TotalDestinationSizeChange 279406571 (266 MB)
Errors 0
-------------------------------------------------

# rclone ls mycompany-POC-phoenix:dr01-duplicity
  1773668 duplicity-full-signatures.20180807T144317Z.sigtar.gpg
      485 duplicity-full.20180807T144317Z.manifest.gpg
209763240 duplicity-full.20180807T144317Z.vol1.difftar.gpg
 69643331 duplicity-full.20180807T144317Z.vol2.difftar.gpg

# PASSPHRASE="mypassphrase" duplicity --encrypt-key 094CA414 collection-status rclone://mycompany-POC-phoenix:dr01-duplicity
Last full backup date: Tue Aug  7 14:43:17 2018
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /root/.cache/duplicity/df529824ba5d10f9e31329e440c5efa6

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Tue Aug  7 14:43:17 2018
Chain end time: Tue Aug  7 14:50:12 2018
Number of contained backup sets: 2
Total number of contained volumes: 3
 Type of backup set:                            Time:      Num volumes:
                Full         Tue Aug  7 14:43:17 2018                 2
         Incremental         Tue Aug  7 14:50:12 2018                 1
-------------------------
No orphaned or incomplete backup sets found.