Rclone and oci s3 interface

wordpress meta

title: 'Rclone and OCI S3 Interface'
date: '2018-03-29T09:35:03-05:00'
status: publish
permalink: /rclone-and-oci-s3-interface
author: admin
excerpt: ''
type: post
id: 1191
category:
    - OCI
    - rclone
tag: []
post_format: []

I am testing rclone to the Oracle Cloud Interface object storage and recording what worked for me.

Note I could not get the swift interface to work with rclone, duplicity or swiftclient yet. Although straightforward curl does work to the swift interface.

rclone configuration generated with rclone config

# cat /root/.config/rclone/rclone.conf
[s3_backups]
type = s3
env_auth = false
access_key_id = ocid1.credential.oc1..a<redacted>ta
secret_access_key = K<redacted>6s=
region = us-ashburn-1
endpoint = <tenancy>.compat.objectstorage.us-ashburn-1.oraclecloud.com
location_constraint = 
acl = private
server_side_encryption = 
storage_class = 

Issue with max-keys. This problem although very difficult to find was also preventing copy/sync of folders although a single file was working. rclone v1.36 was installed form Ubuntu repos and issue resolved with newer version.

# rclone ls s3_backups:repo1
2018/03/29 08:55:44 Failed to ls: InvalidArgument: The 'max-keys' parameter must be between 1 and 1000 (it was 1024) status code: 400, request id: fa704a55-44a8-1146-1b62-688df0366f63

Update and try again.

# curl https://rclone.org/install.sh | sudo bash
[..]
rclone v1.40 has successfully installed.

# rclone -V
rclone v1.40
- os/arch: linux/amd64
- go version: go1.10

# rclone ls s3_backups:repo1
      655 config
       38 hints.3

# rclone copy /root/backup/repo1 s3_backups:repo1

# rclone sync /root/backup/repo1 s3_backups:repo1

# rclone ls s3_backups:repo1
       26 README
      655 config
       38 hints.3
    82138 index.3
  5245384 data/0/1
  3067202 data/0/3

# rclone lsd s3_backups:
          -1 2018-03-27 21:07:11        -1 backups
          -1 2018-03-29 13:39:42        -1 repo1
          -1 2018-03-26 22:23:35        -1 terraform
          -1 2018-03-27 14:34:55        -1 terraform-src

References:
https://rclone.org/docs/
https://docs.us-phoenix-1.oraclecloud.com/api/#/en/s3objectstorage/20160918/

Rclone: Rsync for Cloud Storage

In a future article I will add my testing around BorgBackup + rclone + OCI objectstorage from this interesting idea: https://opensource.com/article/17/10/backing-your-machines-borg