Tar to object storage using rclone

wordpress meta

title: 'Tar to Object Storage Using rclone'
date: '2018-07-18T16:49:29-05:00'
status: publish
permalink: /tar-to-object-storage-using-rclone
author: admin
excerpt: ''
type: post
id: 1228
category:
    - rclone
tag: []
post_format: []

Sometimes using curl and uploading/downloading with an object storage back end will work just fine but in this case I was looking to tar straight into object storage. One option is using rclone with the rcat command. Some example below.

This test was done using Oracle Cloud Infrastructure Object Storage with an Amazon S3 Compatibility API Key. This test consists of:
- 2 196 914 files
- size using df -h 122G
- local tar/gzip file for comparison 52G
- correct rclone.conf setup for the API Key and OCI policies if required for this user

# rclone ls s3_servers_ashburn:SERVERS
 10738097 oci01-20180717_/etc.tgz
  2132252 oci01-20180718_/home/opc.tgz
   286946 oci01-20180717_/home/opc/terraform.tgz

# time tar zcpf - /opt/app2/12.2 | rclone rcat s3_servers_ashburn:SERVERS/oci01-20180718_/opt/app2/12.2.tgz
tar: Removing leading `/' from member names
real    149m48.812s
user    78m13.544s
sys 11m42.817s

# rclone ls s3_servers_ashburn:SERVERS
 10738097 oci01-20180717_/etc.tgz
  2132252 oci01-20180718_/home/opc.tgz
40476682243 oci01-20180718_/opt/app2/12.2.tgz
   286946 ocil01-20180717_/home/opc/terraform.tgz