Object storage listing with rclone and jq

wordpress meta

title: 'Object Storage Listing with Rclone and jq'
date: '2019-01-04T10:59:31-06:00'
status: publish
permalink: /object-storage-listing-with-rclone-and-jq
author: admin
excerpt: ''
type: post
id: 1282
category:
    - rclone
tag: []
post_format: []

Some examples of using rclone and jq to see object listing in a bucket. These examples was using Oracle (OCI) Object Storage but since this is rclone it should not matter what the target is.

Rclone retrieving JSON object listing of a bucket:

``` $ rclone lsjson -R s3_ashburn:APPS [ {"Path":"config","Name":"config","Size":155,"MimeType":"application/octet-stream","ModTime":"2018-11-02T18:01:31.028653533Z","IsDir":false}, {"Path":"data","Name":"data","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533157179Z","IsDir":true}, {"Path":"index","Name":"index","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533226556Z","IsDir":true}, {"Path":"keys","Name":"keys","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533246534Z","IsDir":true}, {"Path":"snapshots","Name":"snapshots","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533266804Z","IsDir":true}, {"Path":"index/6f0870dc3d699c0e550f62c535f11a3e52396f45d9c3439760a5f648ee2f1533","Name":"6f0870dc3d699c0e550f62c535f11a3e52396f45d9c3439760a5f648ee2f1533","Size":37828 350,"MimeType":"application/octet-stream","ModTime":"2019-01-03T21:27:05Z","IsDir":false}, {"Path":"index/b20a6e07f25d834739e3c3fd82cf3b7ade3e7f1f0f286aab61006532621220ae","Name":"b20a6e07f25d834739e3c3fd82cf3b7ade3e7f1f0f286aab61006532621220ae","Size":36726 493,"MimeType":"application/octet-stream","ModTime":"2019-01-03T21:27:02Z","IsDir":false},

</div>**Use jq select to grab older than certain dates:**

<div class="wp-block-syntaxhighlighter-code ">```

$ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | select (."ModTime" < "2018-12-01")|.Name'
ffea09b644533ddcde68a93095bc512646fd0ac0557d39e6e06e004bf73b6bed
ffef7980ade85ea2d9b436c40df46384bbbe8e7e6e71219aff0757ad90f1652f
fff3f56e384ab055c3aa4b6e2dd527c368bf2280863d357e577402460fe9d41a

Use jq csv filter and specific fields

``` $ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | [.Name,.Size] | @csv'

</div>**Use jq select for older than certain date, specific fields and csv**

<div class="wp-block-syntaxhighlighter-code ">```

$ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | select (."ModTime" < "2018-11-01") | [.Name,.Size,.ModTime] | @csv'

Rclone size

``` $ rclone size s3_ashburn:APPS --json {"count":8088,"bytes":38670955795}

</div>**Rclone size and jq csv filter**

<div class="wp-block-syntaxhighlighter-code ">```

$ rclone size s3_ashburn:APPS --json | jq -r '[.count,.bytes] | @csv'
8088,38670955795