Skip to content

Zfsbackup go test with minio server

wordpress meta

title: 'zfsbackup-go test with  minio server'
date: '2019-07-21T08:45:15-05:00'
status: publish
permalink: /zfsbackup-go-test-with-minio-server
author: admin
excerpt: ''
type: post
id: 1391
category:
    - Uncategorized
tag: []
post_format: []

Recording my test with zfsbackup-go. While I am playing around with backup/DR/object storage I also compared the concept here with a previous test around restic/rclone/object storage.

In general ZFS snapshot and replication should work much better with file systems containing huge numbers of files. Most solutions struggle with millions of files and rsync on file level and restic/rclone on object storage level. Walking the tree is just never efficient. So this test works well but has not been scaled yet. I plan to work on that as well as seeing how well the bucket can be synced to different regions.

Minio server

Tip: minio server has a nice browser interface

``` # docker run -p 9000:9000 --name minio1 -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" -v /DATA/minio-repos/:/minio-repos minio/minio server /minio-repos You are running an older version of MinIO released 1 week ago Update: docker pull minio/minio:RELEASE.2019-07-17T22-54-12Z Endpoint: http://172.17.0.2:9000 http://127.0.0.1:9000 Browser Access: http://172.17.0.2:9000 http://127.0.0.1:9000 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide 
</div>**server 1:**

This server simulate our "prod" server. We create an initial data set in /DATA on our server, take snapshot and backup to object storage.

<div class="wp-block-syntaxhighlighter-code ">```

# rsync -a /media/sf_DATA/MyWorkDocs /DATA/

# du -sh /DATA/MyWorkDocs/
1.5G    /DATA/MyWorkDocs/

# export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
# export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# export AWS_S3_CUSTOM_ENDPOINT=http://192.168.1.112:9000
# export AWS_REGION=us-east-1

# zfs snapshot DATA@20190721-0752

# /usr/local/bin/zfsbackup-go send --full DATA s3://zfs-poc
2019/07/21 07:53:12 Ignoring user provided number of cores (2) and using the number of detected cores (1).
Done.
    Total ZFS Stream Bytes: 1514016976 (1.4 GiB)
    Total Bytes Written: 1176757570 (1.1 GiB)
    Elapsed Time: 1m17.522630438s
    Total Files Uploaded: 7

# /usr/local/bin/zfsbackup-go list s3://zfs-poc
2019/07/21 07:56:57 Ignoring user provided number of cores (2) and using the number of detected cores (1).
Found 1 backup sets:

Volume: DATA
    Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)
    Replication: false
    Archives: 6 - 1176757570 bytes (1.1 GiB)
    Volume Size (Raw): 1514016976 bytes (1.4 GiB)
    Uploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s)


There are 4 manifests found locally that are not on the target destination.

server 2:

This server is a possible DR or new server but the idea is somewhere else preferably another cloud region or data center.

``` # export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE # export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY # export AWS_S3_CUSTOM_ENDPOINT=http://192.168.1.112:9000 # export AWS_REGION=us-east-1 # /usr/local/bin/zfsbackup-go list s3://zfs-poc 2019/07/21 07:59:16 Ignoring user provided number of cores (2) and using the number of detected cores (1). Found 1 backup sets: Volume: DATA Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT) Replication: false Archives: 6 - 1176757570 bytes (1.1 GiB) Volume Size (Raw): 1514016976 bytes (1.4 GiB) Uploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s) # zfs list NAME USED AVAIL REFER MOUNTPOINT DATA 2.70M 96.4G 26K /DATA # zfs list -t snapshot no datasets available # ls /DATA/ ** using -F. This is a COLD DR style test with no existing infrastructure/ZFS sets on target systems # /usr/local/bin/zfsbackup-go receive --auto DATA s3://zfs-poc DATA -F 2019/07/21 08:05:28 Ignoring user provided number of cores (2) and using the number of detected cores (1). 2019/07/21 08:06:42 Done. Elapsed Time: 1m13.968871681s 2019/07/21 08:06:42 Done. # ls /DATA/ MyWorkDocs # du -sh /DATA/MyWorkDocs/ 1.5G /DATA/MyWorkDocs/ # zfs list NAME USED AVAIL REFER MOUNTPOINT DATA 1.41G 95.0G 1.40G /DATA # zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT DATA@20190721-0752 247K - 1.40G - 
</div>That concludes one test. In theory that is a cold DR situation where you have nothing really ready until you need it. So think build a server and recover /DATA from zfs backup in object storage. So initial restore will be very long depending on your size.

Read on if you are thinking you want to go more towards pilot light or warm DR we can run incremental backups, then on the target server keep receiving snapshots periodically into our target ZFS file system DATA. You may observe why not just do real ZFS send/receive an no object storage in between. There is no good answer except there are many ways you could solve DR and this is one of them. In this case I could argue object storage is cheap and has some very good redundancy/availability features. And your replication between regions may be using a back haul very fast/cheap channel where your VPN or fastconnect WAN between regions may be slow and/or expensive.

You could also be thinking something between cold and warm DR is where you want to be and therefore only apply the full DATA receive when you are ready. That could mean a lot of snapshots likely to apply afterwards. Or maybe not I have not checked on that aspect of a recovery process.

Regardless I like the idea of leveraging zfs with object storage so you may not have a use for this but I definitely will.

**Incremental snapshots:**

**server 1:**

Add more data to source, snapshot and backup to object storage.

<div class="wp-block-syntaxhighlighter-code ">```

# rsync -a /media/sf_DATA/MySrc /DATA/
# du -sh /DATA/MySrc/
1.1M    /DATA/MySrc/

# zfs snapshot DATA@20190721-0809
# zfs list -t snapshot
NAME                 USED  AVAIL  REFER  MOUNTPOINT
DATA@20190721-0752    31K      -  1.40G  -
DATA@20190721-0809     0B      -  1.41G  -

# /usr/local/bin/zfsbackup-go send --increment DATA s3://zfs-poc
2019/07/21 08:10:49 Ignoring user provided number of cores (2) and using the number of detected cores (1).
Done.
    Total ZFS Stream Bytes: 1202792 (1.1 MiB)
    Total Bytes Written: 254909 (249 KiB)
    Elapsed Time: 228.123591ms
    Total Files Uploaded: 2

# /usr/local/bin/zfsbackup-go list s3://zfs-poc
2019/07/21 08:11:17 Ignoring user provided number of cores (2) and using the number of detected cores (1).
Found 2 backup sets:

Volume: DATA
    Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)
    Replication: false
    Archives: 6 - 1176757570 bytes (1.1 GiB)
    Volume Size (Raw): 1514016976 bytes (1.4 GiB)
    Uploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s)


Volume: DATA
    Snapshot: 20190721-0809 (2019-07-21 08:09:47 -0500 CDT)
    Incremental From Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT)
    Intermediary: false
    Replication: false
    Archives: 1 - 254909 bytes (249 KiB)
    Volume Size (Raw): 1202792 bytes (1.1 MiB)
    Uploaded: 2019-07-21 08:10:49.3280703 -0500 CDT (took 214.139056ms)

There are 4 manifests found locally that are not on the target destination.

server 2:

``` # /usr/local/bin/zfsbackup-go list s3://zfs-poc 2019/07/21 08:11:44 Ignoring user provided number of cores (2) and using the number of detected cores (1). Found 2 backup sets: Volume: DATA Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT) Replication: false Archives: 6 - 1176757570 bytes (1.1 GiB) Volume Size (Raw): 1514016976 bytes (1.4 GiB) Uploaded: 2019-07-21 07:53:12.42972167 -0500 CDT (took 1m16.313538867s) Volume: DATA Snapshot: 20190721-0809 (2019-07-21 08:09:47 -0500 CDT) Incremental From Snapshot: 20190721-0752 (2019-07-21 07:52:31 -0500 CDT) Intermediary: false Replication: false Archives: 1 - 254909 bytes (249 KiB) Volume Size (Raw): 1202792 bytes (1.1 MiB) Uploaded: 2019-07-21 08:10:49.3280703 -0500 CDT (took 214.139056ms) ** not sure why I need to force (-F) maybe because data set is mounted? message like this: ** cannot receive incremental stream: destination DATA has been modified since most recent snapshot *** 2019/07/21 08:12:25 Error while trying to read from volume DATA|20190721-0752|to|20190721-0809.zstream.gz.vol1 - io: read/write on closed pipe # /usr/local/bin/zfsbackup-go receive --auto DATA s3://zfs-poc DATA -F 2019/07/21 08:12:53 Ignoring user provided number of cores (2) and using the number of detected cores (1). 2019/07/21 08:12:54 Done. Elapsed Time: 379.712693ms 2019/07/21 08:12:54 Done. # ls /DATA/ MySrc MyWorkDocs # du -sh /DATA/MySrc/ 1.1M /DATA/MySrc/ # zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT DATA@20190721-0752 30K - 1.40G - DATA@20190721-0809 34K - 1.41G - ````

LINK: https://github.com/someone1/zfsbackup-go