Skip to content
wordpress meta

title: 'Bash Read Json Config File'
date: '2019-12-14T11:31:44-06:00'
status: publish
permalink: /bash-read-json-config-file
author: admin
excerpt: ''
type: post
id: 1437
category:
    - Bash
    - jq
    - json
    - restic
    - Uncategorized
tag: []
post_format: []

Couple of things here:

  • I wanted to do some restic scripts
  • At the same time use a configuration file. The restic developers is working on this functionality for restic and possibly using TOML.

Meanwhile I was trying json since I can definitely use bash/json for other applications. And as you know bash is not great at this kind of thing specifically arrays etc. So this example reads a configuration file and process the json. To further complicate things my json typically need arrays or lists of values as in the restic example you can see for folders, excludes and tags.

You will also note a unique problem with bash. When using while loops with a pipe into the while a subshell is used and you can\'t use variable in your main shell. So my appending to a variable inside the while loop does not produce any strings. In bash 4.2 you can use shopt -s latpipe to get around this. Apparently this is not a problem with ksh.

This is not a working restic script. This is a script to read a configuration file. It just happen to be for something I am going to do with restic.

Example json config file.

```bash $ cat restic-jobs.json { Jobs: [ { jobname: aws-s3, repo: sftp:myuser@192.168.1.112:/TANK/RESTIC-REPO, sets: [ { folders: [ /DATA ], excludes: [ .snapshots,temp], tags: [ data,biz ] }, { folders: [ /ARCHIVE ], excludes: [ .snapshots,temp], tags: [ archive,biz ] } ], quiet: true }, { jobname: azure-onedrive, repo: rclone:azure-onedrive:restic-backups, sets: [ { folders: [ /DATA ], excludes: [ .snapshots,temp], tags: [ data,biz ] }, { folders: [ /ARCHIVE ], excludes: [ .snapshots,temp], tags: [ archive,biz ] } ], quiet: true } ] } ````

Script details.

```bash $ cat restic-jobs.sh

!/bin/bash

v0.9.1

JOB=aws-s3 eval $(jq --arg JOB ${JOB} -r '.Jobs[] | select(.jobname==$JOB) | del(.sets) | to_entries[] | .key + =\ + .value + \' restic-jobs.json) if [[ $jobname == ]]; then echo no job found in config: $JOB exit fi

echo found: $jobname

sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB) | .sets | .[]' restic-jobs.json )

echo

sets=$(jq --arg JOB ${JOB} -r '.Jobs[] | select (.jobname==$JOB)' restic-jobs.json)

backup_jobs=()

need this for bash issue with variables and pipe subshell

shopt -s lastpipe

echo $sets | jq -rc '.sets[]' | while IFS='' read set;do cmd_line=restic backup -q --json 

folders=$(echo $set | jq -r '.folders | .[]')
for st in $folders; do cmd_line+= $st; done
excludes=$(echo $set | jq -r '.excludes | .[]')
for st in $excludes; do cmd_line+= --exclude $st; done
tags=$(echo $set | jq -r '.tags | .[]')
for st in $tags; do cmd_line+= --tag $st; done

backup_jobs+=($cmd_line)

done

for i in ${backup_jobs[@]}; do echo cmd_line: $i done ````

Script run example. Note I am not passing the job name just hard code at the top for my test.

```bash $ ./restic-jobs.sh found: iqonda-aws-s3

cmd_line: restic backup -q --json /DATA --exclude .snapshots --exclude temp --tag iqonda --tag biz cmd_line: restic backup -q --json /ARCHIVE --exclude .snapshots --exclude temp --tag iqonda --tag biz ````