wordpress meta

title: 'Terraform Stateserver Using Python'
date: '2021-03-22T09:22:34-05:00'
status: publish
permalink: /terraform-stateserver-using-python
author: admin
excerpt: ''
type: post
id: 1722
category:
    - Python
    - Terraform
tag: []
post_format: []

Terraform can utilize a http backend for maintaining state. This is a test of a Terraform http backend using a server implemented with python.

NOTE: checked source into https://github.com/rrossouw01/terraform-stateserver-py/

recipe and components

Using Virtualbox Ubuntu 20.10 and followed links:

setup

```bash $ mkdir tf-state-server $ cd tf-state-server

$ virtualenv -p python3 venv created virtual environment CPython3.8.6.final.0-64 in 174ms creator CPython3Posix(dest=/home/rrosso/tf-state-server/venv, clear=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/rrosso/.local/share/virtualenv) added seed packages: pip==20.1.1, pkg_resources==0.0.0, setuptools==44.0.0, wheel==0.34.2 activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

$ source venv/bin/activate

(venv) $ pip install -U -r requirements.txt Collecting flask Using cached Flask-1.1.2-py2.py3-none-any.whl (94 kB) Collecting flask_restful Downloading Flask_RESTful-0.3.8-py2.py3-none-any.whl (25 kB) Collecting itsdangerous>=0.24 Using cached itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB) Collecting Werkzeug>=0.15 Using cached Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB) Collecting Jinja2>=2.10.1 Using cached Jinja2-2.11.3-py2.py3-none-any.whl (125 kB) Collecting click>=5.1 Using cached click-7.1.2-py2.py3-none-any.whl (82 kB) Collecting pytz Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB) |████████████████████████████████| 510 kB 3.0 MB/s Collecting six>=1.3.0 Using cached six-1.15.0-py2.py3-none-any.whl (10 kB) Collecting aniso8601>=0.82 Downloading aniso8601-9.0.1-py2.py3-none-any.whl (52 kB) |████████████████████████████████| 52 kB 524 kB/s Collecting MarkupSafe>=0.23 Using cached MarkupSafe-1.1.1-cp38-cp38-manylinux2010_x86_64.whl (32 kB) Installing collected packages: itsdangerous, Werkzeug, MarkupSafe, Jinja2, click, flask, pytz, six, aniso8601, flask-restful Successfully installed Jinja2-2.11.3 MarkupSafe-1.1.1 Werkzeug-1.0.1 aniso8601-9.0.1 click-7.1.2 flask-1.1.2 flask-restful-0.3.8 itsdangerous-1.1.0 pytz-2021.1 six-1.15.0

(venv) $ python3 stateserver.py * Serving Flask app "stateserver" (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: off * Running on http://192.168.1.235:5000/ (Press CTRL+C to quit) ... ````

terraform point to remote http

```bash ➜ cat main.tf terraform { backend "http" { address = "http://192.168.1.235:5000/terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a" lock_address = "http://192.168.1.235:5000/terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a" lock_method = "PUT" unlock_address = "http://192.168.1.235:5000/terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a" unlock_method = "DELETE" } }

➜ source ../env-vars

➜ terraform init

Initializing the backend... Do you want to copy existing state to the new backend? Pre-existing state was found while migrating the previous "local" backend to the newly configured "http" backend. No existing state was found in the newly configured "http" backend. Do you want to copy this state to the new "http" backend? Enter "yes" to copy and "no" to start with an empty state.

Enter a value: yes

Successfully configured the backend "http"! Terraform will automatically use this backend unless the backend configuration changes.

Initializing provider plugins...

The following providers do not have any version constraints in configuration, so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below.

provider.oci: version = "~> 4.17"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.

If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.

server shows

````bash ... 192.168.1.111 - - [16/Mar/2021 10:50:00] "POST /terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a?ID=84916e49-1b44-1b32-2058-62f28e1e8ee7 HTTP/1.1" 200 - 192.168.1.111 - - [16/Mar/2021 10:50:00] "DELETE /terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 - 192.168.1.111 - - [16/Mar/2021 10:50:00] "GET /terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 - ...

$ ls -la .stateserver/ total 24 drwxrwxr-x 2 rrosso rrosso 4096 Mar 16 10:50 . drwxrwxr-x 4 rrosso rrosso 4096 Mar 16 10:43 .. -rw-rw-r-- 1 rrosso rrosso 4407 Mar 16 10:50 4cdd0c76-d78b-11e9-9bea-db9cd8374f3a -rw-rw-r-- 1 rrosso rrosso 5420 Mar 16 10:50 4cdd0c76-d78b-11e9-9bea-db9cd8374f3a.log

$ more .stateserver/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a* :::::::::::::: .stateserver/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a :::::::::::::: { "lineage": "9b756fb7-e41a-7cd6-d195-d794f377e7be", "outputs": {}, "resources": [ { "instances": [ { "attributes": { "compartment_id": null, "id": "ObjectStorageNamespaceDataSource-0", "namespace": "axwscg6apasa" }, "schema_version": 0 } ], ... "serial": 0, "terraform_version": "0.12.28", "version": 4 } :::::::::::::: .stateserver/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a.log :::::::::::::: lock: { "Created": "2021-03-16T15:49:34.567178267Z", "ID": "7eea0d70-5f53-e475-041b-bcc393f4a92d", "Info": "", "Operation": "migration destination state", "Path": "", "Version": "0.12.28", "Who": "rrosso@desktop01" } unlock: { "Created": "2021-03-16T15:49:34.567178267Z", "ID": "7eea0d70-5f53-e475-041b-bcc393f4a92d", "Info": "", "Operation": "migration destination state", "Path": "", "Version": "0.12.28", "Who": "rrosso@desktop01" } lock: { "Created": "2021-03-16T15:49:45.760917508Z", "ID": "84916e49-1b44-1b32-2058-62f28e1e8ee7", "Info": "", "Operation": "migration destination state", "Path": "", "Version": "0.12.28", "Who": "rrosso@desktop01" } state_write: { "lineage": "9b756fb7-e41a-7cd6-d195-d794f377e7be", "outputs": {}, "resources": [ { "instances": [ { "attributes": { "compartment_id": null, "id": "ObjectStorageNamespaceDataSource-0", "namespace": "axwscg6apasa" }, "schema_version": 0 } ... ````

stateserver shows during plan

```bash ... 192.168.1.111 - - [16/Mar/2021 10:54:12] "PUT /terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 - 192.168.1.111 - - [16/Mar/2021 10:54:12] "GET /terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 - 192.168.1.111 - - [16/Mar/2021 10:54:15] "DELETE /terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 - ... ````

source

```bash $ more requirements.txt stateserver.py :::::::::::::: requirements.txt :::::::::::::: flask flask_restful :::::::::::::: stateserver.py ::::::::::::::

!/usr/bin/python3

import flask import flask_restful import json import logging import os

app = flask.Flask(name) api = flask_restful.Api(app)

@app.before_request def log_request_info(): headers = [] for header in flask.request.headers: headers.append('%s = %s' % (header[0], header[1]))

body = flask.request.get_data().decode('utf-8').split('\n')

app.logger.debug(('%(method)s for %(url)s...\n'
                  '    Header -- %(headers)s\n'
                  '    Body -- %(body)s\n')
                 % {
    'method': flask.request.method,
    'url': flask.request.url,
    'headers': '\n    Header -- '.join(headers),
    'body': '\n           '.join(body)
})

class Root(flask_restful.Resource): def get(self): resp = flask.Response( 'Oh, hello', mimetype='text/html') resp.status_code = 200 return resp

class StateStore(object): def init(self, path): self.path = path os.makedirs(self.path, exist_ok=True)

def _log(self, id, op, data):
    log_file = os.path.join(self.path, id) + '.log'
    with open(log_file, 'a') as f:
        f.write('%s: %s\n' %(op, data))

def get(self, id):
    file = os.path.join(self.path, id)
    if os.path.exists(file):
        with open(file) as f:
            d = f.read()
            self._log(id, 'state_read', {})
            return json.loads(d)
    return None

def put(self, id, info):
    file = os.path.join(self.path, id)
    data = json.dumps(info, indent=4, sort_keys=True)
    with open(file, 'w') as f:
        f.write(data)
        self._log(id, 'state_write', data)

def lock(self, id, info):
    # NOTE(mikal): this is racy, but just a demo
    lock_file = os.path.join(self.path, id) + '.lock'
    if os.path.exists(lock_file):
        # If the lock exists, it should be a JSON dump of information about
        # the lock holder
        with open(lock_file) as f:
            l = json.loads(f.read())
        return False, l

    data = json.dumps(info, indent=4, sort_keys=True)
    with open(lock_file, 'w') as f:
        f.write(data)
    self._log(id, 'lock', data)
    return True, {}

def unlock(self, id, info):
    lock_file = os.path.join(self.path, id) + '.lock'
    if os.path.exists(lock_file):
        os.unlink(lock_file)
        self._log(id, 'unlock', json.dumps(info, indent=4, sort_keys=True))
        return True
    return False

state = StateStore('.stateserver')

class TerraformState(flask_restful.Resource): def get(self, tf_id): s = state.get(tf_id) if not s: flask.abort(404) return s

def post(self, tf_id):
    print(flask.request.form)
    s = state.put(tf_id, flask.request.json)
    return {}

class TerraformLock(flask_restful.Resource): def put(self, tf_id): success, info = state.lock(tf_id, flask.request.json) if not success: flask.abort(423, info) return info

def delete(self, tf_id):
    if not state.unlock(tf_id, flask.request.json):
        flask.abort(404)
    return {}

api.add_resource(Root, '/') api.add_resource(TerraformState, '/terraform_state/') api.add_resource(TerraformLock, '/terraform_lock/')

if name == 'main': # Note this is not run with the flask task runner... app.log = logging.getLogger('werkzeug') app.log.setLevel(logging.DEBUG) #app.run(host='0.0.0.0', debug=True) ````