Using Make for SysAdmin Work

[stacksofplates]

When most people hear about make they think of compiling source code. While that is often what it's used for, there are other good uses as well. Here's an example of using it with Terraform/Ansible to build your infrastructure. Make has some advantages over just using a shell script. One is the native concept of dependencies. You can easily define which targets are dependencies of other targets. Make also has some idempotence baked in (as long as everything isn't a phony target). Another advantage is that parallelism is built directly into make. You can run targets in parallel with other targets as long as they don't have dependencies.

So first off here's the contents of the Makefile:

.PHONY: clean download plan apply inventory roles destroy

all: download plan apply inventory roles

download:
        test -s terraform || wget https://releases.hashicorp.com/terraform/0.12.3/terraform_0.12.3_linux_amd64.zip -O terraform.zip
        test -s terraform || unzip terraform.zip

plan: download
        cd $(env)/services && \
        ../../terraform init -backend-config="bucket=bucket-name"
        cd $(env)/services && \
        ../../terraform plan -out plan

apply: plan
        cd $(env)/services && \
        ../../terraform apply -input=false plan

destroy: download
        cd $(env)/services && \
        ../../terraform init -backend-config="bucket=bucket-name" && \
        ../../terraform destroy -auto-approve

inventory: apply
        cd $(env)/services && \
        cp inventory ../../ansible/

roles:
        git clean -fdx ansible/roles/
        ansible-galaxy install -r ansible/roles/requirements.yml

clean:
        git clean -fd
        git clean -fx
        git reset --hard

It's important to note that you could rename the download target as terraform and make it a file target instead of phony. This way you wouldn't need the test -s terraform statement. Make will look to make sure that a file called the name of the target is created before the target is run. If that file exists, make won't run the target. This is where it's built in idempotence comes in to play.

If you look after each target you will see a name of another target. These are the dependencies for that target. So plan depends on download, apply depends on plan, and so on. However roles, clean, and download don't have any dependencies.

Since the all target is first, if you just run make it would be the same as running make all which includes all of the targets listed beside all. We can run them in parallel with make -j. If you don't give make a number after the -j it will run as many in parallel as possible. If you give it a number it will only run that many jobs at once.

For this Makefile, if we run make -j env=dev make will download Terraform and unzip it while also downloading the Ansible roles we have defined both in parallel. Then it runs terraform init with the specific bucket we put in the Makefile. It then runs terraform plan and writes that to a file named plan. After that target is finished, it runs terraform apply using the plan file we created. Once that target is finished, it copies the inventory file that Terraform created to a directory for Ansible to use afterward.

This workflow is really nice for both local development and through a pipeline. Now you don't have to edit your CI/CD pipeline stages directly you can just edit the Makefile and only have one or two stages in the pipeline. And if you're running locally you can run each target independently and it will only run the dependencies that specific target needs.

This specific Makefile also lets you destroy the infrastructure by running make destroy env=dev (or whatever the directory name for your Terraform environment is). And if you want to wipe out your local changes, you can run make clean and it will reset your local git repo to wherever HEAD is pointed.