Organizing Ansible Playbooks

In this post I will describe how we organize DjaoDjin Ansible playbooks for deployment on AWS as well as the rationale that lead to it.

One script to rule them all

If you are building a freemium service, your strategy is to add as many free loaders as possible hoping to convert a percentage of them to paid customers. In that case, building an infrastructure that scales with exponential user growth is important.

Let's get real, micro-services, distributed cloud computing, etc. It all comes at a price. You have to deal with load balancers, aggregate logging, single sign-on solutions for your DevOps team, etc.

The advantage of charging every single customer, as most boutique services do, is that you won't have exponential user growth. If you do, congratulations, now you have exponential revenue growth, and scaling your cloud infrastructure on a shoe string budget is the last of your trouble.

For most paid-for services, we remain big proponent of the the one single production server, everything on it approach. We regularly see businesses scaling easily to $250K / year running on a single server (Just make sure you have a comprehensive backup policy).

How do you deploy code on such systems?

$ ssh *production-machine*
$ sudo -u app git pull
$ sudo service app reload

done.

What does it have to do with Ansible? Well most likely your first playbook is monolithic. It contains the steps to:

1. Create AWS security groups and roles
2. Run an EC2 instance
3. Install and configure packages on the EC2 instance
4. Deploy your application code

Split of Ansible playbooks

In theory, you read all the docs available, write a playbook for your deployment, run it and everything works.

In practice you will first run through two cycles of trial-and-errors: One to provision your AWS cloud infrastructure and one to configure the system and deploy your app on the EC2 instances. (If you are like us, you will mostly take notes along the way so you can write blog posts like the one you reading now.)

This leads to a natural division of playbooks into:

• Provisioning of AWS resources
• Configuration of servers
• Decommisioning of AWS resources

The advantage is that the server configuration playbooks can easily be run on any machine with an IP address, such as a local virtual machine for example.

Provisioning and decommisioning of AWS resources

Provisioning and decommisioning are both sides of the same coin, so we mirror playbooks following the convention:

• Playbooks prefixed with create- provision resources
• Playbooks prefixed with delete- decommission resources

Finally since security groups and IAM roles often out live EC2 instances, we split our provisioning/decommisioning playbooks accordingly.

• aws-create-authorized.yml
• aws-create-instances.yml
• aws-delete-instances.yml
• aws-delete-authorized.yml

A note on EC2 instances and IAM Roles

One of the mistake we repeatedly made early on was to start an EC2 instance, spend time to understand and tweak the system until it kind of did what we wanted, then realized we needed to access a restricted S3 bucket. That meant associating a IAM Role to the running instance. Unfortunately from IAM FAQs:

Q: Can I associate an IAM role with an already running EC2 instance?

No. You can associate only one IAM role with an EC2 instance.

As a result we changed our process such that we always associate both a IAM Role and a Security Group when starting an EC2 instance. So far it seems that a one-to-one relationship between IAM Role and Security Groups works for us.

Surprisingly there does not seem to be an Ansible module at this time (Mar 2015) to create IAM roles, so we will rely on the AWS cli to do so.

Since we are still about using Ansible, we just create bare command as an Ansible task to be run on localhost in roles/aws_create_roles/tasks/main.yml, then adding aws_create_roles as an Ansible Role (not to be confused here) in our playbook.

$ cat assume-role-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

$ cat roles/aws_create_roles/tasks/main.yml
# Create IAM roles
- command: aws iam create-role --role-name my-role --assume-role-policy-document file://assume-role-policy.json
- command: aws iam create-instance-profile --instance-profile-name my-role-profile
- command: aws iam add-role-to-instance-profile --role-name my-role --instance-profile-name my-role-profile

$ diff -u prev aws-create-authorized.yml
  - name: Create IAM roles and security groups
    hosts: localhost
    connection: local
    gather_facts: False
+   roles:
+   - aws_create_roles
    tasks:
    - name: Create 'my-role' security group

$ ansible-playbook -vvvv \
       -i $VIRTUAL_ENV/etc/ansible/hosts aws-create-authorized.yml