Problems

• Suspect SLA (unplanned/poorly planned downtime)
• Scaling Cost
• Not enough control/insight

Why AWS?

• SLA
• Cheap (relatively)
• Unmanaged Options
• API (Necessary for having an effective custom solution)

What we needed

• Ability to access the project shell

• Ability to easily scale

• Deployments from source control

• Ability to set environment variables

Fabric is a python command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

We use it to provide an easy interface for our developers to interact with Ansible and Boto.

fab prod shell

fab - Invokes a fabric command from the fabfile

prod - Sets fabric's environment dictionary to production

shell - Custom fabric method that will cd into the project directory on the host specified by the environment dictionary, and run the project shell

def shell():
    '''
    Calls the shell_plus of the TrackMaven project
    '''
    with env.cd(env.project_path):
        env.run('foreman run python trackmaven/manage.py shell_plus')

fab prod shell

Python module that interacts with the infrastructural services of AWS.

Enables us to:

• Add/Remove instances

• Control the load balancers

• Create and manage images
• Fetch instance data

Tagging

• Env - Prod, Puppystream, Puppycrate
• Type - Web, Celery, Manager
• Name - EnvType# (prodcelery1)

fab prod scale:web,3

AWSManager

from boto import ec2
from boto.ec2 import elb


class AWSManager():
    REGION = 'us-east-1'

    def __init__(self, env):
        if not env:
            raise Exception('No environment set')
        self.env = env
        self.ec2 = ec2.connect_to_region(self.REGION,

                 profile_name="trackmaven")
        self.elb = elb.connect_to_region(self.REGION,

                 profile_name="trackmaven")


    def scale_web(self, number):
        '''
        Scales the web workers to the number inputted
        '''
        web_instances = self._get_instances('web')
        if number > len(web_instances):
            self._scale_up('web', number - len(web_instances))
        elif number < len(web_instances) and number >= 2:
            self._scale_down('web', len(web_instances) - number)
        else:
            raise Exception('Cannot scale below 2 web instances or scale the same number')

    def _scale_up(self, instance_type, number):
        '''
        Scales the specified instances up
        '''
        # Fetch the main instance to base the new instances off of
        instances = self._get_instances(instance_type)
        main = next(instance for instance in instances
                    if '1' in instance.tags.get('Name'))
        image = self.ec2.get_all_images(owners='self', filters={
            'tag:Name': main.tags.get('Name')}).pop()
        for i in range(0, number):
            reservation = self.ec2.run_instances(
                image.id,
                key_name=main.key_name,
                security_groups=[group.name for group in main.groups],
                instance_type=main.instance_type,
                placement=main.placement
            )
            instance = reservation.instances[0]
            # Add the proper tags
            instance.add_tag('env', self.env)
            instance.add_tag('type', instance_type)
            instance.add_tag('Name', '{}{}{}'.format(
                self.env, instance_type, len(instances) + 1 + (i * 1)))

            # If it is a web worker, register it with the elb
            if instance_type == 'web':
                self.elb.register_instances(self.env, [instance.id])

Ansible is an open source orchestration engine that manages nodes over SSH. (Built on Python)

Playbooks

"Playbooks are the language by which Ansible orchestrates, configures, administers, or deploys systems."

Playbooks consist of plays which are mappings between a set of hosts and the tasks which run on those hosts.

• Human Readable

• YAML(With Jinja2)

• Synchronous/Asynchronous

• Extensive Module Support

Modules

Playbook Example

---
- hosts: webservers
  vars:
    http_port: 80
    max_clients: 200
  remote_user: root
  tasks:
  - name: ensure apache is at the latest version
    yum: pkg=httpd state=latest
  - name: write the apache config file
    template: src=/srv/httpd.j2 dest=/etc/httpd.conf
    notify:
    - restart apache
  - name: ensure apache is running
    service: name=httpd state=started
  handlers:
    - name: restart apache
      service: name=httpd state=restarted

Inventory

A file that describes Hosts and Groups in Ansible (INI or JSON)

mail.example.com

[webservers]
foo.example.com
bar.example.com

[dbservers]
one.example.com
two.example.com
three.example.com

[webservers:vars]
some_var=somevalue

fab prod deploy

ENVRIONMENT=prod ansible-playbook deployment/playbooks/deploy.yml
-i deployment/hosts -e "branch=master"

-i deployment/hosts

from boto import ec2
import os, json


def main():
    instances = {}
    instances['_meta'] = { 'hostvars': {}}

    env = os.environ.get('ENVIRONMENT')
    conn = ec2.connect_to_region('us-east-1', profile_name='trackmaven')
    prod_reservations = ec2.get_all_instances(filters={'tag:env': env})

    for reservation in prod_reservations:
      for instance in reservation.instances:
        instance_type = instance.tags.get('type')
        # Group instances by type
        instances.setdefault(instance_type, 
          []).append(instance.public_dns_name)
        # Group instances by name
        instances.setdefault(instance.tags.get('Name'), 
          []).append(instance.public_dns_name)
        # Set instance specific variables
        instances['_meta']['hostvars'][instance.public_dns_name] = {
          'type': instance_type,
          'env': env,
          'id': instance.id,
          'region': 'us-east-1'
        }
    print json.dumps(instances)

main()

output

{  
   "prodweb1":[  
      "made-up-web1.compute-1.amazonaws.com"
   ],
   "web":[  
      "made-up-web1.compute-1.amazonaws.com",
      "made-up-web2.compute-1.amazonaws.com"
   ],
   "all":[  
      "made-up-web1.compute-1.amazonaws.com",
      "made-up-web2.compute-1.amazonaws.com",
      "made-up-celery1.compute-1.amazonaws.com",
      "made-up-manager1.compute-1.amazonaws.com"
   ],
   "prodweb2":[  
      "made-up-web2.compute-1.amazonaws.com"
   ],
   "prodcelery1":[
      "made-up-celery1.compute-1.amazonaws.com"
   ],
   "prodmanager1"[
      "made-up-manager1.compute-1.amazonaws.com"
   ],
   "celery":[
      "made-up-celery1.compute-1.amazonaws.com"
   ],
   "manager":[
      "made-up-manager1.compute-1.amazonaws.com"
   ],
   "_meta":{  
      "hostvars":{  
         "made-up-web1.compute-1.amazonaws.com":{  
            "type":"web",
            "env":"prod",
            "id":"someid",
            "region":"us-east-1"
         }
      }
   }

}

playbooks/deploy.yml

---
- hosts : web
  vars:
    branch : "{{branch}}"
  roles:
      - deploy
  serial: 1

- hosts : celery:manager
  vars:
    branch : "{{branch}}"
  roles:
      - deploy

roles/deploy/tasks/main.yml

---
- name: Instance De-register
  local_action: ec2_elb
  args:
    instance_id: "{{ hostvars[inventory_hostname].id}}"
    state: 'absent'
    profile: 'trackmaven'
    ec2_elbs: "{{ hostvars[inventory_hostname].env}}"
    wait: 'yes'
    wait_timeout: 10
    region: "{{ hostvars[inventory_hostname].region }}"
  when: '"{{ hostvars[inventory_hostname].type }}" == "web"'


- include: deploy.yml

- name: Instance Register
  local_action: ec2_elb
  args:
    instance_id: "{{ hostvars[inventory_hostname].id}}"
    state: 'present'
    profile: 'trackmaven'
    ec2_elbs: "{{ hostvars[inventory_hostname].env}}"
    wait: 'yes'
    wait_timeout: 20
    region: "{{ hostvars[inventory_hostname].region }}"
  when: '"{{ hostvars[inventory_hostname].type }}" == "web"'

roles/deploy/tasks/deploy.yml

---

- name: turn on maintenance mode
  sudo: yes
  command: chdir=/home/web/www/ touch maintenance
  when: '"{{ hostvars[inventory_hostname].type }}" == "web"'

- name: shut down supervisor
  sudo: yes
  service: name=supervisor state=stopped

- name: pull the latest code
  git: repo=git@github.com:TrackMaven/TrackMaven.git dest=/home/web/www/TrackMaven force=yes update=yes version={{branch}}

- name: install lastest requirements
  pip: requirements=/home/web/www/TrackMaven/requirements.txt virtualenv=/home/web/www/.virtualenvs/trackmaven

- name: recreate supervisor conf files
  sudo: yes
  command: chdir=/home/web/www/TrackMaven/  /home/web/www/.virtualenvs/trackmaven/bin/python deployment/export.py -p {{ hostvars[inventory_hostname].procfile }}

- name: remove pycs
  command: chdir=/home/web/www/TrackMaven/ find . -name '*.pyc' -delete

- name: start supervisor
  sudo: yes
  service: name=supervisor state=started

- name: Django collectstatic
  command: chdir=/home/web/www/TrackMaven/ foreman run python trackmaven/manage.py collectstatic --noinput

- name: turn off maintenance mode
  sudo: yes
  command: chdir=/home/web/www/ rm maintenance
  when: '"{{ hostvars[inventory_hostname].type }}" == "web"'