Ansible deployment-playbooks

This project covers all the playbooks prepared in order to automate our deployment. These playbooks are made up of re-usable Ansible roles.

Read about Ansible-Roles and what each one does before proceeding.

Anything mentioned below, such as code or file can be found in the https://bitbucket.org/errigal/deployment-playbooks repository.

Our current MoP is roughly as mentioned below:

  1. Prework
  2. Upgrade Steps
    1. Shut down applications
    2. Deploy New Wars
    3. Grails Config Changes & StartupScript Change
    4. Startup.sh and Shutdown.sh Changes
    5. Backup certain tables (to prepare for a rollback)
    6. Startup CAS
    7. Apply upgrade scripts & start Apps one by one
  3. Post Upgrade
  4. Post Release Testing
  5. Rollback

At the time of writing this, deployment-playbooks is able to automate the first two steps (Prework and Upgrade Steps) to a certain extent.

An example of ansible commands used during deployment to SFQA is shown below: 

#Pull all the ansible-roles
./pull-required-roles.sh
 
#upload Artifacts
ansible-playbook -i env/sfqa/hosts upload-artefacts.yml --ask-vault-pass
 
#deploy prometheus, startup scripts and shutdown scripts, fix permissions.
ansible-playbook -i env/sfqa/hosts deploy_startup_shutdown_scripts.yml --ask-vault-pass
 
#Shutdown Applications
ansible-playbook -i env/sfqa/hosts shutdown-applications.yml --ask-vault-pass
 
#Deploy the war files
ansible-playbook -i env/sfqa/hosts deploy-artefacts.yml --ask-vault-pass
 
#Backup DB
ansible-playbook -v -i env/sfqa/hosts backup_db.yml --ask-vault-pass
 
#Startup CAS
ansible-playbook -i env/sfqa/hosts startup-cas.yml --ask-vault-pass
 
#Start SNMP
ansible-playbook -i env/sfqa/hosts startup-snmpmanager-on-handlers.yml --ask-vault-pass
 
ansible-playbook -i env/sfqa/hosts startup-ticketer-apps1.yml --ask-vault-pass
ansible-playbook -i env/sfqa/hosts startup-ticketer-apps2.yml --ask-vault-pass
ansible-playbook -i env/sfqa/hosts startup-nocportal.yml --ask-vault-pass
ansible-playbook -i env/sfqa/hosts startup-reportingmanager.yml --ask-vault-pass

As we can see above, every command requires two files:

  1. The host file
  2. The playbook YML file

The hosts file stores the IP address of every server for an environment(Crown-QA, Extenet-Production etc) grouped together based on their role(Load-balancer, App-Handlers, DB-server etc).

The hosts file when loaded will automatically load the file group_vars/all present in the same directory which contains variables that are unique per environment.

The playbook YML file contains the actual steps/states required to complete each step of the deployment process.

Playbooks can be limited to specific hosts using the –limit=apps1/apps2/whatever parameter

Passwords and Vaults

Update: Use the newer ./ansible_encrypt.sh from within the deployment-playbooks repo I have added a video tutorial for it as well in our share Dropbox: https://www.dropbox.com/s/hokvpzh521h8niv/ansible_encryption.mov?dl=0

The passwords within the file group_vars/all are encrypted for two primary reasons

  1. To ensure no server password is part of a VCS
  2. To avoid someone running the command by mistake.

The passwords are encrypted in the following manner:

Create a file called sfqa with the text: #sfqa!# (which is the master password) 

ansible-vault encrypt_string --vault-id sfqa '<secret>' --name 'ssh_pass'

Result:

ssh_pass: !vault |
      $ANSIBLE_VAULT;1.1;AES256
      62313365396662343061393464336163383764373764613633653634306231386433626436623361
      6134333665353966363534333632666535333761666131620a663537646436643839616531643561
      63396265333966386166373632626539326166353965363262633030333630313338646335303630
      3438626666666137650a353638643435666633633964366338633066623234616432373231333331
      6564

A master-password is expected to decrypt the passwords required to connect to any given server.

This master-password will be same as the environment name. if the environment name is ABCQA then the password will be #ABCQA!#.