=====Everything A New Server Cluster Needs=====
Author: Cathal O'keeffe


Here we'll give a brief overview of everything a brand new server cluster needs to install to get a full installation of our applications suite. This is a checklist rather than a guide, specific installation instructions should be linked to rather than added here.

====All servers====

  * Create scotty user
  * Install java (jdk1.7.0_80)
  * Install mysql server
    * create watchdog database
    * ensure watchdog DB is reachable from app handlers (for reporting manager watchdog report)
  * Install watchdog
    * Configure server name in watchdog config
    * Configure appropriate watchdog checks for the server
  * Install cron
    * Configure watchdog cronjob

====App Servers (including LB, EMS instance, etc.)====
  * Cron jobs:
    * Log rotate
    * Daily exception report
  * Create directories for:
    * appfiles
    * logs
    * grails app configs (/usr/local/grails)
    * backups
    * the apps themselves (/var/tomcat/,,,)
      * snmp_manager
      * ticketer
      * noc_portal
      * reporting_manager
      * support_page
      * ebonding
      * soap
  * Install tomcat 7.0.33 & 8.0.33
    * configure tomcat setenv.sh to set
      * GRAILS_CONFIG_DIR
      * GRAILS_LOG_DIR
      * APP_FILE_DIR
  * Install lsyncd & unison
    * set up and run master & slave lsync configs
  * App configurations
    * Ensure hostnames are set correctly in SNMP Manager, and SOAP configs
    * Ensure all apps are connecting to the new cluster DB
    * Ensure application URLs are set correctly for new cluster

====Handlers====
  * Cron jobs
    * Backup wars
    * Backup flat files

====Load Balancer====
  * OpenVPN (ExteNet only)
    * Add EPEL repo to yum
    * Install OpenVPN client through yum
    * Add configuration and key files
  * Configure and run httpd

====DB Servers====
  * Create DBs for each application
    * snmp_manager
    * ticketer
    * noc_portal
    * reporting_manager
    * ebonding
    * soap
    * support_page

====Master DB====
  * Configure my.cnf as master to replicate application DBs
  * Configure access to users from each app server
  * Configure reader and writer accounts


====Slave DB====
  * Configure my.cnf as slave to replicate application DBs (make sure server-id is unique!)
  * Configure reader account


====DB Configuration====

  * The following DB tables need to be updated with the cluster info:
    * nocportal.database_profile
    * noc_portal.ticketer_config
    * ticketer.external_datasource
    * ticketer.neighbor
    * ticketer.snmp_manager_config
    * snmp_manager.thread_config
    * snmp_manager.remote_thread_config
    * snmp_manager.link_poller_configuration
    * snmp_manager.database_profile
    * snmp_manager.external_datasource
    * snmp_manager.ticketer_configuration
    * reporting_manager.database_profile