Internal Errigal Collaboration Wiki

User Tools

Site Tools

Trace:
Writing /app/www/public/data/meta/watchdogs/watchdog_overview.meta failed
watchdogs:watchdog_overview

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
watchdogs:watchdog_overview [2017/04/11 11:15] – mmccwatchdogs:watchdog_overview [2021/06/25 10:09] (current) – external edit 127.0.0.1
Line 1: Line 1:
 +====== The Watchdog Process ======
 +
 +Author: Sophie Renshaw
 +
 +
 +
 +  - We receive a Watchdog indicating that there is an issue with one of the servers that we are monitoring. We will receive this notification via email and, if it is a critical alarm, a text message.
 +  - The person that is on the Watchdog Rota or On Call, if we get the notification at the weekend, will click on the link in the email. This will open the watchdog ticket in the mobile ticketer, which is publicly accessible, so it can be accessed outside of the office.
 +  - NOTE: DO NOT LOG IN WITH THE ADMIN ACCOUNT - PLEASE USE YOUR OWN ACCOUNT PROVIDED
 +  - The person Investigating the Watchdog should move the ticket to the Under Investigation state. We will get another email, sent to developers, informing us that the ticket has been moved to Under Investigation. We will see the login of the person who moved the ticket in the subject of the Under Investigation email.
 +  - Once the Investigation has been completed, the Under Investigation email should be replied to with any findings.
 +  - We will get another email when the alarm has cleared. 
 +
 +Once you are finished onboarding, there is more detail relating to the Watchdogs process which you can find here: [[watchdogs:start|Watchdogs]]
 +
 +----
 +
 +
 +**Errigal Watchdog - Resource Alarm Workflow**
 +
 +{{:watchdogs:screen_shot_2016-09-02_at_10.31.35.png?1000|}}
 +
 +
 +----
 +
 +
 +**SendSMSandEmail Groovlet (ID: 233)**
 +
 +<code groovy>
 +arg, defaultArg, ticket ->
 +  
 +/*
 +This groovlet is used to send a ticket email using a custom email template (defined in the workflow node 
 +entrance rule)to a group of users who should be notified about a ticket. 
 +
 +If the email sends, the ticket will enter a successInSendingEmailState state and if it fails to send the ticket
 +will enter a failedToSendEmailState state. These are also defined in the workflow node rule as arguments.
 +
 +If the groovlet fails to fully execute, any exceptions thrown will be sent to the "Failure email address" listed
 +*/
 +
 +/*
 +Defining the sendSMS closure
 +*/
 +  //Save on Users needed to resolve the issue that was being seen in SUPPORT-216, whereby the groovlet was failing due to different users interacting with the ticket
 +  def ticketStatusesByUsers = ticket.statuses.statusBy
 +  ticketStatusesByUsers.each(){
 +    com.errigal.ticketer.User user = it
 +    user.save()
 +  }
 +  log.info "Ticket Statuses By: $ticketStatusesByUsers"
 +
 +  def sendEmail = {email,body -> final String username = "ticketer@errigal.com"
 +        final String password = "notsame1"
 +
 +        Properties props = new Properties()
 +        props.put("mail.smtp.starttls.enable", "true")
 +        props.put("mail.smtp.auth", "true")
 +        props.put("mail.smtp.host", "smtp.gmail.com")
 +        props.put("mail.smtp.port", "587")
 +
 +        javax.mail.Session session = javax.mail.Session.getInstance(props,
 +          new javax.mail.Authenticator() {
 +            protected javax.mail.PasswordAuthentication getPasswordAuthentication() {
 +                return new javax.mail.PasswordAuthentication(username, password)
 +            }
 +          })
 +
 +        try {
 +
 +            javax.mail.Message message = new javax.mail.internet.MimeMessage(session)
 +            message.setFrom(new javax.mail.internet.InternetAddress("peter.phelan@errigal.com"))
 +            message.setRecipients(javax.mail.Message.RecipientType.TO,
 +                                  javax.mail.internet.InternetAddress.parse(email))
 +            message.setSubject("")
 +            message.setText(body)
 +
 +            javax.mail.Transport.send(message)
 +
 +        } catch (javax.mail.MessagingException e) {
 +            throw new RuntimeException(e)
 +        } 
 +  }
 + 
 +    
 +log.info "Default arg:  $defaultArg"
 +log.info "Argument:  $arg"
 +log.info "Ticket: $ticket"
 +
 +def args = arg.split(",")
 +
 +def stateName
 +def successInSendingEmailState
 +def failedToSendEmailState//what happens if no fail state provided
 +
 + log.info "Groovlet has received correct amount of arguments. Executing" 
 + 
 + stateName = args[0].trim();
 +  
 + def emailService = com.errigal.ticketer.utils.DomainUtils.getGrailsService("emailService")
 +
 +com.errigal.ticketer.Visibility ticketVisibility = com.errigal.ticketer.Visibility.findByName(ticket.visibility)
 +log.info "Visibility: $ticketVisibility"
 +
 +com.errigal.ticketer.EmailAccount fromEmailAccount = ticketVisibility.emailAccount
 +log.info "Email Account: $fromEmailAccount"
 +
 +//TODO: Support exit email too
 +def entranceEmail = ticket.workflow.nodes.find{it.name == ticket.currentStatus.status}.entranceEmail
 +
 +log.info "Email to use: ${entranceEmail}"
 +log.info "Ticket ID: ${ticket.id}"
 +
 +def emailVo = emailService.getCustomEmailVOWithoutSecurityCheck(ticket.id, entranceEmail)
 +log.info "Email VO: $emailVo"
 +
 +try{
 + emailService.sendEmail(emailVo)
 +}
 +catch(Exception e){
 + log.error "Unable to send email in Groovlet", e 
 +}
 +
 +Calendar cal = Calendar.getInstance(); // creates calendar
 +cal.setTime(new Date()); // sets calendar time/date
 +Boolean hasAlarmID = false
 +int currentHour =  cal.get(Calendar.HOUR_OF_DAY)
 +def acceptedAlarmIDs = ["remoteTicketCreationInactive", "LinkPollerInactive","TrapForwarderInactive","QuartzJobsBlocked","NextFireTimeInThePast","DefaultEmailUpdatePollJob","JBossProcessInactive","ApplicationServerProcessInactive","OutOfMemoryErrorFound",
 +                        "PermGenErrorFound","mysqlSlaveReplicationFailure","ScriptNotRunning", "TrapParsingInactive",
 +                        "mysqlFailure","heartbeatMissing", "TrapCountBreached", "IrisLicenseCheckFailed", "UnableToRunCommand","MobiTrapInactivity","MobiTicketInactivity", "handlerFailover", "MySQLJDBCCommunicationsExceptionCommunicationLinkFailure"]
 +acceptedAlarmIDs.each{it -> if(ticket.summary.contains(it)){hasAlarmID = true}}
 +
 +if(hasAlarmID){
 +  log.info "Sending Text for ${ticket.id}"
 +} else {
 +  log.info "Could not find any of ${acceptedAlarmIDs} in '${ticket.summary}' for ${ticket.id}"
 +}
 +
 +
 +String body = """api_id:3513334 \r
 +user:errigalwaterford \r
 +password:#errigal321!# \r
 +from:Errigal \r
 +to:353851190587,353863997160,353870574522,353861568567,353871246696,353868441173,353861703468, 353861703468, 353872777914, 353871477066 \r
 +text:$ticket.idWithPrefix - $ticket.currentStatus.status : $ticket.summary"""
 +
 +if(currentHour >= 16 && currentHour <= 23 && hasAlarmID)
 +{
 +sendEmail("4157480023@vtext.com","$ticket.idWithPrefix - $ticket.currentStatus.status : $ticket.summary")
 +sendEmail("4155598389@vtext.com","$ticket.idWithPrefix - $ticket.currentStatus.status : $ticket.summary")
 +sendEmail("4157264987@vtext.com","$ticket.idWithPrefix - $ticket.currentStatus.status : $ticket.summary")
 +sendEmail("4157206583@vtext.com","$ticket.idWithPrefix - $ticket.currentStatus.status : $ticket.summary")
 +sendEmail("6282287260@tmomail.net","$ticket.idWithPrefix - $ticket.currentStatus.status : $ticket.summary")
 +}
 +else if (currentHour >= 0 && currentHour <= 7 && hasAlarmID)
 +{
 +  sendEmail("sms@messaging.clickatell.com",body)
 +}
 +else if (hasAlarmID)
 +{
 +sendEmail("4157480023@vtext.com","$ticket.idWithPrefix - $ticket.currentStatus.status : $ticket.summary")
 +sendEmail("4155598389@vtext.com","$ticket.idWithPrefix - $ticket.currentStatus.status : $ticket.summary")
 +sendEmail("4157264987@vtext.com","$ticket.idWithPrefix - $ticket.currentStatus.status : $ticket.summary")
 +sendEmail("4157206583@vtext.com","$ticket.idWithPrefix - $ticket.currentStatus.status : $ticket.summary")
 +sendEmail("6282287260@tmomail.net","$ticket.idWithPrefix - $ticket.currentStatus.status : $ticket.summary")
 +sendEmail("sms@messaging.clickatell.com",body)
 +}
 +
 +  
 +// This below code is used to change state if the email has been sent (i.e. "Alarm Received" -> "Notification Sent")
 +if (ticket.parent) {
 +  log.info "Ticket $ticket.id is no longer a top-level Ticket; cancelling automatic State Change."
 +} else {
 +  // Change the Ticket State if it is a top-level Ticket
 +  if (com.errigal.ticketer.GNode.findByWorkflowAndName(ticket.workflow, stateName)) {
 +    log.info "Updating Ticket $ticket.id Status from $ticket.currentStatus.status to $stateName."
 +    ticket.updateStatus(stateName)
 +    def saveResult = ticket.save(failOnError: true)
 +    if (saveResult) {
 +      log.info "State change result:  $saveResult"
 +    } else {
 +      def sb = new StringBuilder("Save failed ")
 +      if (hasErrors()) {
 +        sb.append " errors:  "
 +        errors.allErrors.each {sb.append it}
 +      } else {
 +        sb.append " no error messages."
 +      }
 +      log.info sb
 +    }
 +  } else {
 +    log.error "$stateName was not found in the $ticket.workflow.name Workflow.  Aborting."
 +  }
 + }
 +</code>
 +
 +=== Creating a Watchdog ===
 +  -Log onto the server which you want to create the watchdog on
 +  -Go to the following directory; /export/home/scotty/watchdog/resources
 +  -Add your new rule to the bottom of the ResourceConfig.groovy file in this dir, e.g.<code groovy>
 +'GeneralTrapSummaryCreation' {
 +    type = 'MYSQLDB'
 +        parameters {
 +          driver = 'com.mysql.jdbc.Driver'
 +          host = 'localhost'
 +          port = '3306'
 +          database = 'snmp_manager'
 +          user = 'root'
 +          password = 'ozzrules'
 +          checkRowCountQuery = 'select count(*) from general_trap_summary where received_date > now() - interval 5 minute;'
 +        }
 +        thresholds {
 +          a = [name: 'mysqlRowCount', type: 'MIN', value: [0], level: ['CRITICAL'], alarmId: 'generalTrapSummaryCreationInactive']
 +        }
 +  }
 +</code>
 +  -Double check the DB query on the DB to ensure that it is correct(careful as its production!)
 +  -Once the rule is added and your query is checked, then the rule to the main method(called localSystem in the code) to be invoked, e.g.<code groovy>
 +localSystem {
 +    ....
 +    dd = 'ActiveAlarmCreation'
 +    ee = 'GeneralTrapSummaryCreation'
 +}
 +</code>
  

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki