Troubleshooting - Service Entitlements form not auto adding

Author: Michelle McCausland

Example ticket - ESR-994757

Support, Tickets 533788 and 535434 and 530987 were created via an SNMP trap. 
When the tickets created, the service entitlement form was not automatically added. Please advise ?

Some initial items to check to verify correct functioning of this behaviour include:

1. Ensuring there is a network_identifier entry in the network_element table for the affected element
2. Ensuring there is a matching entry for the network_identifier in the service_level_agreement table
3. Ensuring the associated alarm has a trap_knowledge entry with is_service_impacting populated
4. Ensuring the network element associated with the general trap summary has a valid ne_type

Using 535434 as an example:

• Ensuring there is a network_identifier entry in the network_element table for the affected element

SELECT id, name, network_identifier 
FROM snmp_manager.network_element 
WHERE network_identifier = "NC-IL-515NSTATE-VZW"

A number of rows were returned, so this is not the cause of the issue.

• Ensuring there is a matching entry for the network_identifier in the service_level_agreement table

SELECT * FROM service_level_agreement WHERE network_identifier = "NC-IL-515NSTATE-VZW";

No rows returned, this is the cause of the issue - there is no service level agreement (SLA) for this network! It is important to inform the customer of this in case this network needs to be configured with an SLA.

If you have reviewed the steps above and this still doesn't reveal the cause of the issue then it is necessary to refer back to the logs to find out more.

Using 533788 as an example:

• Start by reviewing the Ticketer logs for when the ticket was created.
• To do this search for the following: Creating new Ticket 533788 :

2017-05-19 09:19:11,626 [ajp-bio-8013-exec-129] INFO  ticketer.RemoteTicketService  - Creating new Ticket with Forms [SNMP Trap] from Remote request
2017-05-19 09:19:11,626 [ajp-bio-8013-exec-129] INFO  ticketer.BouncingTicketService  - Found: AL - Open - fwuHstDARTUnderDriveFault - fwuHstDARTUnderDriveFault Module: 5 - SE-AL-AUBURN-HOST22 -
2017-05-19 09:19:11,672 [ajp-bio-8013-exec-129] INFO  ticketer.RemoteTicketService  - Created Ticket #null with Forms [SNMP Trap].
2017-05-19 09:19:11,677 [ajp-bio-8013-exec-129] INFO  utils.DomainUtils  - Found [name:AL - Open] in the domainClass space for com.errigal.ticketer.Visibility
2017-05-19 09:19:11,683 [ajp-bio-8013-exec-129] INFO  utils.DomainUtils  - Found [name:ExteNet DAS Alarm] in the domainClass space for com.errigal.ticketer.GWorkflow
2017-05-19 09:19:11,685 [ajp-bio-8013-exec-129] INFO  utils.DomainUtils  - Found [login:monitor, active:true] in the domainClass space for com.errigal.ticketer.User
2017-05-19 09:19:11,686 [ajp-bio-8013-exec-129] INFO  ticketer.Ticket  - Setting new Ticket Status to Alarm Received.
2017-05-19 09:19:11,693 [ajp-bio-8013-exec-129] INFO  ticketer.Ticket  - Creating new Ticket 533788 for user 'monitor'
2017-05-19 09:19:11,693 [ajp-bio-8013-exec-129] INFO  audit.AuditProvider  - auditable property is set AND there ARE Inserts
2017-05-19 09:19:11,693 [ajp-bio-8013-exec-129] INFO  audit.AuditProvider  - going to build an insert audit event
2017-05-19 09:19:11,696 [ajp-bio-8013-exec-129] INFO  ticketer.Ticket  - No form name so not adding a form on this state change
2017-05-19 09:19:11,696 [ajp-bio-8013-exec-129] INFO  ticketer.Ticket  - Setting Owners for new state
2017-05-19 09:19:11,696 [ajp-bio-8013-exec-129] INFO  ticketer.Ticket  - New owners for ticket=533788 = [:]
2017-05-19 09:19:11,696 [ajp-bio-8013-exec-129] INFO  ticketer.Ticket  - Assigning Owners [] to Ticket 533788
2017-05-19 09:19:11,696 [ajp-bio-8013-exec-129] INFO  ticketer.Ticket  - Adding owners  to Ticket 533788
2017-05-19 09:19:11,704 [ajp-bio-8013-exec-129] INFO  ticketer.GNode  - Preparing to Execute the Entrance Rule for Node Alarm Received (in ExteNet DAS Alarm)
2017-05-19 09:19:11,704 [ajp-bio-8013-exec-129] INFO  utils.GroovletUtil  - About to Run Node Rule for Ticket: 533788 at Status: Alarm Received.
2017-05-19 09:19:11,707 [ajp-bio-8013-exec-129] INFO  ticketer.Ticket  - No Escalation defined for ExteNet DAS Alarm/Alarm Received to Level 2.  Ignoring auto-escalate.
2017-05-19 09:19:11,715 [ajp-bio-8013-exec-129] INFO  ticketer.ChangeForm  - Attempting to find published form with name SNMP Trap
2017-05-19 09:19:11,717 [ajp-bio-8013-exec-129] INFO  ticketer.ChangeForm  - Found published form : SNMP Trap Form
2017-05-19 09:19:11,730 [ajp-bio-8013-exec-129] INFO  ticketer.Ticket  - Inserting new change form values: [[change_id:606919, context:fwuHstDARTUnderDriveFault Module: 5, ne_name:SE-AL-AUBURN-HOST22, trap_severity:major, received_date:2017-05-19, source:exts304, trap_text:Parent:SE-AL-AUBURN-HOST22,Host:SE-AL-AUBURN-HOST22, Module: 5, Trap:fwuHstDARTUnderDriveFault, Status:major, suggested_remedy:, trap_knowledge_checkbox:false, received_time:08:49, probable_cause:, trap_name:fwuHstDARTUnderDriveFault, affected_carrier:Verizon Wireless, alarm_id:67670255]]
2017-05-19 09:19:11,733 [ajp-bio-8013-exec-129] INFO  ticketer.Ticket  - Flushing Update to Ticket 533788 for user auto.
2017-05-19 09:19:11,733 [ajp-bio-8013-exec-129] INFO  audit.AuditProvider  - auditable property is set AND there ARE Dirty Props
2017-05-19 09:19:11,733 [ajp-bio-8013-exec-129] INFO  audit.AuditProvider  - going to build an update audit event
2017-05-19 09:19:11,733 [ajp-bio-8013-exec-129] INFO  audit.AuditProvider  - Pushing Update Audit Event to Exchange; ticketer_audit_event_exchange_topic for domain: com.errigal.ticketer.Ticket with ticket id: 533788
2017-05-19 09:19:11,734 [ajp-bio-8013-exec-129] INFO  queueing.RabbitMQConnectionService  - Publishing audit event message with routing key com.errigal.ticketer.Ticket to exchange
2017-05-19 09:19:11,740 [ajp-bio-8013-exec-129] INFO  ticketer.GNode  - Preparing to Execute the Entrance Rule for Node Alarm Received (in ExteNet DAS Alarm)
2017-05-19 09:19:11,740 [ajp-bio-8013-exec-129] INFO  utils.GroovletUtil  - About to Run Node Rule for Ticket: 533788 at Status: Alarm Received.

• As you can see from the first line the RemoteTicketService creates a new ticket, however, it only adds an SNMP Trap form to it. (Reviewing other instances of this in the logs confirmed this expected behaviour)
• To figure out why this was happening, it was necessary to review the SNMP Manager logs for the same day to find the Remote Ticket creation for this ticket.
• To do this search for the following: SE-AL-AUBURN-HOST22 - ALARM: fwuHstDARTUnderDriveFault :

 2017-05-19 09:19:11,569 [Thread-20] INFO  schedule.Scheduler  - Preparing to process 1 tasks.
 2017-05-19 09:19:11,582 [Thread-20] INFO  wrapper.GeneralTrapSummaryWrapper  - Creating Remote Ticket for NE - 11298315: SE-AL-AUBURN-HOST22 - ALARM: fwuHstDARTUnderDriveFault Module: 5 - STATUS: major
 2017-05-19 09:19:11,597 [Thread-20] INFO  remote.RemoteTicketCreator  - Checking for Service Level Agreement Information for this Remote Ticket request...
 2017-05-19 09:19:11,600 [Thread-20] INFO  remote.RemoteTicketCreator  - Affected Carriers found via Network Element Carrier Configuration: [Verizon Wireless]
 2017-05-19 09:19:11,602 [Thread-20] INFO  remote.RemoteTicketCreator  - Carrier name in Service Level Agreement form: AT&T
 2017-05-19 09:19:11,602 [Thread-20] INFO  remote.RemoteTicketCreator  - Using the following service level agreements derived from Carrier associations: []
 2017-05-19 09:19:11,604 [Thread-20] INFO  remote.RemoteTicketCreator  - Start neutral host processing for Ticketer
 2017-05-19 09:19:11,605 [Thread-20] INFO  remote.RemoteTicketCreator  - SNMP Trap found, adding affected_carrier field
 2017-05-19 09:19:11,605 [Thread-20] INFO  remote.RemoteTicketCreator  - affected_carrier (Verizon Wireless) was added to SNMP Trap (ActiveAlarm #11298315)
 2017-05-19 09:19:11,605 [Thread-20] INFO  remote.RemoteTicketCreator  - Submitting remote ticket with ExteNet - HESSIAN - http://exts302/Ticketer/hessian/RemoteTicketService and NE - 11298315: SE-AL-AUBURN-HOST22 - ALARM: fwuHstDARTUnderDriveFault Module: 5 - STATUS: major and SE-AL-AUBURN-HOST22
 2017-05-19 09:19:11,606 [com.errigal.shared.remote.RemoteTicketCreator] INFO  remote.RemoteTicketCreationTask  - >>>> Executing com.errigal.shared.remote.RemoteTicketCreationTask.
 2017-05-19 09:19:11,610 [Thread-13] INFO  pollers.ClientActiveAlarmPoller  - Beginning refresh of ClientActiveAlarmPoll at 09:19:11
 2017-05-19 09:19:11,618 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  - Have active alarm in task 'run()' method...
 2017-05-19 09:19:11,618 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  - Running remote ticket with ExteNet - HESSIAN - http://exts302/Ticketer/hessian/RemoteTicketService and NE - 11298315: SE-AL-AUBURN-HOST22 - ALARM: fwuHstDARTUnderDriveFault Module: 5 - STATUS: major
 2017-05-19 09:19:11,618 [com.errigal.shared.remote.RemoteTicketCreator] INFO  remote.RemoteTicketCreator  - Attempting to create RMI link using http://exts302/Ticketer/hessian/RemoteTicketService with protocol HESSIAN
 2017-05-19 09:19:11,618 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  -   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MAKE REMOTE CALL.... (GTSID:67670255)
 2017-05-19 09:19:11,740 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  -   >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> END REMOTE CALL.... (GTSID:67670255)
 2017-05-19 09:19:11,740 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  -  >>>>>>>>>>>>>>>>>>>> TICKET IDENTIFIER: 533788
 2017-05-19 09:19:11,740 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  - Creating Remote Ticket With identifier 533788 and subject SE-AL-AUBURN-HOST22 : DART Under Drive (fwuHstDARTUnderDriveFault Module: 5) - major
 2017-05-19 09:19:11,740 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  - Marking Alarm Acknowledged
 2017-05-19 09:19:11,740 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  - Adding RemoteTicket with identifier: 533788 to ActiveAlarm(11298315)
 2017-05-19 09:19:11,742 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  - Try remote ticket save...
 2017-05-19 09:19:11,748 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  - Saved Remote ticket
 2017-05-19 09:19:11,748 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  - >>>>> ACTIVE ALARM (pre-save): NE - 11298315: SE-AL-AUBURN-HOST22 - ALARM: fwuHstDARTUnderDriveFault Module: 5 - STATUS: major
 2017-05-19 09:19:11,748 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  - >>>>> REMOTE TICKET (pre-save): RemoteTicket{ticketIdentifier='533788', ticketSubject='SE-AL-AUBURN-HOST22 : DART Under Drive (fwuHstDARTUnderDriveFault Module: 5) - major', createDate=Fri May 19 09:19:11 CDT 2017, manuallyCleared=false, manuallyClearedDate=null, autoCleared=false, autoClearedDate=null}
 2017-05-19 09:19:11,748 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  - Saving ActiveAlarm
 2017-05-19 09:19:11,748 [com.errigal.shared.remote.RemoteTicketCreator] INFO  services.RemoteTaskService  - TRY SAVE...

• At this stage it was necessary to bring up the SNMP Manager codebase to review what was happening at each step in the logs.
• The key here in the logs is on lines 4, 5, 6.
• As the carrier value from the SLA entry was different from the affected carrier in the alarm, the service level agreement did not populate for this alarm.
• As it turned out the SLA entry for SE-AL-AUBURN-OPN was configured with the wrong carrier_id of AT&T instead of blank.
• Changing this carrier_id rectified the problem which was confirmed when new tickets started generating for this element with an attached service entitlements form!
• In this instance we probably investigated a bit too far when we could have just examined the SLA entry in the database. However, in following this approach we learned more about the functioning of this feature which will prove useful for future customer requests.