====== ATC Ticket Escalation Process ======
===== Introduction =====
ATC's technical support staff have a standard operational time of ** 7 AM to 8 PM 5 days a week, Monday to Friday**. ATC need to ensure that service affecting outages and issues are still responded to appropriately and efficiently out of hours. ATC provide this cover by having a TSE(Technical Service Engineer) on-call. The TSE on call is rotated amongst the technical support staff in ATC each week.

Since beginning using the Errigal platform, ATC requested for a process/mechanism to be put place to escalate/alert on-call technicians to service affecting issues out of hours. Due to this request, a Ticketer Groovlet was created and put in place to escalate tickets to on-call technicians based on ATC's defined criteria. This wiki entry serves as an overview on how the process was implemented, how it works. It will help you in the instance you are tasked with making any adjustments or changes to the process in the future.

===== Overview Of Process =====
Every time a ticket is created in the Ticketer, in the 'ATC DAS Workflow' with the start state 'Alarm Received'. The 'service level agreement' Groovlet executes to calculate various SLA metrics for that ticket. In this Groovlet is where the Ticket Escalation process is also scripted.

As soon as non-child ticket enter's the 'Alarm Received' state, the ticket escalation process checks if this ticket was created during out of hours(Escalation Time Range) and if so it creates a number of 'Persisted Tasks' to track this ticket. Each Persisted Task contains a closure that executes after a specified time interval, and when executed after the defined duration of time, it checks if the tracked ticket is still in a certain state and if it is still in the escalation time, if so it sends escalations(Email & SMS) to ATC's tech support staff.

Each Persisted Task created for a ticket tracked for escalation, represents a different level of ticket escalation priority. There are 4 Levels of escalation priority defined for each ticket escalation - **Level 1, 2, 3** and **4**. ATC has the ability to change the contact details associated with each level of priority.

Escalation process also considers priority changing by more (child) ticket in same network identifier, by removing the current tasks on the priority updated parent ticket and assign them a new task.

===== Escalation Levels =====
  * **Level 1 Escalation**: A ticket in alarm received 1 Hour -> Email and SMS to On Call Tech and every 15 minutes after this where it remains in that state.

  * **Level 2 Escalation**: A ticket in alarm received 1 Hour and 30 Min -> Email and SMS to Michael Johnson

  * **Level 3 Escalation**: A ticket in alarm received 2 Hours -> Email to Chad Dagenhardt & CC's Michael Johnson

  * **Level 4 Escalation**: A ticket in alarm received 1 Hour -> Email to GNOC & CC's Chad Dagenhardt
===== Escalation Criteria =====
  * **P1 Tickets** are escalated 24 hours a day, 7 Days a week

  * **P2 Tickets** are escalated between the hours 8pm and 7am during weekday, and all day weekend

  * **P3 Tickets** are escalated between all-day weekend and late Friday(20:00--24:00)

  * **P4 Tickets** are not considered for escalation
  
  * **P1 to P3** Tickets are escalated on ATC holiday 
===== How ATC Changes The Escalation Contacts Details =====

When the escalation process was created, there needed to be a mechanism to allow ATC simply change to contact details of the On-Call TSE, as the TSE On-Call is a different person each week, the email and phone number for this escalation level will vary. In order to support this and not have hard-coded contact details in the script, a number of place-holder users where created in the ticketer(see below).

{{:support:escalation_contacts_ticketer.png?800|}}

The Ticket Escalation process keys of off each of these placeholder users in the Ticketer. It uses the contact details in each of these users to send Email and SMS. The key fields used in each user are highlighted below.

{{:support:escalation_user_example.png?800|}}

Email templates are also used when sending Email escalation, the Ticket escalation process looks up the appropriate email template for a given escalation level e.g Level 1, 2, 3 or 4. The email templates used are shown in the below screenshot.

{{:support:escalation_template_example.png?800|}}

===== Trouble-Shooting Tips =====

**Ticket reported by as not having escalated** 
First thing to check is that the ticket was created in the escalation time-frame, a ticket won't escalate even if it is created one second outside of the escalation timeframe. If the ticket was created in the escalation timeframe, check the priority of the ticket at the time of creation and see if it is eligible for escalation. For example, a customer reports a ticket was created at 02:00 am on Wednesday morning which did not escalate. If the ticket was a priority 3 ticket, these tickets are not escalated outside of the weekend.

**Query To Check If A Ticket Should Have Escalated**

<code>
SELECT
	t.*,
        ts.status,
	ts.status_duration_in_seconds
FROM
	ticketer.ticket t
	JOIN ticketer.ticket_status ts ON t.id	= ts.ticket_id
WHERE 
	ts.`status` = 'Alarm Received'
	AND t.priority in (1,2)
        AND ts.status_duration_in_seconds >= 3600 -- 1 Hour Duration In State 
	AND t.create_date BETWEEN '2018-06-26 20:00:00' AND '2018-06-27 07:00:00';
</code>
===== Notes About Modification To Escalation Groovlet =====
** Currently the string closure in the escalation Groovlet logic is at max character capacity, all whitespace has been removed and tabs etc to fit the required logic into the Persisted Task, please be aware of this when making changes. A development request will need to be actioned as this is a domain class and cannot be edited directly in the database as Grails still enforces the constraint at runtime even though the column has been altered at the DB level **