====== RabbitMQ - RemoteTicketFailedToCreate ======

Author: Eoin Hearne

A prometheus alert will come in similar to [FIRING:1] RemoteTicketFailedToCreate prodext (extlb1:15692 rabbitmq_3_8 remote.ticket.dead.letter.queue critical admin). If this happens, it signifies there's a remote ticket that failed to create a ticket for an unknown reason. The application sends JSON to the RabbitMQ remote.ticket.create.queue and will try to create a ticket, with a given number of retries, currently defaulted to 5. If it fails every try, the remote ticket is deleted and the request is added to the remote.ticket.dead.letter.queue. Follow the below instructions, similar to the failed email wiki entry, to investigate.

The interface for the RabbitMQ email queues can be found at loadbalancer:15672 -> `http://extlb.ext:15672` or `http://atclb1.atc:15672` and the credentials can be found in pwsafe.
Once logged in you will be presented with this interface. Click on the **Queues** tab.
{{ :support:rabbitmq_interface_landing.png?nolink |}}

Then navigate to the **remote.ticket.dead.letter.queue**. The queue might be at the bottom of the multi page list, use the search bar if necessary.

{{:watchdogs:queues-overview.png?noLink|}}

Scroll down the page and click on the **Get Messages** tab, then click the **Get Message(s)** button.** Make sure to use the 'Nack message requeue' true option so that messages are not deleted.** Copy the payload field into your clipboard and investigate what the cause of the issue was. The exceptionTrace field in the JSON should give a clear indicator to the problem. 

{{:watchdogs:get-messages.png?noLink|}}

Please see example fail below.In this case the payload was missing the WORKFLOW parameter under ticketValues, caused by the trap rule not having a fallback value when the network elements workflow is blank or null. The alarm_id field is the creating general_trap_summary id. Check in the database first to see if the alarm has cleared since. If it has, it may not be necessary to complete the rest of the steps, and the dead letter queue could be purged (see last step).
<code>{"correlationId":"0126ea0c-d31e-4d6f-854b-540e74ea5cbf","eventCode":"TICKET_CREATE_FAILED","exceptionTrace":"com.errigal.ticketer.workbench.client.exceptions.LoggableClientErrorException:Workflow[null]wasnotfoundinthedatabase.","failedRequest":{"correlationId":"0126ea0c-d31e-4d6f-854b-540e74ea5cbf","formValues":[{"trap_name":"txCardFaultAlarm","affected_carrier":"VerizonWireless","trap_text":"3200002170Alarm:TransmissionFault","source":"extapps1","alarm_id":"383598839","received_time":"16:59:30","context":"TransmissionFault","ne_name":"SE-FL-MARLINS-OPN-VOLTSERVER3-3200002170","received_date":"WedApr0116:59:30CDT2020","trap_severity":"Major","FORM_NAME_PARAM":"SNMPTrap"},{"head_end_equip_replacement_spares_exte_net_distributed_network":"","head_end_b_b_u_repair_general":"","node_equip_repair_combinersand_splitters":"","head_end_b_b_u_replacement_spares_customer_b_b_u":"","network_backbone_replacement_spares_general":"","node_equip_maintenance_active_node_components":"","major_restore":"90","head_end_equip_initial_spares_customer_r_f_source":"","network_backbone_monitoring_general":"Included","carrier":"VerizonWireless","node_equip_replacement_spares_combinersand_splitters":"","network_backbone_initial_spares_general":"","network_backbone_initial_spares_sheath_conduit_innerduct":"","node_equip_monitoring_combinersand_splitters":"","hub_site_general":"","head_end_comp_monitoring_general":"","network_backbone_repair_fiber_system_components":"","node_equip_initial_spares_transport_leadsand_connectors":"","node_equip_repair_power_leadsand_connectors":"","node_equip_repair_antenna":"","node_equip_initial_spares_active_node_components":"","network_backbone_repair_splice_boxes":"","node_equip_monitoring_transport_leadsand_connectors":"","head_end_b_b_u_initial_spares_exte_net_b_b_u":"","network_backbone_replacement_spares_splice_boxes":"","head_end_b_b_u_maintenance_general":"","FORM_NAME_PARAM":"ServiceEntitlements","node_equip_b_b_u_initial_spares_general":"","head_end_equip_replacement_spares_general":"","network_backbone_repair_sheath_conduit_innerduct":"","node_equip_b_b_u_maintenance_general":"","minor_restore":"120","node_equip_replacement_spares_transport_leadsand_connectors":"","node_equip_monitoring_power_leadsand_connectors":"","hub_id":"","node_equip_replacement_spares_antenna":"","node_id":"n/a","node_equip_initial_spares_combinersand_splitters":"","hub_site_maintenance":"","node_equip_monitoring_general":"Included","network_backbone_maintenance_fiber_optic_cabling":"","network_backbone_maintenance_sheath_conduit_innerduct":"","node_equip_b_b_u_general":"","node_equip_replacement_spares_power_leadsand_connectors":"","critical_restore":"60","node_equip_initial_spares_antenna":"","node_equip_maintenance_general":"","head_end_comp_maintenance_general":"","node_equip_b_b_u_repair_customer_b_b_u":"","node_equip_repair_active_node_components":"","head_end_comp_repair_customer_r_f_source":"","node_equip_maintenance_planned_maintenance":"","head_end_b_b_u_replacement_spares_exte_net_b_b_u":"","head_end_comp_maintenance_exte_net_distributed_network":"","head_end_b_b_u_general":"","node_equip_b_b_u_exte_net_b_b_u":"","network_backbone_monitoring_fiber_optic_cabling":"","network_backbone_repair_general":"","network_backbone_monitoring_splice_boxes":"","node_site_replacement_spares_pole_general":"","network_backbone_monitoring_fiber_system_components":"","comments":"null","network_backbone_repair_fiber_optic_cabling":"","network_backbone_maintenance_splice_boxes":"","network_backbone_maintenance_fiber_system_components":"","head_end_b_b_u_exte_net_b_b_u":"","node_equip_b_b_u_maintenance_customer_b_b_u":"","node_equip_maintenance_transport_leadsand_connectors":"","hub_site_power":"Pro-Rata","head_end_comp_repair_general":"","network_backbone_monitoring_sheath_conduit_innerduct":"","head_end_b_b_u_maintenance_exte_net_b_b_u":"","node_equip_b_b_u_repair_exte_net_b_b_u":"","node_equip_monitoring_active_node_components":"","head_end_b_b_u_repair_customer_b_b_u":"","hub_site_repair":"","head_end_b_b_u_initial_spares_customer_b_b_u":"","network_backbone_maintenance_general":"","node_equip_initial_spares_general":"","node_equip_initial_spares_power_leadsand_connectors":"","node_equip_replacement_spares_general":"","head_end_comp_monitoring_customer_r_f_source":"","head_end_equip_replacement_spares_customer_r_f_source":"","major_threshold":"10.00","node_equip_b_b_u_customer_b_b_u":"","node_equip_repair_general":"","node_equip_maintenance_power_leadsand_connectors":"","node_equip_monitoring_antenna":"","critical_threshold":"50.00","network_backbone_initial_spares_fiber_system_components":"","node_equip_b_b_u_repair_general":"","head_end_equip_initial_spares_general":"","node_equip_b_b_u_replacement_spares_customer":"","node_equip_maintenance_combinersand_splitters":"","head_end_comp_monitoring_exte_net_distributed_network":"","node_equip_repair_transport_leadsand_connectors":"","is_service_impacting":"false","network_backbone_replacement_spares_sheath_conduit_innerduct":"","node_equip_replacement_spares_active_node_components":"","head_end_b_b_u_customer_b_b_u":"","head_end_b_b_u_repair_exte_net_b_b_u":"","head_end_b_b_u_maintenance_customer_b_b_u":"","network_backbone_replacement_spares_fiber_system_components":"","head_end_equip_initial_spares_exte_net_distributed_network":"","network_name":"SE-FL-MARLINS-OPN","head_end_comp_repair_exte_net_distributed_network":"","minor_threshold":"0.00","node_equip_maintenance_antenna":"","element_total":"129","node_equip_b_b_u_maintenance_exte_net_b_b_u":"","network_backbone_replacement_spares_fiber_optic_cabling":"","head_end_comp_maintenance_customer_r_f_source":"","node_equip_b_b_u_replacement_spares_exte_net":""},{"head_end_equip_replacement_spares_exte_net_distributed_network":"","head_end_b_b_u_repair_general":"","node_equip_repair_combinersand_splitters":"","head_end_b_b_u_replacement_spares_customer_b_b_u":"","network_backbone_replacement_spares_general":"","node_equip_maintenance_active_node_components":"","major_restore":"90","head_end_equip_initial_spares_customer_r_f_source":"","network_backbone_monitoring_general":"","carrier":"NoCarrierAssigned","node_equip_replacement_spares_combinersand_splitters":"","network_backbone_initial_spares_general":"","network_backbone_initial_spares_sheath_conduit_innerduct":"","node_equip_monitoring_combinersand_splitters":"","hub_site_general":"","head_end_comp_monitoring_general":"","network_backbone_repair_fiber_system_components":"","node_equip_initial_spares_transport_leadsand_connectors":"","node_equip_repair_power_leadsand_connectors":"","node_equip_repair_antenna":"","node_equip_initial_spares_active_node_components":"","network_backbone_repair_splice_boxes":"","node_equip_monitoring_transport_leadsand_connectors":"","head_end_b_b_u_initial_spares_exte_net_b_b_u":"","network_backbone_replacement_spares_splice_boxes":"","head_end_b_b_u_maintenance_general":"","FORM_NAME_PARAM":"ServiceEntitlements","node_equip_b_b_u_initial_spares_general":"","head_end_equip_replacement_spares_general":"","network_backbone_repair_sheath_conduit_innerduct":"","node_equip_b_b_u_maintenance_general":"","minor_restore":"120","node_equip_replacement_spares_transport_leadsand_connectors":"","node_equip_monitoring_power_leadsand_connectors":"","hub_id":"","node_equip_replacement_spares_antenna":"","node_id":"n/a","node_equip_initial_spares_combinersand_splitters":"","hub_site_maintenance":"","node_equip_monitoring_general":"","network_backbone_maintenance_fiber_optic_cabling":"","network_backbone_maintenance_sheath_conduit_innerduct":"","node_equip_b_b_u_general":"","node_equip_replacement_spares_power_leadsand_connectors":"","critical_restore":"120","node_equip_initial_spares_antenna":"","node_equip_maintenance_general":"","head_end_comp_maintenance_general":"","node_equip_b_b_u_repair_customer_b_b_u":"","node_equip_repair_active_node_components":"","head_end_comp_repair_customer_r_f_source":"","node_equip_maintenance_planned_maintenance":"","head_end_b_b_u_replacement_spares_exte_net_b_b_u":"","head_end_comp_maintenance_exte_net_distributed_network":"","head_end_b_b_u_general":"","node_equip_b_b_u_exte_net_b_b_u":"","network_backbone_monitoring_fiber_optic_cabling":"","network_backbone_repair_general":"","network_backbone_monitoring_splice_boxes":"","node_site_replacement_spares_pole_general":"","network_backbone_monitoring_fiber_system_components":"","comments":"","network_backbone_repair_fiber_optic_cabling":"","network_backbone_maintenance_splice_boxes":"","network_backbone_maintenance_fiber_system_components":"","head_end_b_b_u_exte_net_b_b_u":"","node_equip_b_b_u_maintenance_customer_b_b_u":"","node_equip_maintenance_transport_leadsand_connectors":"","hub_site_power":"","head_end_comp_repair_general":"","network_backbone_monitoring_sheath_conduit_innerduct":"","head_end_b_b_u_maintenance_exte_net_b_b_u":"","node_equip_b_b_u_repair_exte_net_b_b_u":"","node_equip_monitoring_active_node_components":"","head_end_b_b_u_repair_customer_b_b_u":"","hub_site_repair":"","head_end_b_b_u_initial_spares_customer_b_b_u":"","network_backbone_maintenance_general":"","node_equip_initial_spares_general":"","node_equip_initial_spares_power_leadsand_connectors":"","node_equip_replacement_spares_general":"","head_end_comp_monitoring_customer_r_f_source":"","head_end_equip_replacement_spares_customer_r_f_source":"","major_threshold":"10.00","node_equip_b_b_u_customer_b_b_u":"","node_equip_repair_general":"","node_equip_maintenance_power_leadsand_connectors":"","node_equip_monitoring_antenna":"","critical_threshold":"50.00","network_backbone_initial_spares_fiber_system_components":"","node_equip_b_b_u_repair_general":"","head_end_equip_initial_spares_general":"","node_equip_b_b_u_replacement_spares_customer":"","node_equip_maintenance_combinersand_splitters":"","head_end_comp_monitoring_exte_net_distributed_network":"","node_equip_repair_transport_leadsand_connectors":"","is_service_impacting":"false","network_backbone_replacement_spares_sheath_conduit_innerduct":"","node_equip_replacement_spares_active_node_components":"","head_end_b_b_u_customer_b_b_u":"","head_end_b_b_u_repair_exte_net_b_b_u":"","head_end_b_b_u_maintenance_customer_b_b_u":"","network_backbone_replacement_spares_fiber_system_components":"","head_end_equip_initial_spares_exte_net_distributed_network":"","network_name":"SE-FL-MARLINS-OPN","head_end_comp_repair_exte_net_distributed_network":"","minor_threshold":"0.00","node_equip_maintenance_antenna":"","element_total":"129","node_equip_b_b_u_maintenance_exte_net_b_b_u":"","network_backbone_replacement_spares_fiber_optic_cabling":"","head_end_comp_maintenance_customer_r_f_source":"","node_equip_b_b_u_replacement_spares_exte_net":""}],"ticketValues":{"SUMMARY":"SE-FL-MARLINS-OPN-VOLTSERVER3-3200002170:TransmissionFault","CRITICAL_ELEMENT_TICKET":"false","VISIBILITY":"FL-Marlins","PRIORITY":"2","CREATOR":"monitor","STATUS":"AlarmReceived","NETWORK_IDENTIFIER":"SE-FL-MARLINS-OPN","MARKET":"FL-Marlins"},"eventCode":"TICKET_CREATE_INITIATED","requestRetryCount":"5"},"timestamp":"2020-04-01T17:29:35.014-05:00"}</code>


Alter the failed JSON object to allow the create queue to recreate the ticket. Please see example below of expected JSON for creating tickets.

<code>{"correlationId":"0126ea0c-d31e-4d6f-854b-540e74ea5cbf","formValues":[{"trap_name":"txCardFaultAlarm","affected_carrier":"VerizonWireless","trap_text":"3200002170Alarm:TransmissionFault","source":"extapps1","alarm_id":"383598839","received_time":"16:59:30","context":"TransmissionFault","ne_name":"SE-FL-MARLINS-OPN-VOLTSERVER3-3200002170","received_date":"WedApr0116:59:30CDT2020","trap_severity":"Major","FORM_NAME_PARAM":"SNMPTrap"},{"head_end_equip_replacement_spares_exte_net_distributed_network":"","head_end_b_b_u_repair_general":"","node_equip_repair_combinersand_splitters":"","head_end_b_b_u_replacement_spares_customer_b_b_u":"","network_backbone_replacement_spares_general":"","node_equip_maintenance_active_node_components":"","major_restore":"90","head_end_equip_initial_spares_customer_r_f_source":"","network_backbone_monitoring_general":"Included","carrier":"VerizonWireless","node_equip_replacement_spares_combinersand_splitters":"","network_backbone_initial_spares_general":"","network_backbone_initial_spares_sheath_conduit_innerduct":"","node_equip_monitoring_combinersand_splitters":"","hub_site_general":"","head_end_comp_monitoring_general":"","network_backbone_repair_fiber_system_components":"","node_equip_initial_spares_transport_leadsand_connectors":"","node_equip_repair_power_leadsand_connectors":"","node_equip_repair_antenna":"","node_equip_initial_spares_active_node_components":"","network_backbone_repair_splice_boxes":"","node_equip_monitoring_transport_leadsand_connectors":"","head_end_b_b_u_initial_spares_exte_net_b_b_u":"","network_backbone_replacement_spares_splice_boxes":"","head_end_b_b_u_maintenance_general":"","FORM_NAME_PARAM":"ServiceEntitlements","node_equip_b_b_u_initial_spares_general":"","head_end_equip_replacement_spares_general":"","network_backbone_repair_sheath_conduit_innerduct":"","node_equip_b_b_u_maintenance_general":"","minor_restore":"120","node_equip_replacement_spares_transport_leadsand_connectors":"","node_equip_monitoring_power_leadsand_connectors":"","hub_id":"","node_equip_replacement_spares_antenna":"","node_id":"n/a","node_equip_initial_spares_combinersand_splitters":"","hub_site_maintenance":"","node_equip_monitoring_general":"Included","network_backbone_maintenance_fiber_optic_cabling":"","network_backbone_maintenance_sheath_conduit_innerduct":"","node_equip_b_b_u_general":"","node_equip_replacement_spares_power_leadsand_connectors":"","critical_restore":"60","node_equip_initial_spares_antenna":"","node_equip_maintenance_general":"","head_end_comp_maintenance_general":"","node_equip_b_b_u_repair_customer_b_b_u":"","node_equip_repair_active_node_components":"","head_end_comp_repair_customer_r_f_source":"","node_equip_maintenance_planned_maintenance":"","head_end_b_b_u_replacement_spares_exte_net_b_b_u":"","head_end_comp_maintenance_exte_net_distributed_network":"","head_end_b_b_u_general":"","node_equip_b_b_u_exte_net_b_b_u":"","network_backbone_monitoring_fiber_optic_cabling":"","network_backbone_repair_general":"","network_backbone_monitoring_splice_boxes":"","node_site_replacement_spares_pole_general":"","network_backbone_monitoring_fiber_system_components":"","comments":"null","network_backbone_repair_fiber_optic_cabling":"","network_backbone_maintenance_splice_boxes":"","network_backbone_maintenance_fiber_system_components":"","head_end_b_b_u_exte_net_b_b_u":"","node_equip_b_b_u_maintenance_customer_b_b_u":"","node_equip_maintenance_transport_leadsand_connectors":"","hub_site_power":"Pro-Rata","head_end_comp_repair_general":"","network_backbone_monitoring_sheath_conduit_innerduct":"","head_end_b_b_u_maintenance_exte_net_b_b_u":"","node_equip_b_b_u_repair_exte_net_b_b_u":"","node_equip_monitoring_active_node_components":"","head_end_b_b_u_repair_customer_b_b_u":"","hub_site_repair":"","head_end_b_b_u_initial_spares_customer_b_b_u":"","network_backbone_maintenance_general":"","node_equip_initial_spares_general":"","node_equip_initial_spares_power_leadsand_connectors":"","node_equip_replacement_spares_general":"","head_end_comp_monitoring_customer_r_f_source":"","head_end_equip_replacement_spares_customer_r_f_source":"","major_threshold":"10.00","node_equip_b_b_u_customer_b_b_u":"","node_equip_repair_general":"","node_equip_maintenance_power_leadsand_connectors":"","node_equip_monitoring_antenna":"","critical_threshold":"50.00","network_backbone_initial_spares_fiber_system_components":"","node_equip_b_b_u_repair_general":"","head_end_equip_initial_spares_general":"","node_equip_b_b_u_replacement_spares_customer":"","node_equip_maintenance_combinersand_splitters":"","head_end_comp_monitoring_exte_net_distributed_network":"","node_equip_repair_transport_leadsand_connectors":"","is_service_impacting":"false","network_backbone_replacement_spares_sheath_conduit_innerduct":"","node_equip_replacement_spares_active_node_components":"","head_end_b_b_u_customer_b_b_u":"","head_end_b_b_u_repair_exte_net_b_b_u":"","head_end_b_b_u_maintenance_customer_b_b_u":"","network_backbone_replacement_spares_fiber_system_components":"","head_end_equip_initial_spares_exte_net_distributed_network":"","network_name":"SE-FL-MARLINS-OPN","head_end_comp_repair_exte_net_distributed_network":"","minor_threshold":"0.00","node_equip_maintenance_antenna":"","element_total":"129","node_equip_b_b_u_maintenance_exte_net_b_b_u":"","network_backbone_replacement_spares_fiber_optic_cabling":"","head_end_comp_maintenance_customer_r_f_source":"","node_equip_b_b_u_replacement_spares_exte_net":""},{"head_end_equip_replacement_spares_exte_net_distributed_network":"","head_end_b_b_u_repair_general":"","node_equip_repair_combinersand_splitters":"","head_end_b_b_u_replacement_spares_customer_b_b_u":"","network_backbone_replacement_spares_general":"","node_equip_maintenance_active_node_components":"","major_restore":"90","head_end_equip_initial_spares_customer_r_f_source":"","network_backbone_monitoring_general":"","carrier":"NoCarrierAssigned","node_equip_replacement_spares_combinersand_splitters":"","network_backbone_initial_spares_general":"","network_backbone_initial_spares_sheath_conduit_innerduct":"","node_equip_monitoring_combinersand_splitters":"","hub_site_general":"","head_end_comp_monitoring_general":"","network_backbone_repair_fiber_system_components":"","node_equip_initial_spares_transport_leadsand_connectors":"","node_equip_repair_power_leadsand_connectors":"","node_equip_repair_antenna":"","node_equip_initial_spares_active_node_components":"","network_backbone_repair_splice_boxes":"","node_equip_monitoring_transport_leadsand_connectors":"","head_end_b_b_u_initial_spares_exte_net_b_b_u":"","network_backbone_replacement_spares_splice_boxes":"","head_end_b_b_u_maintenance_general":"","FORM_NAME_PARAM":"ServiceEntitlements","node_equip_b_b_u_initial_spares_general":"","head_end_equip_replacement_spares_general":"","network_backbone_repair_sheath_conduit_innerduct":"","node_equip_b_b_u_maintenance_general":"","minor_restore":"120","node_equip_replacement_spares_transport_leadsand_connectors":"","node_equip_monitoring_power_leadsand_connectors":"","hub_id":"","node_equip_replacement_spares_antenna":"","node_id":"n/a","node_equip_initial_spares_combinersand_splitters":"","hub_site_maintenance":"","node_equip_monitoring_general":"","network_backbone_maintenance_fiber_optic_cabling":"","network_backbone_maintenance_sheath_conduit_innerduct":"","node_equip_b_b_u_general":"","node_equip_replacement_spares_power_leadsand_connectors":"","critical_restore":"120","node_equip_initial_spares_antenna":"","node_equip_maintenance_general":"","head_end_comp_maintenance_general":"","node_equip_b_b_u_repair_customer_b_b_u":"","node_equip_repair_active_node_components":"","head_end_comp_repair_customer_r_f_source":"","node_equip_maintenance_planned_maintenance":"","head_end_b_b_u_replacement_spares_exte_net_b_b_u":"","head_end_comp_maintenance_exte_net_distributed_network":"","head_end_b_b_u_general":"","node_equip_b_b_u_exte_net_b_b_u":"","network_backbone_monitoring_fiber_optic_cabling":"","network_backbone_repair_general":"","network_backbone_monitoring_splice_boxes":"","node_site_replacement_spares_pole_general":"","network_backbone_monitoring_fiber_system_components":"","comments":"","network_backbone_repair_fiber_optic_cabling":"","network_backbone_maintenance_splice_boxes":"","network_backbone_maintenance_fiber_system_components":"","head_end_b_b_u_exte_net_b_b_u":"","node_equip_b_b_u_maintenance_customer_b_b_u":"","node_equip_maintenance_transport_leadsand_connectors":"","hub_site_power":"","head_end_comp_repair_general":"","network_backbone_monitoring_sheath_conduit_innerduct":"","head_end_b_b_u_maintenance_exte_net_b_b_u":"","node_equip_b_b_u_repair_exte_net_b_b_u":"","node_equip_monitoring_active_node_components":"","head_end_b_b_u_repair_customer_b_b_u":"","hub_site_repair":"","head_end_b_b_u_initial_spares_customer_b_b_u":"","network_backbone_maintenance_general":"","node_equip_initial_spares_general":"","node_equip_initial_spares_power_leadsand_connectors":"","node_equip_replacement_spares_general":"","head_end_comp_monitoring_customer_r_f_source":"","head_end_equip_replacement_spares_customer_r_f_source":"","major_threshold":"10.00","node_equip_b_b_u_customer_b_b_u":"","node_equip_repair_general":"","node_equip_maintenance_power_leadsand_connectors":"","node_equip_monitoring_antenna":"","critical_threshold":"50.00","network_backbone_initial_spares_fiber_system_components":"","node_equip_b_b_u_repair_general":"","head_end_equip_initial_spares_general":"","node_equip_b_b_u_replacement_spares_customer":"","node_equip_maintenance_combinersand_splitters":"","head_end_comp_monitoring_exte_net_distributed_network":"","node_equip_repair_transport_leadsand_connectors":"","is_service_impacting":"false","network_backbone_replacement_spares_sheath_conduit_innerduct":"","node_equip_replacement_spares_active_node_components":"","head_end_b_b_u_customer_b_b_u":"","head_end_b_b_u_repair_exte_net_b_b_u":"","head_end_b_b_u_maintenance_customer_b_b_u":"","network_backbone_replacement_spares_fiber_system_components":"","head_end_equip_initial_spares_exte_net_distributed_network":"","network_name":"SE-FL-MARLINS-OPN","head_end_comp_repair_exte_net_distributed_network":"","minor_threshold":"0.00","node_equip_maintenance_antenna":"","element_total":"129","node_equip_b_b_u_maintenance_exte_net_b_b_u":"","network_backbone_replacement_spares_fiber_optic_cabling":"","head_end_comp_maintenance_customer_r_f_source":"","node_equip_b_b_u_replacement_spares_exte_net":""}],"ticketValues":{	"WORKFLOW":"ExteNetDASAlarm","SUMMARY":"SE-FL-MARLINS-OPN-VOLTSERVER3-3200002170:TransmissionFault","CRITICAL_ELEMENT_TICKET":"false","VISIBILITY":"FL-Marlins","PRIORITY":"2","CREATOR":"monitor","STATUS":"AlarmReceived","NETWORK_IDENTIFIER":"SE-FL-MARLINS-OPN","MARKET":"FL-Marlins"},"eventCode":"TICKET_CREATE_INITIATED","requestRetryCount":"5"}</code>


Once the JSON is corrected, navigate to the remote.ticket.create.queue and under publish message, paste the correct JSON into the payload field, ensuring **Delivery Mode** is set to **2 - Persistent** and click **Publish Message**

{{:watchdogs:publish-message.png?nolink|}}


At this stage, the ticket will try to recreate. To verify, check in the ticketer ticket table to see if a ticket was created. No remote ticket will be created, as this is deleted after all the retries fail.

Finally, once the messages have been re-published, return to the **remote.ticket.dead.letter** queue and click **Purge** to clear the dead queue and clear the alert.

{{:watchdogs:purge.png?nolink|}}

Another scenario is the one found below. A failed ticket close was found in the dead letter queue. This one failed because it tried to parent the ticket 1780630 to itself. This was due to a grand child issue. Once the grandchildren were cleared, the second message was published to the remote.ticket.update.queue and the dead letter queue was emptied.

Dead letter message
<code>
{
  "correlationId":"18f3ad4f-44c3-48a5-b3a6-acad10ed1074",
  "ticketValues":{
     "user":"monitor",
     "ticket_id":"1780630",
     "fromState":"*",
     "toState":"Alarm Clear Received"
  },
  "eventCode":"TICKET_CLOSE_INITIATED",
  "requestRetryCount":"5"
}
</code>

Update queue message
<code>
{
   "correlationId":"18f3ad4f-44c3-48a5-b3a6-acad10ed1074",
   "ticketId":1780630,
   "eventCode":"TICKET_CLOSE_FAILED",
   "exceptionTrace":"grails.validation.ValidationException: Validation Error(s) occurred during save():\n- Field error in object \u0027com.errigal.ticketer.Ticket\u0027 on field \u0027parent\u0027: rejected value [Ticket #1780630  Create Date: 2021-11-12 01:55:22.0  Created By: IDMS Monitor  (monitor)  Status: Alarm Clear Received:monitor (1)\nWorkflow: ExteNet DAS Alarm  Summary: NW-CA-FAIRMTSJ-SECTOR-6-OPN - RIM-4-4/VZW 1900B S1 SOUTH TOWER:rimDLRFLowPowerSet | Nightly MIMO Update Date: 2021-11-12 03:10:59.0  Market: CA - Open  DueDate: 2021-11-12 09:53:00.0  Priority:  2\nForms: [Tier 1 Preliminary Investigations Form, SNMP Trap, ExteNet NOC Form, Service Entitlements]]; codes [com.errigal.ticketer.Ticket.parent.validator.error.com.errigal.ticketer.Ticket.parent,com.errigal.ticketer.Ticket.parent.validator.error.parent,com.errigal.ticketer.Ticket.parent.validator.error.com.errigal.ticketer.Ticket,com.errigal.ticketer.Ticket.parent.validator.error,ticket.parent.validator.error.com.errigal.ticketer.Ticket.parent,ticket.parent.validator.error.parent,ticket.parent.validator.error.com.errigal.ticketer.Ticket,ticket.parent.validator.error,com.errigal.ticketer.Ticket.parent.validator.invalid.com.errigal.ticketer.Ticket.parent,com.errigal.ticketer.Ticket.parent.validator.invalid.parent,com.errigal.ticketer.Ticket.parent.validator.invalid.com.errigal.ticketer.Ticket,com.errigal.ticketer.Ticket.parent.validator.invalid,ticket.parent.validator.invalid.com.errigal.ticketer.Ticket.parent,ticket.parent.validator.invalid.parent,ticket.parent.validator.invalid.com.errigal.ticketer.Ticket,ticket.parent.validator.invalid,validator.invalid.com.errigal.ticketer.Ticket.parent,validator.invalid.parent,validator.invalid.com.errigal.ticketer.Ticket,validator.invalid]; arguments [parent,class com.errigal.ticketer.Ticket,Ticket #1780630  Create Date: 2021-11-12 01:55:22.0  Created By: IDMS Monitor  (monitor)  Status: Alarm Clear Received:monitor (1)\nWorkflow: ExteNet DAS Alarm  Summary: NW-CA-FAIRMTSJ-SECTOR-6-OPN - RIM-4-4/VZW 1900B S1 SOUTH TOWER:rimDLRFLowPowerSet | Nightly MIMO Update Date: 2021-11-12 03:10:59.0  Market: CA - Open  DueDate: 2021-11-12 09:53:00.0  Priority:  2\nForms: [Tier 1 Preliminary Investigations Form, SNMP Trap, ExteNet NOC Form, Service Entitlements]]; default message [The {1}\u0027s \u0027{0}\u0027 value ({2}) does not pass validation]\n",
   "failedRequest":{
      "correlationId":"18f3ad4f-44c3-48a5-b3a6-acad10ed1074",
      "ticketValues":{
         "user":"monitor",
         "ticket_id":"1780630",
         "fromState":"*",
         "toState":"Alarm Clear Received"
      },
      "eventCode":"TICKET_CLOSE_INITIATED",
      "requestRetryCount":"5"
   },
   "timestamp":"2021-11-12T03:10:59.565-06:00"
}
</code>