Author: Eoin Hearne

A prometheus alert will come in similar to [FIRING:1] RemoteTicketFailedToCreate prodext (extlb1:15692 rabbitmq_3_8 remote.ticket.dead.letter.queue critical admin). If this happens, it signifies there's a remote ticket that failed to create a ticket for an unknown reason. The application sends JSON to the RabbitMQ remote.ticket.create.queue and will try to create a ticket, with a given number of retries, currently defaulted to 5. If it fails every try, the remote ticket is deleted and the request is added to the remote.ticket.dead.letter.queue. Follow the below instructions, similar to the failed email wiki entry, to investigate.

The interface for the RabbitMQ email queues can be found at loadbalancer:15672 → `http://extlb.ext:15672` or `http://atclb1.atc:15672` and the credentials can be found in pwsafe. Once logged in you will be presented with this interface. Click on the Queues tab.

Then navigate to the remote.ticket.dead.letter.queue.

Scroll down the page and click on the Get Messages tab, then click the Get Message(s) button. Make sure to use the 'Nack message requeue' true option so that messages are not deleted. Copy the payload field into your clipboard and investigate what the cause of the issue was. The exceptionTrace field in the JSON should give a clear indicator to the problem.

Please see example fail below.In this case the payload was missing the WORKFLOW parameter under ticketValues, caused by the trap rule not having a fallback value when the network elements workflow is blank or null. The alarm_id field is the creating general_trap_summary id. Check in the database first to see if the alarm has cleared since. If it has, it may not be necessary to complete the rest of the steps, and the dead letter queue could be purged (see last step).

{"correlationId":"0126ea0c-d31e-4d6f-854b-540e74ea5cbf","eventCode":"TICKET_CREATE_FAILED","exceptionTrace":"com.errigal.ticketer.workbench.client.exceptions.LoggableClientErrorException:Workflow[null]wasnotfoundinthedatabase.","failedRequest":{"correlationId":"0126ea0c-d31e-4d6f-854b-540e74ea5cbf","formValues":[{"trap_name":"txCardFaultAlarm","affected_carrier":"VerizonWireless","trap_text":"3200002170Alarm:TransmissionFault","source":"extapps1","alarm_id":"383598839","received_time":"16:59:30","context":"TransmissionFault","ne_name":"SE-FL-MARLINS-OPN-VOLTSERVER3-3200002170","received_date":"WedApr0116:59:30CDT2020","trap_severity":"Major","FORM_NAME_PARAM":"SNMPTrap"},{"head_end_equip_replacement_spares_exte_net_distributed_network":"","head_end_b_b_u_repair_general":"","node_equip_repair_combinersand_splitters":"","head_end_b_b_u_replacement_spares_customer_b_b_u":"","network_backbone_replacement_spares_general":"","node_equip_maintenance_active_node_components":"","major_restore":"90","head_end_equip_initial_spares_customer_r_f_source":"","network_backbone_monitoring_general":"Included","carrier":"VerizonWireless","node_equip_replacement_spares_combinersand_splitters":"","network_backbone_initial_spares_general":"","network_backbone_initial_spares_sheath_conduit_innerduct":"","node_equip_monitoring_combinersand_splitters":"","hub_site_general":"","head_end_comp_monitoring_general":"","network_backbone_repair_fiber_system_components":"","node_equip_initial_spares_transport_leadsand_connectors":"","node_equip_repair_power_leadsand_connectors":"","node_equip_repair_antenna":"","node_equip_initial_spares_active_node_components":"","network_backbone_repair_splice_boxes":"","node_equip_monitoring_transport_leadsand_connectors":"","head_end_b_b_u_initial_spares_exte_net_b_b_u":"","network_backbone_replacement_spares_splice_boxes":"","head_end_b_b_u_maintenance_general":"","FORM_NAME_PARAM":"ServiceEntitlements","node_equip_b_b_u_initial_spares_general":"","head_end_equip_replacement_spares_general":"","network_backbone_repair_sheath_conduit_innerduct":"","node_equip_b_b_u_maintenance_general":"","minor_restore":"120","node_equip_replacement_spares_transport_leadsand_connectors":"","node_equip_monitoring_power_leadsand_connectors":"","hub_id":"","node_equip_replacement_spares_antenna":"","node_id":"n/a","node_equip_initial_spares_combinersand_splitters":"","hub_site_maintenance":"","node_equip_monitoring_general":"Included","network_backbone_maintenance_fiber_optic_cabling":"","network_backbone_maintenance_sheath_conduit_innerduct":"","node_equip_b_b_u_general":"","node_equip_replacement_spares_power_leadsand_connectors":"","critical_restore":"60","node_equip_initial_spares_antenna":"","node_equip_maintenance_general":"","head_end_comp_maintenance_general":"","node_equip_b_b_u_repair_customer_b_b_u":"","node_equip_repair_active_node_components":"","head_end_comp_repair_customer_r_f_source":"","node_equip_maintenance_planned_maintenance":"","head_end_b_b_u_replacement_spares_exte_net_b_b_u":"","head_end_comp_maintenance_exte_net_distributed_network":"","head_end_b_b_u_general":"","node_equip_b_b_u_exte_net_b_b_u":"","network_backbone_monitoring_fiber_optic_cabling":"","network_backbone_repair_general":"","network_backbone_monitoring_splice_boxes":"","node_site_replacement_spares_pole_general":"","network_backbone_monitoring_fiber_system_components":"","comments":"null","network_backbone_repair_fiber_optic_cabling":"","network_backbone_maintenance_splice_boxes":"","network_backbone_maintenance_fiber_system_components":"","head_end_b_b_u_exte_net_b_b_u":"","node_equip_b_b_u_maintenance_customer_b_b_u":"","node_equip_maintenance_transport_leadsand_connectors":"","hub_site_power":"Pro-Rata","head_end_comp_repair_general":"","network_backbone_monitoring_sheath_conduit_innerduct":"","head_end_b_b_u_maintenance_exte_net_b_b_u":"","node_equip_b_b_u_repair_exte_net_b_b_u":"","node_equip_monitoring_active_node_components":"","head_end_b_b_u_repair_customer_b_b_u":"","hub_site_repair":"","head_end_b_b_u_initial_spares_customer_b_b_u":"","network_backbone_maintenance_general":"","node_equip_initial_spares_general":"","node_equip_initial_spares_power_leadsand_connectors":"","node_equip_replacement_spares_general":"","head_end_comp_monitoring_customer_r_f_source":"","head_end_equip_replacement_spares_customer_r_f_source":"","major_threshold":"10.00","node_equip_b_b_u_customer_b_b_u":"","node_equip_repair_general":"","node_equip_maintenance_power_leadsand_connectors":"","node_equip_monitoring_antenna":"","critical_threshold":"50.00","network_backbone_initial_spares_fiber_system_components":"","node_equip_b_b_u_repair_general":"","head_end_equip_initial_spares_general":"","node_equip_b_b_u_replacement_spares_customer":"","node_equip_maintenance_combinersand_splitters":"","head_end_comp_monitoring_exte_net_distributed_network":"","node_equip_repair_transport_leadsand_connectors":"","is_service_impacting":"false","network_backbone_replacement_spares_sheath_conduit_innerduct":"","node_equip_replacement_spares_active_node_components":"","head_end_b_b_u_customer_b_b_u":"","head_end_b_b_u_repair_exte_net_b_b_u":"","head_end_b_b_u_maintenance_customer_b_b_u":"","network_backbone_replacement_spares_fiber_system_components":"","head_end_equip_initial_spares_exte_net_distributed_network":"","network_name":"SE-FL-MARLINS-OPN","head_end_comp_repair_exte_net_distributed_network":"","minor_threshold":"0.00","node_equip_maintenance_antenna":"","element_total":"129","node_equip_b_b_u_maintenance_exte_net_b_b_u":"","network_backbone_replacement_spares_fiber_optic_cabling":"","head_end_comp_maintenance_customer_r_f_source":"","node_equip_b_b_u_replacement_spares_exte_net":""},{"head_end_equip_replacement_spares_exte_net_distributed_network":"","head_end_b_b_u_repair_general":"","node_equip_repair_combinersand_splitters":"","head_end_b_b_u_replacement_spares_customer_b_b_u":"","network_backbone_replacement_spares_general":"","node_equip_maintenance_active_node_components":"","major_restore":"90","head_end_equip_initial_spares_customer_r_f_source":"","network_backbone_monitoring_general":"","carrier":"NoCarrierAssigned","node_equip_replacement_spares_combinersand_splitters":"","network_backbone_initial_spares_general":"","network_backbone_initial_spares_sheath_conduit_innerduct":"","node_equip_monitoring_combinersand_splitters":"","hub_site_general":"","head_end_comp_monitoring_general":"","network_backbone_repair_fiber_system_components":"","node_equip_initial_spares_transport_leadsand_connectors":"","node_equip_repair_power_leadsand_connectors":"","node_equip_repair_antenna":"","node_equip_initial_spares_active_node_components":"","network_backbone_repair_splice_boxes":"","node_equip_monitoring_transport_leadsand_connectors":"","head_end_b_b_u_initial_spares_exte_net_b_b_u":"","network_backbone_replacement_spares_splice_boxes":"","head_end_b_b_u_maintenance_general":"","FORM_NAME_PARAM":"ServiceEntitlements","node_equip_b_b_u_initial_spares_general":"","head_end_equip_replacement_spares_general":"","network_backbone_repair_sheath_conduit_innerduct":"","node_equip_b_b_u_maintenance_general":"","minor_restore":"120","node_equip_replacement_spares_transport_leadsand_connectors":"","node_equip_monitoring_power_leadsand_connectors":"","hub_id":"","node_equip_replacement_spares_antenna":"","node_id":"n/a","node_equip_initial_spares_combinersand_splitters":"","hub_site_maintenance":"","node_equip_monitoring_general":"","network_backbone_maintenance_fiber_optic_cabling":"","network_backbone_maintenance_sheath_conduit_innerduct":"","node_equip_b_b_u_general":"","node_equip_replacement_spares_power_leadsand_connectors":"","critical_restore":"120","node_equip_initial_spares_antenna":"","node_equip_maintenance_general":"","head_end_comp_maintenance_general":"","node_equip_b_b_u_repair_customer_b_b_u":"","node_equip_repair_active_node_components":"","head_end_comp_repair_customer_r_f_source":"","node_equip_maintenance_planned_maintenance":"","head_end_b_b_u_replacement_spares_exte_net_b_b_u":"","head_end_comp_maintenance_exte_net_distributed_network":"","head_end_b_b_u_general":"","node_equip_b_b_u_exte_net_b_b_u":"","network_backbone_monitoring_fiber_optic_cabling":"","network_backbone_repair_general":"","network_backbone_monitoring_splice_boxes":"","node_site_replacement_spares_pole_general":"","network_backbone_monitoring_fiber_system_components":"","comments":"","network_backbone_repair_fiber_optic_cabling":"","network_backbone_maintenance_splice_boxes":"","network_backbone_maintenance_fiber_system_components":"","head_end_b_b_u_exte_net_b_b_u":"","node_equip_b_b_u_maintenance_customer_b_b_u":"","node_equip_maintenance_transport_leadsand_connectors":"","hub_site_power":"","head_end_comp_repair_general":"","network_backbone_monitoring_sheath_conduit_innerduct":"","head_end_b_b_u_maintenance_exte_net_b_b_u":"","node_equip_b_b_u_repair_exte_net_b_b_u":"","node_equip_monitoring_active_node_components":"","head_end_b_b_u_repair_customer_b_b_u":"","hub_site_repair":"","head_end_b_b_u_initial_spares_customer_b_b_u":"","network_backbone_maintenance_general":"","node_equip_initial_spares_general":"","node_equip_initial_spares_power_leadsand_connectors":"","node_equip_replacement_spares_general":"","head_end_comp_monitoring_customer_r_f_source":"","head_end_equip_replacement_spares_customer_r_f_source":"","major_threshold":"10.00","node_equip_b_b_u_customer_b_b_u":"","node_equip_repair_general":"","node_equip_maintenance_power_leadsand_connectors":"","node_equip_monitoring_antenna":"","critical_threshold":"50.00","network_backbone_initial_spares_fiber_system_components":"","node_equip_b_b_u_repair_general":"","head_end_equip_initial_spares_general":"","node_equip_b_b_u_replacement_spares_customer":"","node_equip_maintenance_combinersand_splitters":"","head_end_comp_monitoring_exte_net_distributed_network":"","node_equip_repair_transport_leadsand_connectors":"","is_service_impacting":"false","network_backbone_replacement_spares_sheath_conduit_innerduct":"","node_equip_replacement_spares_active_node_components":"","head_end_b_b_u_customer_b_b_u":"","head_end_b_b_u_repair_exte_net_b_b_u":"","head_end_b_b_u_maintenance_customer_b_b_u":"","network_backbone_replacement_spares_fiber_system_components":"","head_end_equip_initial_spares_exte_net_distributed_network":"","network_name":"SE-FL-MARLINS-OPN","head_end_comp_repair_exte_net_distributed_network":"","minor_threshold":"0.00","node_equip_maintenance_antenna":"","element_total":"129","node_equip_b_b_u_maintenance_exte_net_b_b_u":"","network_backbone_replacement_spares_fiber_optic_cabling":"","head_end_comp_maintenance_customer_r_f_source":"","node_equip_b_b_u_replacement_spares_exte_net":""}],"ticketValues":{"SUMMARY":"SE-FL-MARLINS-OPN-VOLTSERVER3-3200002170:TransmissionFault","CRITICAL_ELEMENT_TICKET":"false","VISIBILITY":"FL-Marlins","PRIORITY":"2","CREATOR":"monitor","STATUS":"AlarmReceived","NETWORK_IDENTIFIER":"SE-FL-MARLINS-OPN","MARKET":"FL-Marlins"},"eventCode":"TICKET_CREATE_INITIATED","requestRetryCount":"5"},"timestamp":"2020-04-01T17:29:35.014-05:00"}

Alter the failed JSON object to allow the create queue to recreate the ticket. Please see example below of expected JSON for creating tickets.

{"correlationId":"0126ea0c-d31e-4d6f-854b-540e74ea5cbf","formValues":[{"trap_name":"txCardFaultAlarm","affected_carrier":"VerizonWireless","trap_text":"3200002170Alarm:TransmissionFault","source":"extapps1","alarm_id":"383598839","received_time":"16:59:30","context":"TransmissionFault","ne_name":"SE-FL-MARLINS-OPN-VOLTSERVER3-3200002170","received_date":"WedApr0116:59:30CDT2020","trap_severity":"Major","FORM_NAME_PARAM":"SNMPTrap"},{"head_end_equip_replacement_spares_exte_net_distributed_network":"","head_end_b_b_u_repair_general":"","node_equip_repair_combinersand_splitters":"","head_end_b_b_u_replacement_spares_customer_b_b_u":"","network_backbone_replacement_spares_general":"","node_equip_maintenance_active_node_components":"","major_restore":"90","head_end_equip_initial_spares_customer_r_f_source":"","network_backbone_monitoring_general":"Included","carrier":"VerizonWireless","node_equip_replacement_spares_combinersand_splitters":"","network_backbone_initial_spares_general":"","network_backbone_initial_spares_sheath_conduit_innerduct":"","node_equip_monitoring_combinersand_splitters":"","hub_site_general":"","head_end_comp_monitoring_general":"","network_backbone_repair_fiber_system_components":"","node_equip_initial_spares_transport_leadsand_connectors":"","node_equip_repair_power_leadsand_connectors":"","node_equip_repair_antenna":"","node_equip_initial_spares_active_node_components":"","network_backbone_repair_splice_boxes":"","node_equip_monitoring_transport_leadsand_connectors":"","head_end_b_b_u_initial_spares_exte_net_b_b_u":"","network_backbone_replacement_spares_splice_boxes":"","head_end_b_b_u_maintenance_general":"","FORM_NAME_PARAM":"ServiceEntitlements","node_equip_b_b_u_initial_spares_general":"","head_end_equip_replacement_spares_general":"","network_backbone_repair_sheath_conduit_innerduct":"","node_equip_b_b_u_maintenance_general":"","minor_restore":"120","node_equip_replacement_spares_transport_leadsand_connectors":"","node_equip_monitoring_power_leadsand_connectors":"","hub_id":"","node_equip_replacement_spares_antenna":"","node_id":"n/a","node_equip_initial_spares_combinersand_splitters":"","hub_site_maintenance":"","node_equip_monitoring_general":"Included","network_backbone_maintenance_fiber_optic_cabling":"","network_backbone_maintenance_sheath_conduit_innerduct":"","node_equip_b_b_u_general":"","node_equip_replacement_spares_power_leadsand_connectors":"","critical_restore":"60","node_equip_initial_spares_antenna":"","node_equip_maintenance_general":"","head_end_comp_maintenance_general":"","node_equip_b_b_u_repair_customer_b_b_u":"","node_equip_repair_active_node_components":"","head_end_comp_repair_customer_r_f_source":"","node_equip_maintenance_planned_maintenance":"","head_end_b_b_u_replacement_spares_exte_net_b_b_u":"","head_end_comp_maintenance_exte_net_distributed_network":"","head_end_b_b_u_general":"","node_equip_b_b_u_exte_net_b_b_u":"","network_backbone_monitoring_fiber_optic_cabling":"","network_backbone_repair_general":"","network_backbone_monitoring_splice_boxes":"","node_site_replacement_spares_pole_general":"","network_backbone_monitoring_fiber_system_components":"","comments":"null","network_backbone_repair_fiber_optic_cabling":"","network_backbone_maintenance_splice_boxes":"","network_backbone_maintenance_fiber_system_components":"","head_end_b_b_u_exte_net_b_b_u":"","node_equip_b_b_u_maintenance_customer_b_b_u":"","node_equip_maintenance_transport_leadsand_connectors":"","hub_site_power":"Pro-Rata","head_end_comp_repair_general":"","network_backbone_monitoring_sheath_conduit_innerduct":"","head_end_b_b_u_maintenance_exte_net_b_b_u":"","node_equip_b_b_u_repair_exte_net_b_b_u":"","node_equip_monitoring_active_node_components":"","head_end_b_b_u_repair_customer_b_b_u":"","hub_site_repair":"","head_end_b_b_u_initial_spares_customer_b_b_u":"","network_backbone_maintenance_general":"","node_equip_initial_spares_general":"","node_equip_initial_spares_power_leadsand_connectors":"","node_equip_replacement_spares_general":"","head_end_comp_monitoring_customer_r_f_source":"","head_end_equip_replacement_spares_customer_r_f_source":"","major_threshold":"10.00","node_equip_b_b_u_customer_b_b_u":"","node_equip_repair_general":"","node_equip_maintenance_power_leadsand_connectors":"","node_equip_monitoring_antenna":"","critical_threshold":"50.00","network_backbone_initial_spares_fiber_system_components":"","node_equip_b_b_u_repair_general":"","head_end_equip_initial_spares_general":"","node_equip_b_b_u_replacement_spares_customer":"","node_equip_maintenance_combinersand_splitters":"","head_end_comp_monitoring_exte_net_distributed_network":"","node_equip_repair_transport_leadsand_connectors":"","is_service_impacting":"false","network_backbone_replacement_spares_sheath_conduit_innerduct":"","node_equip_replacement_spares_active_node_components":"","head_end_b_b_u_customer_b_b_u":"","head_end_b_b_u_repair_exte_net_b_b_u":"","head_end_b_b_u_maintenance_customer_b_b_u":"","network_backbone_replacement_spares_fiber_system_components":"","head_end_equip_initial_spares_exte_net_distributed_network":"","network_name":"SE-FL-MARLINS-OPN","head_end_comp_repair_exte_net_distributed_network":"","minor_threshold":"0.00","node_equip_maintenance_antenna":"","element_total":"129","node_equip_b_b_u_maintenance_exte_net_b_b_u":"","network_backbone_replacement_spares_fiber_optic_cabling":"","head_end_comp_maintenance_customer_r_f_source":"","node_equip_b_b_u_replacement_spares_exte_net":""},{"head_end_equip_replacement_spares_exte_net_distributed_network":"","head_end_b_b_u_repair_general":"","node_equip_repair_combinersand_splitters":"","head_end_b_b_u_replacement_spares_customer_b_b_u":"","network_backbone_replacement_spares_general":"","node_equip_maintenance_active_node_components":"","major_restore":"90","head_end_equip_initial_spares_customer_r_f_source":"","network_backbone_monitoring_general":"","carrier":"NoCarrierAssigned","node_equip_replacement_spares_combinersand_splitters":"","network_backbone_initial_spares_general":"","network_backbone_initial_spares_sheath_conduit_innerduct":"","node_equip_monitoring_combinersand_splitters":"","hub_site_general":"","head_end_comp_monitoring_general":"","network_backbone_repair_fiber_system_components":"","node_equip_initial_spares_transport_leadsand_connectors":"","node_equip_repair_power_leadsand_connectors":"","node_equip_repair_antenna":"","node_equip_initial_spares_active_node_components":"","network_backbone_repair_splice_boxes":"","node_equip_monitoring_transport_leadsand_connectors":"","head_end_b_b_u_initial_spares_exte_net_b_b_u":"","network_backbone_replacement_spares_splice_boxes":"","head_end_b_b_u_maintenance_general":"","FORM_NAME_PARAM":"ServiceEntitlements","node_equip_b_b_u_initial_spares_general":"","head_end_equip_replacement_spares_general":"","network_backbone_repair_sheath_conduit_innerduct":"","node_equip_b_b_u_maintenance_general":"","minor_restore":"120","node_equip_replacement_spares_transport_leadsand_connectors":"","node_equip_monitoring_power_leadsand_connectors":"","hub_id":"","node_equip_replacement_spares_antenna":"","node_id":"n/a","node_equip_initial_spares_combinersand_splitters":"","hub_site_maintenance":"","node_equip_monitoring_general":"","network_backbone_maintenance_fiber_optic_cabling":"","network_backbone_maintenance_sheath_conduit_innerduct":"","node_equip_b_b_u_general":"","node_equip_replacement_spares_power_leadsand_connectors":"","critical_restore":"120","node_equip_initial_spares_antenna":"","node_equip_maintenance_general":"","head_end_comp_maintenance_general":"","node_equip_b_b_u_repair_customer_b_b_u":"","node_equip_repair_active_node_components":"","head_end_comp_repair_customer_r_f_source":"","node_equip_maintenance_planned_maintenance":"","head_end_b_b_u_replacement_spares_exte_net_b_b_u":"","head_end_comp_maintenance_exte_net_distributed_network":"","head_end_b_b_u_general":"","node_equip_b_b_u_exte_net_b_b_u":"","network_backbone_monitoring_fiber_optic_cabling":"","network_backbone_repair_general":"","network_backbone_monitoring_splice_boxes":"","node_site_replacement_spares_pole_general":"","network_backbone_monitoring_fiber_system_components":"","comments":"","network_backbone_repair_fiber_optic_cabling":"","network_backbone_maintenance_splice_boxes":"","network_backbone_maintenance_fiber_system_components":"","head_end_b_b_u_exte_net_b_b_u":"","node_equip_b_b_u_maintenance_customer_b_b_u":"","node_equip_maintenance_transport_leadsand_connectors":"","hub_site_power":"","head_end_comp_repair_general":"","network_backbone_monitoring_sheath_conduit_innerduct":"","head_end_b_b_u_maintenance_exte_net_b_b_u":"","node_equip_b_b_u_repair_exte_net_b_b_u":"","node_equip_monitoring_active_node_components":"","head_end_b_b_u_repair_customer_b_b_u":"","hub_site_repair":"","head_end_b_b_u_initial_spares_customer_b_b_u":"","network_backbone_maintenance_general":"","node_equip_initial_spares_general":"","node_equip_initial_spares_power_leadsand_connectors":"","node_equip_replacement_spares_general":"","head_end_comp_monitoring_customer_r_f_source":"","head_end_equip_replacement_spares_customer_r_f_source":"","major_threshold":"10.00","node_equip_b_b_u_customer_b_b_u":"","node_equip_repair_general":"","node_equip_maintenance_power_leadsand_connectors":"","node_equip_monitoring_antenna":"","critical_threshold":"50.00","network_backbone_initial_spares_fiber_system_components":"","node_equip_b_b_u_repair_general":"","head_end_equip_initial_spares_general":"","node_equip_b_b_u_replacement_spares_customer":"","node_equip_maintenance_combinersand_splitters":"","head_end_comp_monitoring_exte_net_distributed_network":"","node_equip_repair_transport_leadsand_connectors":"","is_service_impacting":"false","network_backbone_replacement_spares_sheath_conduit_innerduct":"","node_equip_replacement_spares_active_node_components":"","head_end_b_b_u_customer_b_b_u":"","head_end_b_b_u_repair_exte_net_b_b_u":"","head_end_b_b_u_maintenance_customer_b_b_u":"","network_backbone_replacement_spares_fiber_system_components":"","head_end_equip_initial_spares_exte_net_distributed_network":"","network_name":"SE-FL-MARLINS-OPN","head_end_comp_repair_exte_net_distributed_network":"","minor_threshold":"0.00","node_equip_maintenance_antenna":"","element_total":"129","node_equip_b_b_u_maintenance_exte_net_b_b_u":"","network_backbone_replacement_spares_fiber_optic_cabling":"","head_end_comp_maintenance_customer_r_f_source":"","node_equip_b_b_u_replacement_spares_exte_net":""}],"ticketValues":{	"WORKFLOW":"ExteNetDASAlarm","SUMMARY":"SE-FL-MARLINS-OPN-VOLTSERVER3-3200002170:TransmissionFault","CRITICAL_ELEMENT_TICKET":"false","VISIBILITY":"FL-Marlins","PRIORITY":"2","CREATOR":"monitor","STATUS":"AlarmReceived","NETWORK_IDENTIFIER":"SE-FL-MARLINS-OPN","MARKET":"FL-Marlins"},"eventCode":"TICKET_CREATE_INITIATED","requestRetryCount":"5"}

Once the JSON is corrected, navigate to the remote.ticket.create.queue and under publish message, paste the correct JSON into the payload field, ensuring Delivery Mode is set to 2 - Persistent and click Publish Message

At this stage, the ticket will try to recreate. To verify, check in the ticketer ticket table to see if a ticket was created. No remote ticket will be created, as this is deleted after all the retries fail.

Finally, once the messages have been re-published, return to the remote.ticket.dead.letter queue and click Purge to clear the dead queue and clear the alert.