Internal Errigal Collaboration Wiki

User Tools

Site Tools

Trace: • res-p1407 • res-p1408 • res-p1406 • res-p1403 • res-p1404 • res-p1401
resolution_area:prometheus_resolutions:res-p1407

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

resolution_area:prometheus_resolutions:res-p1407 [2022/09/30 12:32] – created 10.91.110.100resolution_area:prometheus_resolutions:res-p1407 [2022/09/30 12:52] (current) – 10.91.110.100
Line 1: Line 1:
-=====AtcDasTicketMissingSalesForceLink=====+=====LogFileSizeIncreasing=====
  
-**Level:** Major+**Level:** Major/Critical
  
  
-**Purpose:** Notify operations that an ATC Das ticket was not created in Salesforce+**Purpose:** Notify operations that a log file on the server has a rapidly growing log line count
  
-**Scenario:** More than one ATC DAS ticket was not created in Salesforce for the previous 7 days.+**Scenario:** An error has occurred in an application and is appending the log at a rapid rate, filling up the disk space.
  
-   
  
-**Resolution:** After the 3.21 release a API will be available to recreate the missing link.+**Resolution:** Log into the server and check which log file is increasing.
  
-Ticketer/api/v1/integration/salesforce/link/${ticketId}+<code>find ~/logs/ -name "*.log" -exec ls -lh {} \;</code>
  
-This can be completed in a Postman POST request or using the below cURL command: +If the file size is not too large, tail it and note the exception/error that is causing the file increase.
- +
-<code>curl --request POST -i -u '<USERNAME>':'<PASSWORD>' <DOMAIN>/Ticketer/api/v1/integration/salesforce/link/<TICKET-ID></code> +
- +
-For ATC +
-<code>curl --request POST -i -u '<USERNAME>':'<PASSWORD>' http://atcwatchdog.com/Ticketer/api/v1/integration/salesforce/link/<TICKET-ID></code> +
-Before this is implemented the only way an alert will be cleared is if ATC DAS ticket is auto cleared.+
  
 **Manual Action Steps:** **Manual Action Steps:**
-Check for ATC Tickets with no Salesforce link +If necessary, truncate the file, do not delete it. 
- +
-    select +
-        t.* +
-    from +
-        ticketer.ticket t +
-            left join +
-        ticketer.external_source es ON es.ticket_id = t.id +
-        left join +
-        snmp_manager.network_element child on child.id = t.network_element_id +
-        left join +
-        snmp_manager.network_element parent on parent.id = child.parent_id +
-        left join +
-        snmp_manager.network_element grandparent on grandparent.id = parent.parent_id +
-    where +
-        es.ticket_id is null +
-        and t.creator_id = 4 -- IDMS Monitor +
-        and t.workflow_id = 909 -- ATC DAS Workflow +
-        and t.create_date <= now() - interval 1 MINUTE +
-        and (t.create_date >= now() - Interval 7 day) +
-        and t.parent_id is null +
-        and (parent.name != 'ORPHANS' and grandparent.name != 'ORPHANS'); +
- +
-To fix the problem you will have to requeue the dropped messages +
-Here is a template for this request.  +
-    { +
-        "event_type": "PostInsert", +
-        "domain": "f_snmp_trap", +
-        "CN_Priority__c": "2", +
-        "Due_Date__c": null, +
-        "SLA_Date__c": null, +
-        "Originating_System_ID__c": "ERRIGALTICKET" +
-        "Description": " ", +
-        "Alarm_Start_Date__c": "", +
-        "Alarm_Type__c": "errigalMonitoredCarrierDeviceMissingAlarm", +
-        "trap_name": "errigalMonitoredCarrierDeviceMissingAlarm", +
-        "Alarm_System_Link__c": "https://atcwatchdog.com/SnmpManager/errigalEMS#parent/223393", +
-        "Alarm_IP_Source__c": "10.32.30.57", +
-        "INT_Integration_ID__c": "223393", +
-        "Optima_Network_Element__c": "209694 MBTA - 209694 MBTA - Solid DMS#2" +
-    } +
- +
- +
-You will need to replace the IP, NE ID, Description, Errigal Ticket, Alarm Start Date, Network Element and others. +
- +
-  - Once you have edited the json with the appropriate values from the query, then you can try requeue it.  +
-  - Go to http://atclb1.atc:15672 +
-  - Click on the exchanges tab +
-  - Find `ticketer_audit_event_exchange_topic` +
-  - Under publish, use the routing key as f_snmp_trap and paste the json into the payload section +
-  - Publish the message. If there is an error it will tell you +
-  - Rerun the query to double check that the Salesforce has a case for the Ticket.  +
- +
-For more details, please check this. https://docs.google.com/document/d/1_zwi02B8WoysEUEzfIK2A046xsmFfnrYqiaO4kA7QHk/edit#heading=h.z4rx22cg5uw8 +
- +
-====Troubleshooting==== +
- +
-If the curl request is performed but the alaert doesn't clear, check the ticketer-integration logs. Example error: +
- +
-<code> +
-2022-09-29 03:08:20.563 ERROR 25776 --- [Camel (camel-1) thread #27 - rabbitmq://ticketer_audit_event_exchange_topic] utbound$$EnhancerBySpringCGLIB$$bb0998e1 : The HTTP response: {"error":"duplicate value found: Originating_System_ID__c duplicates value on record with id: 5003g00000N6V2w","success":"false"} +
-</code>+
  
-This is likely caused by a RabbitMQ issue. The message will need to be requeued.+<code>sudo echo > LOG_FILE</code>
  
-FIX: Requeue the message in the rabbit MQ queue ticketer_inbound_salesforce_queue +**Auto Clear:** Yes, when the log lines stop increasing
-<code> +
-{    "caseId": "5003g00000N6Z0TAAV",    "ticketId": 764517,    "caseNumber": "02475388"} +
-</code> +
-The case Id and caseNumber fields can be found in the original ticketer-integration logs. +
-<code> +
-zgrep  "Originating_System_ID__c=764919" ~/logs/spring-boot/ticketer-integration.log* | grep CaseNumber +
-</code> +
-<code> +
-2022-09-28 10:11:16.149  INFO 25776 --- [SalesforceHttpClient@44be69aa-42] org.apache.camel.Tracing                 :      [route11     ] [rabbitmq://{{rabbitmq.queues.sale] Exchange[Id: 7B60355912D6F25-000000000000EC78, BodyType: byte[], Body: {"Status":"New","Carrier_Update__c":null,"Force_Majeure__c":false,"CaseNumber":"02475388","CN_Priority__c":"4","Id":"5003g00000N6Z0TAAV","Originating_System_ID__c":"764517","correlationId":"bf2d5435-abed-4f0e-aa40-01fb8aa30cff"}] +
-</code> +
-**Auto Clear:** Yes, Alert will clear when the count of ATC DAS ticket was not created in Salesforce is 0+
resolution_area/prometheus_resolutions/res-p1407.1664537552.txt.gz · Last modified: 2022/09/30 12:32 by 10.91.110.100

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki