Internal Errigal Collaboration Wiki

User Tools

Site Tools

Trace: • res-p1701
resolution_area:prometheus_resolutions:res-p1809

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
resolution_area:prometheus_resolutions:res-p1809 [2021/07/06 14:11] – wflahertyresolution_area:prometheus_resolutions:res-p1809 [2022/05/03 15:28] (current) – 10.5.5.52
Line 26: Line 26:
 You can check specifically for what server RabbitMQ is on for an environment with this.  You can check specifically for what server RabbitMQ is on for an environment with this. 
 https://docs.google.com/spreadsheets/d/1Ebj0kWPl63Q4L3f_oo_kvWoh5ZRq2vFI8FmrWbjoPlo/edit?usp=sharing https://docs.google.com/spreadsheets/d/1Ebj0kWPl63Q4L3f_oo_kvWoh5ZRq2vFI8FmrWbjoPlo/edit?usp=sharing
-Scroll across to see what RabbitMQ is running on. +Scroll across to see what RabbitMQ is running on. However, this information should be in the alert received.
  
  
Line 49: Line 49:
 ps aux | grep 'java' | grep -i 'snmp'  ps aux | grep 'java' | grep -i 'snmp' 
 ps aux | grep 'java' | grep -i 'reporting'  ps aux | grep 'java' | grep -i 'reporting' 
-ps aux | grep 'elastic' +ps aux | grep 'elastic'
 </code> </code>
  
Line 93: Line 93:
  
 If there are many 'HANDLER is DEAD' messages then there is trouble connecting the loadbalancer with the handlers. It should be mostly 'HANDLER is Alive' If there are many 'HANDLER is DEAD' messages then there is trouble connecting the loadbalancer with the handlers. It should be mostly 'HANDLER is Alive'
 +
 +
 +Check the database with
 +<code>
 +select * from thread_config tc
 +</code>
  
 ----- -----
Line 114: Line 120:
 There is also a possibility that a queue was refactored in the codebase or script but not updated in RabbitMQ. Check with the devs for this.  There is also a possibility that a queue was refactored in the codebase or script but not updated in RabbitMQ. Check with the devs for this. 
  
 +**Ebonding**
 +
 +If the alert is related to an eBonding queue:
 +  * eBonding_outbound_queue
 +  * eBonding_inbound_queue
 +
 +Check if soap and ebonding applications are running
 +
 +<code>
 +ps -edf | grep -i soap
 +ps -edf | grep -i ebonding 
 +</code>
 +Verify that the logs are not frozen: log location is: logs/grails
 +
 +An application restart will be required if ebonding & soap logs are frozen.
  
 **Auto Clear:** **Auto Clear:**
  
 The alert will auto clear once there are fewer than 1000 messages in the queue.  The alert will auto clear once there are fewer than 1000 messages in the queue. 
resolution_area/prometheus_resolutions/res-p1809.1625577096.txt.gz · Last modified: 2021/07/06 14:11 by wflaherty

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki