http://3.86.49.49/ 2026-04-17T13:31:40+00:00 http://3.86.49.49/ http://3.86.49.49/lib/tpl/docnavwiki/images/favicon.ico text/html 2021-06-25T10:09:56+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w1503&rev=1624612196&do=diff LineThreeFound Level: Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-25T10:09:56+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w1504&rev=1624612196&do=diff LineTwoFound Level: Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-05T12:21:07+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8101&rev=1625484067&do=diff CasApplicationServerProcessInactive Level: Critical FIXME Purpose: This alert tells us when the CAS application server is not found in the server process list Scenario: CAS is not running or has gone down on a handler Resolution: Restart the application text/html 2021-12-23T14:10:34+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8102&rev=1640268634&do=diff casApplicationServerProcessInactive Level: Critical FIXME Purpose: This alert tells us when the CAS application server is not found in the server process list Scenario: CAS is not running or has gone down on a handler Resolution: Restart the application text/html 2021-07-05T12:19:54+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8201&rev=1625483994&do=diff ROWCannotReachRabbitMQ Level: Critical FIXME Purpose: Alerts if the Right of Way application cannot reach Rabbit MQ by checking the right-of-way.log for the existence of 'Failed delivery for'. Scenario: Right of way application cannot reach RabbitMQ text/html 2021-06-25T10:09:56+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8202&rev=1624612196&do=diff RowCountBreached Level: minor Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T16:45:56+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8203&rev=1624981556&do=diff ROWTimeout Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-08T12:55:47+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8301&rev=1625745347&do=diff GisIntegrationError Level: Critical Purpose: This Watchdog picks up if the GIS Intergration has reported an error during the automated data import. If it has it will leave a line in the log. GIS Integration is responsible for handling API calls to transfer KMZ files from specified endpoints to the NOC Portal. text/html 2021-06-29T16:46:32+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8302&rev=1624981592&do=diff GisIntegrationImportMissingFor48hrs Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-05T12:29:48+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8401&rev=1625484588&do=diff BubbleAppConnectionTimeoutError Level: Major :!: Purpose: This alert tells us when there is an issue with bubble app making a database connection Scenario: Bubble app cannot connect to a database due to either a connectivity issue with the database or the number of max connections set text/html 2021-12-23T19:05:45+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8402&rev=1640286345&do=diff KeyStatsGenerationFailed Level: Major :!: Purpose: The key_status provides customers with key information from their SnmpManager&Ticketer database. It's a database of important data generated from the two databases. Scenario: Generation of the keystats table takes a considerable amount of time. If the process timeouts or is killed it will not complete. The watchdog checks for a text/html 2021-07-05T12:22:06+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8501&rev=1625484126&do=diff ebondingApplicationServerProcessInactive Level: Critical FIXME Purpose: Alerts if the eBonding process is running. Scenario: The eBonding application has stopped running. Resolution: Look at the ebonding logs for any exceptions. Check if the service is running and restart. text/html 2021-07-06T16:05:18+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8601&rev=1625583918&do=diff FiberConnectionError Level: Critical Purpose: Alert if the Fiber application has connection errors. Scenario: Checks the Fiber logs for 'Caught an exception during connection recovery' in the previous 4 hours Resolution: Restart Application Manual Action Steps: text/html 2021-06-29T16:47:48+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8602&rev=1624981668&do=diff FiberImportError Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-05T16:55:25+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w8603&rev=1625500525&do=diff FiberImportMissingFor48hrs Level: Major Purpose: This is to ensure that the Fiber tool is working and updated regularly. Fiber is only used for ExteNet. Scenario: Watchdog: [name: 'checkLog', type: 'EQUAL', value: ['Stale import check - no automated imports found for past 48 hours'], level: ['MAJOR'], alarmId: 'FiberImportMissingFor48hrs'] text/html 2021-07-05T11:55:46+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9101&rev=1625482546&do=diff activeAlarmCreationInactive Level: Critical FIXME Purpose: Alert Errigal Engineers when the alarm creation process on the SNMP Manager application is no longer functional. This is defined as the application not creating an active alarm within a defined period e.g 10 minutes without creating an alarm. text/html 2021-07-05T14:24:10+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9102&rev=1625491450&do=diff alarmAuditStuck Level: Critical FIXME Purpose: This alert is depricated Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-12-23T15:28:38+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9103&rev=1640273318&do=diff alarmCacheApplicationServerProcessInactive Level: Critical FIXME Purpose: This alert tells us when the AlarmCache application is not found in the server process list Scenario: AlarmCache is not running or has gone down on a handler Resolution: Restart the application text/html 2021-07-05T12:30:49+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9104&rev=1625484649&do=diff AlarmCacheOutOfSync Level: Critical (1200) FIXME | Major (1000) :!: | minor (800) Purpose: Alerts if the active alarm table in the SNMP manager database is out of sync with the active alarm table in the alarm cache database. Scenario: Alarm cache may not be consuming messages. Check if the logs are processing messages. Alarm cache may need a restart and the RabbitMQ queue may need to be purged. text/html 2021-07-06T16:25:53+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9105&rev=1625585153&do=diff AlarmsNotGeneratingRemoteTickets Level: minor Purpose: Alert if a remote ticket is not created for an Active Alarm Scenario: No remote ticket created for active alarm with active alarm create date between 1 day and 2 hours Resolution: Find active alarm causing issue and search SnmpManager logs for why issue occurred. text/html 2021-06-25T10:09:56+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9106&rev=1624612196&do=diff AlarmsNotInAlarmCache Level: Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-08T15:03:47+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9107&rev=1625753027&do=diff AlarmsShouldBeCleared Level: + 800 - MINOR + 1000 - MAJOR + 1200 - CRITICAL Purpose: Scenario: This watchdog checks the database for the AlarmCache to ensure that the alarms that are cleared in the SNMP Manager are removed from the AlarmCache's active alarm table text/html 2021-06-25T10:09:56+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9108&rev=1624612196&do=diff AlarmsShouldNotExist Level: Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2026-01-20T11:19:01+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9201&rev=1768907941&do=diff apps1Failover Level: Major :!: Purpose: Alerts when a handler is not recognised as a trap receiver by the load balancer Scenario: Apps1 is flagged as inactive and is not receiving traps to process. Resolution: Check the thread config to see if the handler is active or not. text/html 2021-06-29T16:59:59+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9202&rev=1624982399&do=diff autoCloseTaskBlocked Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-05T12:52:51+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9203&rev=1625485971&do=diff Cannot connect to SMTP Server Level: Critical FIXME Purpose: There was an error in the logs to indicate there's an issue connecting to the SMTP server used for sending emails. Scenario: SMTP connection has failed. Resolution: Check the SMTP connection details in the database, groovy grails config file or application.properties. Also, check the servers network. text/html 2021-07-06T16:42:44+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9204&rev=1625586164&do=diff ClientAlarmPollerInactive Level: Critical Purpose: Alerts if client active alarm poller in the SNMP Manager has stopped running; this is used to update the Node Monitor. Scenario: If the following is missing from the SNMP Manager logs in the previous 10 minutes 'CLIENT ACTIVE ALARM POLLER IS ALIVE:' text/html 2021-12-23T15:52:04+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9205&rev=1640274724&do=diff ConnectAttemptFailed Level: Critical FIXME | Major :!: Purpose: Alerts operations that there has been a connection error to the application database. Scenario: The application database couldn't be reached from the server. Resolution: Fix database connection issue. text/html 2021-12-23T13:00:49+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9206&rev=1640264449&do=diff CriticalThreadLimit Level: Critical Purpose: To ensure we don't accidentally hit a linux thread limit. Scenario: How its calculated is like this. First we get the maximum number of threads allowed on the system for scotty with ulimit -u On EXT this is around 5120. text/html 2021-06-25T10:09:56+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9207&rev=1624612196&do=diff ExceptionInvalidEmailAddress Level: minor Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-06T12:39:49+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9208&rev=1625571589&do=diff ExceptionSMTPServerUnavailable Level: Critical FIXME Purpose: This alert tells us that there is an issue with Ticketer connecting to the SMTP host for emailing. Scenario: When this line is detected in logs Could not connect to SMTP host Resolution: text/html 2021-06-29T17:01:11+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9209&rev=1624982471&do=diff ExceptionWhileSendingEmail Level: Major :!: Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-05T12:57:32+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9210&rev=1625486252&do=diff handlerFailover Level: Critical FIXME Purpose: Alerts when a handler has been marked as inactive in the snmp_manager.thread_config table Scenario: The load balancer cannot connect to the handler or the handler has died. Resolution: Check the handler logs to see if there are any obvious exceptions. Restart the SNMP manager if needed. If this does not clear it, restart the load balancer SNMP manager to allow connection to the handler again. text/html 2021-07-07T15:44:39+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9211&rev=1625669079&do=diff InvalidEmailRecipient Level: Critical Purpose: Notify operations when email sending fails in the Ticketer application due to invalid email syntax in the previous 30 minutes. Scenario: Alert is received when the following appears in the Ticketer logs: 'Invalid Email Recipients' text/html 2021-06-25T10:09:56+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9212&rev=1624612196&do=diff MinorThreadLimit Level: minor Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T17:02:10+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9213&rev=1624982530&do=diff mNetOpenVpnClientProcessInactive Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-25T10:09:56+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9214&rev=1624612196&do=diff MobiTicketInactivity Level: minor Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-06T12:32:00+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9215&rev=1625571120&do=diff MonthlyNetIdentifierCountIsZero Level: Major :!: Purpose: This alert tells us that the network identifier count query returned 0 results. This is used for billing purposes Scenario: This watchdog runs this query and alerts us if the result is 0 text/html 2021-12-23T16:51:27+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9216&rev=1640278287&do=diff monthlyNetIdentifierCountIsZero Level: Critical FIXME Purpose: The stats_network_identifier_node_count table that this watchdog checks is used by the “Availability - Carrier Availability Per Network and Month” report specific to EXT. It checks if the required data has been populated. text/html 2021-07-05T13:58:34+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9217&rev=1625489914&do=diff NocPortalApplicationServerProcessInactive Level: Critical FIXME Purpose: Checks if the NOC portal service is running on the server. Scenario: The NOC portal process has stopped running. Resolution: Check the NOC portal process - ps -edf | grep -i NocPortal text/html 2021-07-08T15:13:51+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9218&rev=1625753631&do=diff NtpdProcessInactive Level: Critical Purpose: Ntpd (Network Time Protocol Daemon) is used maintain accurate server time. Alert indicates that the process has stopped. Scenario: Ntpd process in not running Resolution: Verify that ntpd is running and restart if necessary text/html 2022-08-18T10:36:49+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9219&rev=1660815409&do=diff Quartz Job Inactive Level: Critical FIXME Purpose: Checks the quartz table for each application to see if there are any blocked jobs. select * from QRTZ_CLUSTER_TRIGGERS where TRIGGER_STATE = 'BLOCKED'; Scenario: For various reasons, a job can block. This job will need to be restarted. text/html 2021-06-29T17:03:12+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9220&rev=1624982592&do=diff QuartzJobsBlocked Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T17:03:20+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9221&rev=1624982600&do=diff remoteFileGetJobFailure Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T17:03:28+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9222&rev=1624982608&do=diff remoteFilePutJobFailure Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-12-23T17:06:02+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9223&rev=1640279162&do=diff remoteTicketCreationInactive Level: Critical FIXME Purpose: Notifies when there are no entries in the snmp_manager.remote_ticket table for over 90 minutes, which could signify that there is something wrong with the ticket creation process. Scenario: text/html 2021-07-05T13:59:14+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9224&rev=1625489954&do=diff ReportingManagerApplicationServerProcessInactive Level: Critical FIXME Purpose: Checks if the reporting manager service is running on the server. Scenario: The reporting manager process has stopped running. Resolution: Check the reporting manager process - text/html 2021-12-08T22:24:35+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9225&rev=1639002275&do=diff slaGroovletChanged Level: Critical Purpose: Alert when the American Tower 'Service Level Agreement' groovlet change is undone. User case for watchdog is that changes to the gooovlet were getting reverted automatically. Scenario: Watchdog concats DB entry for 'Service Level Agreement' and compares to a hardcoded md5sum value. text/html 2021-06-29T17:04:14+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9226&rev=1624982654&do=diff SnmpManagerApplicationServerProcessInactive Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T17:04:21+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9227&rev=1624982661&do=diff SnmpManagerProcessInactive Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T17:04:29+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9228&rev=1624982669&do=diff soapApplicationServerProcessInactive Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-06T12:14:27+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9229&rev=1625570067&do=diff statsNextTicketStatusTableNotPopulated Level: Major :!: Purpose: This alert checks whether there are entries in the reporting_stats.stats_next_ticket_status table within a time period and if there are no rows found we will be alerted Scenario: When this query returns 0 rows text/html 2021-12-23T17:16:49+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9230&rev=1640279809&do=diff SupportRequestServerProcessInactive Level: Critical FIXME Purpose: This alert tells us when the SupportRequest application server is not found in the server process list Scenario: SupportRequest is not running or has gone down on a handler Resolution: Restart the application text/html 2021-07-05T14:04:53+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9231&rev=1625490293&do=diff SyncTasksTakingTooLong Level: Major :!: Purpose: Checks if there are more than 10 sync jobs running for more than 12 hours. Scenario: Syncs jobs can take a long time and bottleneck other sync processes. The alarm checks the below query to see if there are more than 10 jobs taking longer than 12 hours. text/html 2021-07-08T15:36:11+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9232&rev=1625754971&do=diff TicketerApplicationServerProcessInactive Level: Critical Purpose: Alerts if the Ticketer application stops running. Scenario: Ticketer is not running on server. Resolution: Verify that Ticketer application is running, check logs for exceptions, & restart if required. text/html 2021-06-29T17:05:19+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9233&rev=1624982719&do=diff TrapCountBreached Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T17:05:29+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9234&rev=1624982729&do=diff TrapParsingInactive Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-12-30T15:03:07+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9235&rev=1640876587&do=diff ConfigSyncTasksNotRunning Level: Critical FIXME Purpose: This alert informs that the Config Sync Tasks have not run for the previous day. Scenario: The SnmpManager instance on which the syncs are configured to run on was down on the previous day. text/html 2021-12-30T15:02:59+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9236&rev=1640876579&do=diff PerformanceDiscoveryTasksNotRunning Level: Critical FIXME Purpose: This alert informs that the Performance Sync Task has not run for the previous day. Scenario: The SnmpManager instance on which the syncs are configured to run on was down on the previous day. text/html 2021-12-30T15:02:51+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9237&rev=1640876571&do=diff AlarmSyncTasksNotRunning Level: Critical FIXME Purpose: This alert informs that the Performance Sync Task has not run for the previous day. Scenario: The SnmpManager instance on which the syncs are configured to run on was down on the previous day. text/html 2021-12-30T15:02:39+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9238&rev=1640876559&do=diff AlarmSeveritySyncTasksNotRunning Level: Critical FIXME Purpose: This alert informs that the Performance Sync Task has not run for the previous day. Scenario: The SnmpManager instance on which the syncs are configured to run on was down on the previous day. text/html 2022-11-08T14:25:56+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9239&rev=1667917556&do=diff ElasticSearchKibanaError Level: Critical FIXME Purpose: This alert informs that the MDC tracker cannot write to Elastic Search. Scenario: The tracker log contains the following: “Failed to write object due to ElasticSearch Status Exception”. Check the reason in exception message. text/html 2022-10-16T19:18:37+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9240&rev=1665944317&do=diff NetaceTicketCreation Level: Critical Purpose: Notify operations that Netace tickets are not been created. Scenario: No Netace tickets have been created in the previous hour. Resolution: Verify that the netace_adapter is running ps -edf | grep -i netace-adapter text/html 2021-07-05T20:28:36+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9301&rev=1625513316&do=diff HighCPU Level: Critical FIXME Purpose: Alert Errigal engineers to excessive CPU usage on an application server. Excessive CPU usage over a period of time is indicative of an underlying problem that should be investigated. Scenario: CPU usage over period of time has been very high. text/html 2021-06-25T10:09:56+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9302&rev=1624612196&do=diff LineOOMFound Level: Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T17:06:22+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9303&rev=1624982782&do=diff NetworkIdentifierCountStatsNotPopulating Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-05T14:09:52+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9304&rev=1625490592&do=diff PotentialHandlerDeath Level: Critical FIXME Purpose: Checks if the handler is about to be removed from the active handlers in the snmp_manager.thread_config table. Scenario: The logs show the following error message in the last 10 minutes: Timeout: Pool empty. Unable to fetch a connection text/html 2021-06-29T17:06:39+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9305&rev=1624982799&do=diff The Sync Reports repository is unavailable Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T17:06:48+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9306&rev=1624982808&do=diff TheURLisUnavailable Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T17:06:55+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9401&rev=1624982815&do=diff FreeRAMLow Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-05T20:47:43+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9402&rev=1625514463&do=diff GarbageCollectionOutOfMemoryErrorFound Level: Critical FIXME Purpose: The monitored application has used all its memory resources, garbage collection is yields little to no return resources. If the application is still running at this stage is unlikely or barely unusable. Actions need to be taken to resolve the problem text/html 2021-07-05T16:25:59+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9403&rev=1625498759&do=diff HeapOutOfMemoryErrorFound Level: Critical FIXME Purpose: This alert comes in if the application has run out of available heap. Scenario: This appears in application logs java.lang.OutOfMemoryError: Java heap space Resolution: Check the available heap on the application and increase if needed. Then restart the application. text/html 2021-12-23T17:53:51+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9404&rev=1640282031&do=diff JVMHeapSizeHigh Level: Critical FIXME Purpose: Notifies that the Java Heap for the application has reached a critical size. This signifies that the application might failover soon. Scenario: The Java heap is the area of memory used to store objects instantiated by applications running on the JVM. If a lot of objects are being held in memory the the application might run out of heap - could occur during a heavy task such as report export etc. text/html 2021-07-05T14:14:00+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9405&rev=1625490840&do=diff JVMNonHeapMemoryHigh Level: Critical FIXME Purpose: Reports if the non heap memory for the application reaches above 90%. Scenario: The application is holding data in non heap memory which is nearly full. Resolution: Take the outage scripts which includes a heap dump and thread dump. Restart application. text/html 2021-07-08T16:12:01+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9406&rev=1625757121&do=diff OutOfMemoryErrorFound Level: Critical Purpose: Alerts if the application has run out of memory in the previous 30 minutes. Scenario: Watchdog will fire if the following appears in any of the groovy application logs 'java.lang.OutOfMemoryError' Resolution: text/html 2021-06-29T17:07:33+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9407&rev=1624982853&do=diff PermGenErrorFound Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T17:07:41+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9408&rev=1624982861&do=diff PermGenOutOfMemoryErrorFound Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-05T20:37:50+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9501&rev=1625513870&do=diff LinkDownPollerProcessInActive Level: Critical FIXME Purpose: Link Polling checks the customer's network elements for their link status i.e are they down or not. This process creates alarms and tickets for the customer's equipment if it is not reachable. If this process has stopped working, it means our customers are not getting alerted to potential outages. text/html 2022-03-06T10:43:19+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9502&rev=1646563399&do=diff LinkUpPollerProcessInActive Level: Critical FIXME Purpose: To let us know if the Link Poller is down on a system Scenario: This log line is not found in the SnmpManager logs Total LinkPoller tasks for LinkUpPoller Resolution: Restart the link poller text/html 2021-12-23T18:00:14+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9601&rev=1640282414&do=diff LsyncMasterProcessInactive Level: Critical FIXME Purpose: Notify that the lsync master process crashed. Scenario: If the configuration is incorrect or if the process fails for any reason this watchdog will go off. Resolution: The process becomes active again. text/html 2022-10-07T10:50:25+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9602&rev=1665136225&do=diff LsyncSlaveProcessInactive Level: Critical FIXME Purpose: Alerts if the lsync process is not running on a slave server Scenario: The lsync process uses a master and slave servers. If the process is not running, this alert will fire. Check the process with command text/html 2021-07-08T16:25:12+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9701&rev=1625757912&do=diff freeSpacePercentLow Level: Critical (10%) | minor (20%) Purpose: Identify when there is only 20% or 10% of disk space left Scenario: * Critical - 10% disk space left on server * Minot - 20% disk space left on server Resolution: Check free space on serve & remove any large temporary files text/html 2021-12-23T16:41:28+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9702&rev=1640277688&do=diff generalTrapSummaryCreationInactive Level: Critical FIXME Purpose: Notify operations that generalTrapSummary creation has stopped. Scenario: No generalTrapSummary entries have been created in the previous 5 minutes. Resolution: Will clear when new general trap summary entries are created. text/html 2022-02-14T16:38:52+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9801&rev=1644856732&do=diff DatabaseReconnectionIssue Level: Critical Purpose: This alert is triggered by a log line in the displaying an error connecting to the MySQL database. This will only appear for ATC, KLA and SCO. It is a common one if the AWS connection was denied. text/html 2021-06-29T17:09:14+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9802&rev=1624982954&do=diff mysqlDefaultEngineIncorrect Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-07-05T16:44:05+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9803&rev=1625499845&do=diff mysqlFailure Level: Critical FIXME Purpose: To tell us when replication is not working Scenario: When the results of this query are 0 (EXT example) SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST WHERE user='slave' AND host LIKE '${expectedConnectedSlaveHost}% text/html 2023-03-23T14:34:16+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9804&rev=1679582056&do=diff mysqlLongQueryRunning Level: Critical FIXME Purpose: To notify about long running queries which can negatively affect database performance. Some of these queries might not finish executing for days. Scenario: A query can be executed, usually by the user, which requires too many resources to process in a feasible amount of time. This also happens often for reports that use too wide of a date range, or in heavy reports that are usually scheduled. text/html 2021-07-05T14:22:59+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9805&rev=1625491379&do=diff mysqlSlaveReplicationFailure Level: Critical FIXME Purpose: Alerts when the command SHOW SLAVE STATUS; has seconds behind master greater than 5 minutes. Scenario: The replica database has fallen behind the master. This is usually caused by an error in replication or the replica server running slower than the master due to a long running query. text/html 2021-06-29T17:09:57+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9806&rev=1624982997&do=diff nonInnoDBApplicationTablesPresent Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2021-06-29T17:10:04+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9807&rev=1624983004&do=diff TablesExperiencingLocking Level: Critical FIXME Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2022-01-04T16:07:43+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9901&rev=1641312463&do=diff Config Sync Tasks Not Running Level: Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: text/html 2022-01-04T16:07:10+00:00 Anonymous (anonymous@undisclosed.example.com) http://3.86.49.49/doku.php?id=resolution_area:watchdog_resolutions:res-w9902&rev=1641312430&do=diff Performance Discovery Tasks Not Running Level: Purpose: Scenario: Resolution: Manual Action Steps: Auto Clear: