User Tools

Site Tools


resolution_area:watchdog_resolutions:res-w8603

FiberImportMissingFor48hrs

Level: Major

Purpose:

This is to ensure that the Fiber tool is working and updated regularly.

Fiber is only used for ExteNet.

Scenario: Watchdog: [name: 'checkLog', type: 'EQUAL', value: ['Stale import check - no automated imports found for past 48 hours'], level: ['MAJOR'], alarmId: 'FiberImportMissingFor48hrs']

This watchdog looks back about 4 hours in the logs for the line in the value field above. If there hasn't been a successful import in 48 hours, this will be printed to the logs.

Further details on the specifics are here https://docs.google.com/document/d/1zO4FdZSOijXBcqedM8afvp3xKOgzIVuHEOi_UdQMREo/edit#

Resolution:

Manual Action Steps: Probably have to restart the Fiber application

Ensure you have the latest of the env-config repo and the deployments-playbook repo from bitbucket. Assuming you have both repositories in the same folder then run

cd /your/env-config/location
git checkout master && git pull && cd ../deployment-playbooks && git checkout master && git pull

Restart the application with

ansible-playbook -i ../env-configuration/ServerName/hosts.ini fiber.yml --vault-id @prompt -e"actions='stop,start'" 

NOTE: Change ServerName and AppName for the actual server and application name

Auto Clear: It will autoclear once the watchdog doesn't see the log line 4 hours back. So it may take 4 hours to auto clear even if it recovered. However, this is unlikely to automatically clear on its own since the line wouldn't have printed if there was a sucess in the last 48 hours.

resolution_area/watchdog_resolutions/res-w8603.txt · Last modified: 2021/07/05 16:55 by wflaherty