Internal Errigal Collaboration Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.

=====ElastAlertNoScrape=====

**Level:** Critical


**Purpose:** Notify Errigal Operations staff that the elastalert application on prometheus prod has not completed a scrape on the DF-80 (80% disk full) or DF-60 (60% disk full) Elast Alert rules in the last 2 minutes.


**Resolution:**
Elastalert runs in a docker container on prometheus prod. It is monitoring NUCS Disk space in KLAs network by checking elastic search data that has pulled the Operating system stats. If these dont catch log files filling up the HDD the NUC will go offline. 

**Manual Action Steps:**
<code>sudo docker logs --tail 100 elastalert --follow</code>

<code>sudo docker restart elastalert</code>

A restart should do the trick if there was an error and this caused the application to disable the rules.

Observe logs and be sure that the logs show elastalert running the query for both DF-80 and DF-60 and that none of them are disabled. These two lines should be present. 

<code>INFO:elastalert:Queried rule df-80 from 2024-08-06 19:41 UTC to 2024-08-06 19:52 UTC: 0 / 0 hits</code>

<code>INFO:elastalert:Queried rule df-60 from 2024-08-06 19:41 UTC to 2024-08-06 19:53 UTC: 0 / 0 hits</code>

This line below is showing that df-80 is disabled. Thats no good, restart the application to get it back.

<code>INFO:elastalert:Disabled rules are: ['df-80']</code>

Then you should see 

<code>INFO:elastalert:Disabled rules are: []</code>

Another thing to check is the rule yaml files

<code>cd /home/scotty/elastalert_config/rules</code>

<code>sudo vi df-60.yaml or sudo vi df-80.yaml</code>

Ensure the setting is_enabled is set to true.

**Auto Clear:** Will auto clear if the elastalert_scrapes_total counter increases again

Sidebar

Internal Errigal Collaboration Wiki

User Tools

Site Tools

Page Tools