Differences

This shows you the differences between two versions of the page.

--- resolution_area:watchdog_resolutions:res-w9206 [2021/06/25 10:09] – external edit 127.0.0.1
+++ resolution_area:watchdog_resolutions:res-w9206 [2021/12/23 13:00] (current) – wflaherty
@@ Line 3: / Line 3: @@
 **Level:** __Critical__
+**Purpose:** To ensure we don't accidentally hit a linux thread limit.
-**Purpose:**
+**Scenario:**
+How its calculated is like this.
+First we get the maximum number of threads allowed on the system for scotty with <code bash> ulimit -u </code>
+On EXT this is around 5120.
-**Scenario:**
+Next we target the number of active processes in PS using WC to count it.
+<code bash> top -b -H -u scotty -n 1 | wc -l </code>
+Then if that result of is greater than 90% of the allowed limit, this Watchdog will fire.
 **Resolution:**
+Get the thread count under control
+You'll probably have to ''kill'' something.
+There are two other things.
+Ansible eats up threads at an alarming rate for a brief moment. This might trigger it. So be sure to check if its still a problem after the Watchdog has fired.
+As well as that, you might have to raise the ulimit for the number of allowed threads. Bear in mind that this ulimit is for the user scotty and it shouldn't get this high.
+Its likely indicative of a worse problem. Each thread will comsume a certain amount of memory and that limit may be hit before the thread limit on the system itself.
 **Manual Action Steps:**
+Please run
+<code bash> top -b -H -u scotty -n 1 | wc -l </code>
+to ensure its still a problem.
+If it is use top to see what is taking all the threads, you may have to kill it
+''top -b -H -u scotty -n 1''
 **Auto Clear:**
+Yes, it can. Hopefully it will.

Sidebar

Internal Errigal Collaboration Wiki

Differences

Sidebar

Internal Errigal Collaboration Wiki

User Tools

Site Tools

Differences

Page Tools