Level: Critical

Purpose: To ensure we don't accidentally hit a linux thread limit.

Scenario: How its calculated is like this. First we get the maximum number of threads allowed on the system for scotty with

 ulimit -u 

On EXT this is around 5120.

Next we target the number of active processes in PS using WC to count it.

 top -b -H -u scotty -n 1 | wc -l 

Then if that result of is greater than 90% of the allowed limit, this Watchdog will fire.

Resolution: Get the thread count under control

You'll probably have to kill something.

There are two other things. Ansible eats up threads at an alarming rate for a brief moment. This might trigger it. So be sure to check if its still a problem after the Watchdog has fired.

As well as that, you might have to raise the ulimit for the number of allowed threads. Bear in mind that this ulimit is for the user scotty and it shouldn't get this high. Its likely indicative of a worse problem. Each thread will comsume a certain amount of memory and that limit may be hit before the thread limit on the system itself.

Manual Action Steps: Please run

 top -b -H -u scotty -n 1 | wc -l 

to ensure its still a problem.

If it is use top to see what is taking all the threads, you may have to kill it top -b -H -u scotty -n 1

Auto Clear: Yes, it can. Hopefully it will.