Overall investigation is a difficult task to summarise as there are a variety of things that could be the issue. It is important to maintain a level head and not be overwhelmed. A good method of following for actually solving a problem is 1) Recreate the Problem and verify it is still there 2) Apply your proposed solution and try recreate the problem multiple times. If it does not happen again you can say you've fixed the issue but if it happens again then your solution is not correct and needs to be rethinked and retested.
As regards logs the word Exception is always a good one to look out for….except in the Ticketer (circa Eoin Joy 2017) as some exception are “okay” (more so they happen and are not fixed yet but don't cause any issues).
OutOfMemory is a big issue which usually occurs when the JVM heap has gone over what it is allowed, The Perm Size have gone over what it is allowed or Java isn't allow spin up any more threads. There are other causes but if you see this it is a major red flag, needs investigation and of course the App will need to be restarted.
Some points to check :
Snmp Active Alarm Exceptions may not be serious* - does need to be fixed eventually
MySql Exceptions can be a big issue
Trap processing fail over and how to verify: https://www.evernote.com/Home.action#n=11c03334-052f-443b-878a-ac6c5d204025&ses=1&sh=5&sds=5&x=fail%2520over&
It is good to shadow other people on Watchdogs as watchdogs let us know when problems we have encountered before have occurred and some of these problems can be esoteric. A good rule of thumb is that if you don't know what the watchdog email is for, go to the server that sent the alarm, go into the watchdog ResourceConfig.groovy, find the watchdog and see if you can see what parameters must be met to send the alarm and from this you can figure out what the alarm is monitoring. From here then you can investigate the issue.
Alarms remember if you think the issue you are investigating is big or possibly could be a larger issue and you cannot fix it, tell other people and ask for help!