Author: Cathal O'Keeffe 2018/11/01

This is a quick overview of the lifecycle of a trap as it goes through the SNMP Manager and becomes a ticket, with references to important classes and methods along the way.

On an SNMP Manager instance assigned as a distributor (see: ThreadConfig and ThreadType) the SnmpDistributor class binds to a port (default 162) and listens for any incoming messages. It uses a library created by iReasoning, together with data from the server's MIB files, to parse the trap information. This starts in the handleMsg method.

It then checks the IgnoreTrap and TrapAggregate tables to see if received trap should be ignored or aggregated instead of processed now.

If neither of those are needed it forwards the trap onto an appropriate handler instance. The first time a particular trap type comes in from a given host it will be sent to the currently least busy handler, and from then on if the same type of trap comes in from the same host it will be sent to the same handler until the SnmpDistributorStatisticSet is cleared either manually or by restart of the distributor instance.

It is sent via DatagramSender with a default port of 4445.

On the handler the DatagramReceiver listens for incoming messages (again, default port 4445) and gets passed to the local SnmpDistributor and then to the TrapHandlerPool. The TrapHandlerPool finds a read TrapHandler and adds it to its TrapQueue while saving the Trap and Varbind entries to the database.

Each TrapHandler thread will continuously work through any traps in its queue and hand them over to a TrapProcessor. The TrapProcessor will use the CompiledRuleCache to retreieve a TrapRule entry as a closure and runs it, passing in the Trap domain object.

Trap rules are defined and runtime and can do just about anything, but there is a standard trap rule template we use.

Generally the trap rules take the given Trap object and build a standardised GeneralTrapSummary from this information and saves it. It then wraps it in a GeneralTrapSummaryWrapper and tries to find a matching, open ActiveAlarm for the given GeneralTrapSummary.

If it finds an open alarm and the trap is a clear trap then the alarm is cleared and saved. If no alarm exists yet it builds a map of appropriate ticket and form values before calling GeneralTrapSummaryWrapper.scheduleAlarmAndTicketForLaterWithForms with a ticket creation delay calculated based off severity of the alarm status.

GeneralTrapSummaryWrapper.scheduleAlarmAndTicketForLaterWithForms defines a groovy closure and passes into the scheduler to run (usually after 10 seconds) via the scheduleAlarmForLater method. This closure creates and saves an ActiveAlarm and defines a second closure that is also passed into the scheduler to be run between fifteen minutes to two hours later depending on the alarm deverity. This second closure calls RemoteTicketCreator.openTicketWithForms to create a ticket iff the alarm created in the previous closure still exists, hasn't cleared and doesn't already have a RemoteTicket associated with it.

RemoteTicketCreator.openTicketWithForms creates a new RemoteTicketCreationTask with all the necessary data and forms and passes it to an ExecutorService to run. This passes on the ticket information to RemoteTaskService.initRemoteTicketProcess which uses Java Remote Method Invocation (RMI) to call the RemoteTicketSerice in the local Ticketer instance, through the TicketerInterface interface which exists in both the SNMP Manager and Ticketer codebases.