David McGee
Weds, 24th May 2017
This article assumes prior knowledge of AMI (Advanced Metering Infrastructure) Networks, ANSI C12.19 standards as well as an indepth knowledge of Silver Spring Networks (SSN) UIQ platform. This article has purposely not referenced customer details and/or cloud deployment for security purposes. It has, however, been written for a specific installation/deployment.
Important Entities
There are a number of important classes/entities with regards to the HSIM temperature scanning and trending functionality. This list is not exhaustive, but should give you an idea of how the process works.
AmiFullTableScanSchedule - A representation of an AMI table that needs to be scanned/processed/analysed on a given interval. This class does the necessary work for checking if a table result needs to be analysed, if its already being processed, if a new SoapMessageAuditEvent needs to be created on the queue to pull new results, or if the schedule has already been analysed for the given time interval. See performFullTableScanProcessingWork() method in this class. Configurable properties allow the data type, parameter name, sync interval, job id, threshold collection, associated soap message, trending config, active alarm identifier and trap oid to be set. This is an important entity that allows communication between Errigal HSIM and SSN (Silver Springs Networks) UIQ platform via SOAP web services.
AmiFullTableScanTrendingConfig - This allows the configuration of a number of trending options that can be changed at run-time. Trending configs are associated with AmiFullTableScanSchedules (object referenced above). This allows us to specify trending interval (SyncInterval), trending repeat count (i.e. 24), associated trending soap messages, whether we should continue/discontinue trending after an alarm clears, and also allows us to mark it active=true/false.
AmiFullTableScanResult - Everytime a schedule runs, it creates an AmiFullTableScanResult. This allows the application to become aware that something is currently persisting DiscoveredData for a schedule, or some part of the application is currently analysing this data. Due to the highly concurrent nature of requesting information (as well as the high volume of data) from AMI software such as SSN UIQ, it is important to be aware of what is currently being processed/analysed. For this reason, important date/time fields were added to this table that are automatically timestamped at the start of discovering data/the end of discovering data, as well as at the start of analysing data/the end of analysing data. Fields are as follows: discoveredDataBeginDate, discoveredDataFinishDate, analyseDataBeginDate, analyseDataFinishDate.
SoapMessage - A customisable message that can be associated with an AmiFullTableScanSchedule. Important fields include: name, xmlBody, messageType, soapEndpoint, asynchronous, jobIdRequest, resultsRequest.
ErrigalAmiNetworkSnmpTrap - An SNMP trap that can be associated with a schedule. Provided that there is also a ThresholdCollection associated with the schedule, this trap can throw an AdvancedThreshold is breached. A custom alarming OID can be associated with the schedule for when there are new notification/trap types added to the Errigal MIB.
Walkthrough
As of writing (24th May, 2017) - Here is how the HSIM system is currently scanning and trending temperature data:
It is important to note that the SoapMessageAuditEvent queue is the backbone to scanning and trending temperature data functionality working. A number of SoapMessages are configured on our AWS cloud deployment with associated Remote Discovery Framework (RDF) scripts to prepare XML requests using parameters, as well as parse responses from SSN UIQ. Please see the important SOAP discovery configuration section below for further info.
A number of statuses can be found based on method returns (isInSync(), isCurrentlyProcessingData(), hasResultReadyForAnalysis()). If there is no schedule data (it is unsynced) and no result ready to process, a pending event is created on the queue (SoapMessageAuditEvent) for the schedule in question. See Schedule States below for further detail on how the code works out the current state of a schedule.
If a schedule needs to be processed (it has response data ready to be processed from a SOAP Message web services request), the analyse() method on AmiFullTableScanResult will be called. There is a number of aims here: Find all relevant schedule data (using schedule.discoveredDataParameterName - i.e. 'Temperature'), check the data for threshold breaches, send SNMP traps to create active alarms/clear active alarms where necessary, and create/stop trending events where necessary.
For each trending event in a schedule, if the current scheduledRepeatCount > 0, a follow-up trending event is scheduled for the future (based on a sync interval defined in the associated TrendingConfig). This is 1 hour by default. The scheduledRepeatCount on the initial audit event is set to 0, and a new event is created with scheduledRepeatCount-1.
Schedule States - Criteria
Currently Processing Data: Is the schedule's AmiFullTableScanResult currently being worked on? We use the following fields to determine this: discoveredDataBeginDate, discoveredDataFinishDate, analyseDataBeginDate, analyseDataFinishDate. We also check the SoapMessageAuditEvent queue to determine if there is a recent Soap Message response relating to this schedule. This Soap check ignores responses outside of an hour timeframe (at this point, is it deemed out of date or erroneous due to a bad network connection/UIQ issue)
Important SOAP Discovery Configuration
With regards to SOAP message and remote discovery script configuration for temperature scanning and trending activity - In order to determine what messages and scripts are being used, you should start off by clicking into the relevant AmiFullTableScanSchedule entry via the HSIM SNMP Manager controllers view. From here, click into the associated soap message. From the associated soap message view, you will be able to see the XML body (Usually a JOB ID request), associated results request (another SOAP message), relevant associated SOAP discovery configurations and associated SOAP endpoints.
The framework is setup in a way that allows a message to be sent to a SSN UIQ web services endpoint to initiate request for a job status and request results for the associated data when the relevant job has completed. All associated scripts are in the SoapDiscoveryConfigurations associated with the SoapMessage objects themselves.
With regards to C12.19 data/temperature data on our deployment, the important AmiFullTableScanSchedules to check out are:
Known Limitations/Future Improvements
Trending events with breached thresholds continue to trend every <SyncInterval> (i.e. every hour) until the temperatures drop. This can be fixed easily but Steve decided on not addressing just yet. This has the added benefit of collecting a lot of data during a proof of concept phase. When a group of trenders complete their trending, they create once again when it's parent schedule rolls over and analyses it's data.