Internal Errigal Collaboration Wiki

User Tools

Site Tools

Trace: • organising_your_bookmarks_and_common_links
Writing /app/www/public/data/meta/development/applications/snmpmanager/hsim.meta failed
development:applications:snmpmanager:hsim

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
development:applications:snmpmanager:hsim [2017/05/24 22:30] – dmcgeedevelopment:applications:snmpmanager:hsim [2021/06/25 10:09] (current) – external edit 127.0.0.1
Line 1: Line 1:
 +====== HSIM - Temperature Scanning and Trending Overview ======
 +
 +David McGee 
 +
 +Weds, 24th May 2017
 +
 +This article assumes prior knowledge of AMI (Advanced Metering Infrastructure) Networks, ANSI C12.19 standards as well as an indepth knowledge of Silver Spring Networks (SSN) UIQ platform. This article has purposely not referenced customer details and/or cloud deployment for security purposes. It has, however, been written for a specific installation/deployment. 
 +
 +** Important Entities **
 +
 +There are a number of important classes/entities with regards to the HSIM temperature scanning and trending functionality. This list is not exhaustive, but should give you an idea of how the process works.
 +
 +  * AnalyseAmiFullTableScanResultsJob - Runs every 60 seconds to determine the status of all active AmiFullTableScanSchedules. This also triggers awaiting SoapMessageAuditEvent trending events (with a date in the past)
 +
 +  * AmiFullTableScanSchedule - A representation of an AMI table that needs to be scanned/processed/analysed on a given interval. This class does the necessary work for checking if a table result needs to be analysed, if its already being processed, if a new SoapMessageAuditEvent needs to be created on the queue to pull new results, or if the schedule has already been analysed for the given time interval. See performFullTableScanProcessingWork() method in this class. Configurable properties allow the data type, parameter name, sync interval, job id, threshold collection, associated soap message, trending config, active alarm identifier and trap oid to be set. This is an important entity that allows communication between Errigal HSIM and SSN (Silver Springs Networks) UIQ platform via SOAP web services. 
 +
 +  * AmiFullTableScanTrendingConfig - This allows the configuration of a number of trending options that can be changed at run-time. Trending configs are associated with AmiFullTableScanSchedules (object referenced above). This allows us to specify trending interval (SyncInterval), trending repeat count (i.e. 24), associated trending soap messages, whether we should continue/discontinue trending after an alarm clears, and also allows us to mark it active=true/false. 
 +
 +  * AmiFullTableScanResult - Everytime a schedule runs, it creates an AmiFullTableScanResult. This allows the application to become aware that something is currently persisting DiscoveredData for a schedule, or some part of the application is currently analysing this data. Due to the highly concurrent nature of requesting information (as well as the high volume of data) from AMI software such as SSN UIQ, it is important to be aware of what is currently being processed/analysed. For this reason, important date/time fields were added to this table that are automatically timestamped at the start of discovering data/the end of discovering data, as well as at the start of analysing data/the end of analysing data. Fields are as follows:  discoveredDataBeginDate, discoveredDataFinishDate, analyseDataBeginDate, analyseDataFinishDate. 
 +
 +  * ThresholdCollection (AdvancedThreshold) - This allows a collection of AdvancedThresholds to be defined. This comes into play when we are analysing a set of DiscoveredData rows for breached thresholds. An example is analysing C12.19 temperature readings from SSN UIQ. Our client can specify custom temperature thresholds at run-time. ThresholdCollections can be associated with AmiFullTableScanSchedules.
 +
 +  * SoapMessage - A customisable message that can be associated with an AmiFullTableScanSchedule. Important fields include: name, xmlBody, messageType, soapEndpoint, asynchronous, jobIdRequest, resultsRequest. 
 +
 +  * SoapMessageAuditEvent - An event that is used to track a request and/or response from Errigal HSIM to SSN UIQ via SOAP Web Services. This forms a SOAP Web Services request queue which is central to schedule scanning/trending and on-demand activity within HSIM. You should check out the SoapMessageResponseStatus domain and you may need to study the response types if debugging an issue. 
 +
 +  * DiscoveredData - This is a name/value store where we can persist many different types of data, identified by dataType and parameter (i.e. 'FocusMeterData' and 'Temperature')
 +
 +  * ErrigalAmiNetworkSnmpTrap - An SNMP trap that can be associated with a schedule. Provided that there is also a ThresholdCollection associated with the schedule, this trap can throw an AdvancedThreshold is breached. A custom alarming OID can be associated with the schedule for when there are new notification/trap types added to the Errigal MIB. 
 +
 +
 +
 +**Walkthrough**
 +
 +As of writing (24th May, 2017) - Here is how the HSIM system is currently scanning and trending temperature data:
 +
 +  * It is important to note that the SoapMessageAuditEvent queue is the backbone to scanning and trending temperature data functionality working. A number of SoapMessages are configured on our AWS cloud deployment with associated Remote Discovery Framework (RDF) scripts to prepare XML requests using parameters, as well as parse responses from SSN UIQ. Please see the important SOAP discovery configuration section below for further info. 
 +
 +  * Every 60 seconds, the AnalyseAmiFullTableScanResultsJob runs to check all active schedule's status.
 +
 +  * A number of statuses can be found based on method returns (isInSync(), isCurrentlyProcessingData(), hasResultReadyForAnalysis()). If there is no schedule data (it is unsynced) and no result ready to process, a pending event is created on the queue (SoapMessageAuditEvent) for the schedule in question. See Schedule States below for further detail on how the code works out the current state of a schedule. 
 +
 +  * If a schedule needs to be processed (it has response data ready to be processed from a SOAP Message web services request), the analyse() method on AmiFullTableScanResult will be called. There is a number of aims here: Find all relevant schedule data (using schedule.discoveredDataParameterName - i.e. 'Temperature'), check the data for threshold breaches, send SNMP traps to create active alarms/clear active alarms where necessary, and create/stop trending events where necessary. 
 +
 +  * For each schedule with an active trending config, any trending events (SoapMessageAuditEvent) that are due to trigger will execute (scheduledRepeatCount >0, scheduledTriggerDate < now)
 +
 +  * For each trending event in a schedule, if the current scheduledRepeatCount > 0, a follow-up trending event is scheduled for the future (based on a sync interval defined in the associated TrendingConfig). This is 1 hour by default. The scheduledRepeatCount on the initial audit event is set to 0, and a new event is created with scheduledRepeatCount-1. 
 +
 +
 +
 +**Schedule States - Criteria**
 +
 +  * Is In Sync: Can we find a recorded processed AmiFullTableScanResult for this schedule within the expected sync interval? (i.e. 'FocusTemperatureScanResult' created 23 hours ago on a 24 hour sync interval means the schedule is in sync! If it is older than 24 hours, it is not in sync)
 +
 +  * Currently Processing Data: Is the schedule's AmiFullTableScanResult currently being worked on? We use the following fields to determine this: discoveredDataBeginDate, discoveredDataFinishDate, analyseDataBeginDate, analyseDataFinishDate. We also check the SoapMessageAuditEvent queue to determine if there is a recent Soap Message response relating to this schedule. This Soap check ignores responses outside of an hour timeframe (at this point, is it deemed out of date or erroneous due to a bad network connection/UIQ issue)
 +
 +  * Has Result Ready for Analysis: Can we find a recorded AmiFullTableScanResult for this schedule that has a discoveredDataFinishDate within the schedule's sync interval?
 +
 +
 +
 +**Important SOAP Discovery Configuration**
 +
 +With regards to SOAP message and remote discovery script configuration for temperature scanning and trending activity - In order to determine what messages and scripts are being used, you should start off by clicking into the relevant AmiFullTableScanSchedule entry via the HSIM SNMP Manager controllers view. From here, click into the associated soap message. From the associated soap message view, you will be able to see the XML body (Usually a JOB ID request), associated results request (another SOAP message), relevant associated SOAP discovery configurations and associated SOAP endpoints. 
 +
 +The framework is setup in a way that allows a message to be sent to a SSN UIQ web services endpoint to initiate request for a job status and request results for the associated data when the relevant job has completed. All associated scripts are in the SoapDiscoveryConfigurations associated with the SoapMessage objects themselves. 
 +
 +With regards to C12.19 data/temperature data on our deployment, the important AmiFullTableScanSchedules to check out are:  
 +
 +  * getArbitaryMeterCommandResultsByJobId_Focus
 +
 +  * getArbitaryMeterCommandResultsByJobId_GEi210
 +
 +  * getArbitaryMeterCommandResultsByJobId_GEkV2c
 +
 +
 +
 +**Known Limitations/Future Improvements**
 +  * Trending events with breached thresholds continue to trend every <SyncInterval> (i.e. every hour) until the temperatures drop. This can be fixed easily but Steve decided on not addressing just yet. This has the added benefit of collecting a lot of data during a proof of concept phase. When a group of trenders complete their trending, they create once again when it's parent schedule rolls over and analyses it's data. 
  

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki