The SCO applications are hosted on AWS on N. Virginia region

Production Servers:

• http://mynocinabox.com/SnmpManager/errigalEMS
• Loadbalancer: scolb1.err
• Handler: scoapps1.err
• Master DB: scolb1.err
• Slave DB: scoapps1.err
• Load Balancer Public IP: 3.219.166.49

QA Servers:

• http://sco.errigal.com/SnmpManager/errigalEMS
• Loadbalancer: scoqalb1.err
• Handler: scoqaapps1.err
• Master DB: scoqalb1.err
• Slave DB: scoqaapps1.err
• Load Balancer Public IP: 3.228.195.12

Passwords available on PW Safe

Reporting Manager Considerations

• When creating a new report, the report and the variables need to check for the currently logged in user and then check its visibilities on ticketer before return the report results or variables available.
• Customer sensitive variables should always be marked as restricted so the user is not able to manually add a value that would display another user's data
• An availability calculation 'cache' similarly to the one on ATC is setup.

Snmp Manager Considerations

• SCO users should never been given admin role
• Noc User role has been modified to remove some permissions.
• Extra care needs to be taken to ensure that users don't have visibilities of another customer assigned
• Some of the SCO customers(E.g. NGEN) will have more than one controller sending traps from the same IP Address, this breaks our standard trap rule process, as it relies on the trap IP address to associate it to the controller. For some technologies, the traps have a field for the controller name, and that can be used. If that is not the case, the current solution is to ask the costumer to set the 'trap community string' on the controller to reference the controller name.
• E.g. Differently from other customers, Andrews TrapRule will look for the Controller Unique identifier in the name in trap of the controller

Ticketer Considerations

• When creating a new form, the groovlets to populate the form should check for the currently logged in user and then check its visibilities on ticketer before populating the results/select options available
• “is_multitenancy_deployment” Boolean in the Ticketer configuration domain should be marked as true, as otherwise an user can see other customer's users when changing ticket owners
• There is a simplified version of the Extenet SLA process ticket grouping.
• For the moment, there is only one worfklow used by all customers
• There is a custom notification for KLA Labs visibility, where ticket email notification is held-off till working hours
• SLA : It is a simplified grouping, it basically groups tickets of the same controller created in the last 45minutes https://mynocinabox.com/Ticketer/groovlet/show/35147

Deployment Considerations

• Ansible Env Configurations are available for QA and Prod
• Customers should be informed of deployments/downtime on production, but customer approval is not needed. Customer email distribution list to be created

• Add new cluster to snmp_manager database with name of customer (requires sub-steps below)
  • Add a new errigal_customer_company to ticketer database
  • Add a new region to ticketer database
  • Add a new network element to the database, with the customer name as its name, and “CUSTOMER-NETWORK” as its ne_type, you can search for others like these and duplicate
  • Add a new visibility to ticketer database (I recommend following steps in link) http://wiki.err/doku.php?id=onboarding:advanced:creating_a_new_cluster_visibility_in_all_applications - NOTE: This funcionality was removed in 4.17 - we will be adding it back in 4.21 hopefully.
  • Add a new errigal_customer_company to snmp_manager database
• Add new carrier to snmp_manager database with name of customer
• Go to userprofile, click Application Admin, click Snmp Manager, click “Sync SNMP Manager” in the top corner
• Again in userprofile, click Visiblities, click “Sync Visibilities”, select “cluster” from drop down
• Load a network element into this cluster via excel file loader, go to snmp manager homepage, the one with all the links and alphabetic filtering options and click excel file loader in the toolbar (ask a colleague for a template)
• Create visibility in userprofile with carrier and cluster created above selected (Setting a user with this visiblity will show them only their own devices within this cluster)
• Clear the cache for device trees in snmp manager, at the homepage, click “C” and navigate to “Cache”, then click the link under “Clear caches for carrier host tree”
• You may need to wait for a while for the elements to appear in the device tree, move to another task and check back later in the day.
• If the devices havent appeared after a few hours, try to navigate to the newly added device via URL https://mynocinabox.com/SnmpManager/errigalEMS#parent/<id-here>. This might wake it up (?)
• If they still are not in the list, try clearing your browser cache.
• Do a topology rediscovery on the device you added via loadsheet, be sure to check “on-air” for all the elements when you go to the topology diff screen and confirm changes.
• Add new devices if the customer has more than one by clicking on their cluster name on the tree and selecting Sync → Topology Discovery.
• Sync the network elements to nova by running the elastic replictor bulk import (be sure you have any extra details like map details populated beforehand) http://10.91.201.86:8100/elastic-replicator/bulk?routes=networkElement

In general, SCO users will use the IDMS similarly to the non-admin users of our standard customers. The main difference is that they will only see the information for their company's visibilities. Reports, widgets and forms available to them are already customised to filter the data. So the main task here is, once we have connectivity, is to load the elements to the customer cluster/visibility and create accounts with those visibilities assigned.

After the sale is confirmed and the contract is signed, we need to start monitoring the customer's devices as soon as possible.

With our standard customers, the IDMS is usually given access to their internal VPN. This will not be the case with SCO, so the customer need to give us access over the public internet.

Some of the SCO customers might not be as used to configuring networks in general, so even though at the end of the day it is their responsibility, they might need help on how to give us access/make the controllers reachable. They need to fill in the following sheet initial_controller_info_ngen.xlsx

First the customer should inform us of public IPs for each controller and port redirects to SNMP port 161 and http port, login credentials and public/private community. There is a template sheet to be filled in. They might either setup a modem for each of the controllers (which will be simpler for us, as each controller will have its own public IP address), or be already behind a firewall which will be configured to forward ports to the controllers, which may require further configuration if many controllers' traps are coming from the same IP address.

The customer should set-up our public load balancer IP, 3.219.166.49, as their devices' SNMP Trap destination (162 - default SNMP trap port).

Create an Errigal customer company (e.g. NGEN) in the ticketer at Ticketer/errigalCustomerCompany/, to be added to the visibility.

Create visibility for that customer on ticketer, the name of the visibility should be the same or at least include the customer name if the customer has many networks and would like/need to group them. E.g. “ECD” or “KLA - DAS” and “KLA - Nom-Das”. This http://wiki.err/doku.php?id=onboarding:advanced:creating_a_new_cluster_visibility_in_all_applications is a good guide to creating ticketer visibilities, although you may have to manually create the NOC portal cluster.

Go to NocPortal overlay management and associate that Cluster/Visibility with the Weather/Fire overlays.

Create a site for the customer at /SnmpManager/site/create e.g. UCH Memorial Central.

Add the customer name as a CUSTOMER-NETWORK ne_type to the network element table at /SnmpManager/networkElement/create e.g CONCISE

Add street address for network element via database.

Add map details latitude and longitude via /SnmpManager/mapDetails/create.

Add the customer's devices under that visibility, either by the topology discovery on EMS or the loadsheets

Associate the elements to the customer site, customer network and map details.

Add customer to company at /SnmpManager/company/create.

In the contact management controller in the SNMP manager, add contact(s) via the manager contact tab and assign it to the created network element via the assign contact tab.

Add carrier with customer name via /SnmpManager/carrier/create.

Export the loaded controllers loadsheets and send the simplified version (excel example equipment_details_-_ngen_-_sprint.xlsx ) for the customer to populate carrier and location details.

Create users for the customer via the user profile application.

• - User Can't be Admin in any application
• - User Can't have carrier host tree
• - User can't have global/admin visibilities
• - Reporting Manager SCO role
• - Unordered List Item - SNMP Manager NOC User Role
• - Set Noc Portal Default Cluster
• - Set Ticketer Default Ticket visibility

On the user profile, set up the new user to use the 'SCO - User' profile. This is a general one which accommodates all SCO users. The profile ensures that users have no admin rights in any applications and do not have access to the user profile. To sync the added ticketer visibilities, press the 'Sync Visibility Elements' under the visibility section. You may need to sync all applications under application admin for some added users roles to show up if necessary. The user business profile should be made up of application profiles specific to the customer, with the exception of the support page. Ensure the profile is set to be visible to SCO Customer and the correct customer is selected in the dropdown. Is it extemely important to not have other companies applications visible to the new customer and vice versa. For each applciation profile, ensure this is also the case.

Add a visibility profile for the customer. These profiles are generally the same except the cluster will be the specific one created above.

You may also need to add a new location for the customer to allow the EMS map to focus in on a different part of the world. Currently, this needs to be added through the database into the location table, where a name and coordinates are required. You can then assign it per user.

Create a test user with the above user profile and test extensively. Use a random password generator and store the details in pwSafe. When you're happy, add the users, which will send a password reset email to their given email address.

Create a Ticketer Home search for the user, with all open tickets and open tickets in alarm received

Create contacts for the emails the user want to be notified for ticket creation and closure. Assign that contact to the customer hubs.

Verify that traps are being processed and ticketing, and no device missing alarms are created. Verify if ticket email notification is being sent.

Verify if Layout is correct for different Levels (TODO: Query to associate a standard layout for a user)

Schedule training/ Demo (Sample Alarms and Tickets can be useful).

As past of 3.10.1 release, customer aware Global user admin will be implemented ( https://docs.google.com/document/d/13G3OAIPOm6XJx52RqQmnXuWhZO94Jb4uquiYJ2ZAyCA/edit?ts=5e2773e2#heading=h.lyzk2613nymk )

This will allow users with “SCO admin role” to access the global user admin tool but only see the users from its company. This will make use of a new “ErrigalCustomerCompany” domain, that will be associated to users and visibilities.

*Further technical details to be added here.*

Similar Process (TODO). Billing should be calculated automatically.

If loading using topology discovery, make sure that topology scripts are associating the correct ne_type for the elements.

Though for the first month after onboarding Sales will be the point of contact for any general/business queries, after that period SCO Customers will have the same support as the standard customers, with Waterford/San Francisco business hours cover for email and access to the critical notification for 24/7 support for critical issues.

They will raise requests via the Support Request tool which will raise a ticket on the Scotty Pro SCO queue for the given customer.

A dedicated customerName_support@errigal.com distro will be created to handle communication within the first 30 days to ensure nothing gets lost. However every effort will be used to drive the customer to the support request page.

Short videos will be created to demonstrate/teach the common tasks of the applications (TODO)