Internal Errigal Collaboration Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Fiber Project ======
TODO: Introduction, description, links to documents on drive, etc.

===== In a nutshell =====
There are following google drive documents, which describe the whole project in a detail: [[https://docs.google.com/document/d/1zO4FdZSOijXBcqedM8afvp3xKOgzIVuHEOi_UdQMREo/edit#heading=h.fykw06wp8law|Fiber - Requirements & Analysis]].
[[https://docs.google.com/document/d/1VdXMLhMuUXx4zM-ydEG0VzjO8uNG7V8m_93FIox0-lo/edit#heading=h.ltqknorm1tv3|Fiber Architecture/Design]]

Fiber is a Google Map Client which draws custom elements over the Map.
These custom elements along with their attributes and latitude-longitudes are loaded from a XML file provided by ExteNet GIS team.

===== Operations =====
==== Deployment ====
Fiber is deployed from [[http://10.91.100.112:8080/job/Fiber/|Jenkins]] - you have to pick one of the deploy jobs for the desired branch and choose the environment when running the build. You can then watch the build logs to see the deployment progress (should take approx. 2 minutes).

==== Application Service ====
Fiber is installed as an init.d service in linux:
<code>
[scotty@blade12(sfqaapps1) ~]$ls -la /etc/init.d/ | grep fiber
lrwxrwxrwx.  1 scotty root    32 Sep 28 01:47 fiber -> /var/spring-boot/fiber/fiber.war
</code>

This means there is no startup/shutdown script and it can be stopped/started/restarted using the standard service command:
<code>
[scotty@blade12(sfqaapps1) ~]$service fiber stop
Stopped [22697]
[scotty@blade12(sfqaapps1) ~]$service fiber start
Started [20705]
[scotty@blade12(sfqaapps1) ~]$service fiber restart
Stopped [20705]
Started [7755]
</code>

Please check the fiber.conf below to see how to set the JAVA_OPTS like heap etc.

==== Server folder structure ====
All the applications artefacts reside in the ///var/spring-boot/fiber/// folder.
<code>
[scotty@blade12(sfqaapps1) ~]$ls -la /var/spring-boot/fiber/
total 149700
drwxr-xr-x. 2 scotty root       4096 Dec 15 08:33 .
drwxr-xr-x. 4 scotty root       4096 Sep 28 01:45 ..
-rw-r--r--. 1 scotty root       3455 Dec  5 05:05 application.properties
-rw-r--r--. 1 scotty root        342 Oct 11 04:45 fiber.conf
-rw-r--r--. 1 scotty scotty      753 Dec 15 08:43 fiber-json.log
-rw-r--r--. 1 scotty scotty   168318 Dec  5 07:58 fiber-json.log.2017-12-05
-rw-r--r--. 1 scotty scotty    45773 Dec  6 07:33 fiber-json.log.2017-12-06
-rw-r--r--. 1 scotty scotty     1844 Dec  7 03:25 fiber-json.log.2017-12-07
-rw-r--r--. 1 scotty scotty     5480 Dec 11 09:20 fiber-json.log.2017-12-11
-rw-r--r--. 1 scotty scotty    21765 Dec 12 09:44 fiber-json.log.2017-12-12
-rw-r--r--. 1 scotty scotty     5308 Dec 13 06:38 fiber-json.log.2017-12-13
-rw-r--r--. 1 scotty scotty      552 Dec 14 09:20 fiber-json.log.2017-12-14
-rw-r--r--. 1 scotty scotty  4278844 Dec 15 08:43 fiber.log
-rw-r--r--. 1 scotty scotty 10651370 Dec  5 05:05 fiber.log.1
-rw-r--r--. 1 scotty scotty 11009850 Dec  5 05:05 fiber.log.2
-rw-r--r--. 1 scotty root   50237211 Dec  5 05:07 fiber.log.3
-rw-r--r--. 1 scotty root          6 Dec  5 07:46 fiber.pid
-rwxr--r--. 1 scotty root   76798493 Dec  5 07:46 fiber.war
-rw-r--r--. 1 scotty root        777 Dec  5 07:46 logback-spring.xml
</code>
=== fiber.conf ===
The fiber.conf contains the basic java options for the application (location of the java, JAVA_OPTS, heap, etc.). Please check [[https://docs.spring.io/spring-boot/docs/current/reference/html/deployment-install.html|Spring Documentation]] for more details on these options.

=== application.properties ===
This is equivalent to our Groovy.configs - all the DB connection string, URLs and other settings are there.

The application needs to be restarted in order to any changes to this config take effect.

=== Adding new user to Fiber ===
Fiber is not integrated with Global Admin tool and there is no UI for creating the users (e.g. in case there is a new NOC team member). Following SQL script can be used in order to create the user manually (please modify the username to the desired one). It assumes that the same username is already in the snmp_manager (so that the SSO will check password against it and we don't have to specify the user's password in the fiber db, but the password value cannot be null, otherwise it can cause issues with CAS authentication for that user in ALL applications).
<code>
use fiber;
insert into fiber.user(id, username,password) values((select next_val from fiber.user_seq), 'another_user1','AnyHashedPasswordValue');
update fiber.user_seq set next_val = next_val + 1 LIMIT 1;
commit;
</code>

=== Adding new state to fiber ===
Since version 3.6 Fiber states are stored in the fiber.state table along with all the geometries (center, bounds and polygon). Inserting the geometries manually is tricky so there is a utility to support that.
First you have to create the state record (please change the abbreviation and name for something real):
<code>
use fiber;
insert into fiber.state(id, abbreviation, name) values((select next_val from fiber.state_seq), 'CZ','Czech Republic');
update fiber.state_seq set next_val = next_val + 1 LIMIT 1;
commit;
</code>
Then you have to populate the geometries, there is a Fiber REST Endpoint to support this (which utilises OpenStreetMap API to get the actual polygon).
You have to login to a fiber in browser (e.g. sfqa) and then use following URL in the browser (please change the params per your state):

http://localhost:8080/fiber/api/0.1/codes/stateGeometryUpdateQuery?abbreviation=**CZ**&query=**Czech%20Republic**&tolerance=**0.05**

The query is a fulltext query to OpenStreetMaps and the results are update statements for whatever is returned. It is possible that results will contain multiple records, just like this:

Česko: **UPDATE state SET bounds = GeomFromText('POLYGON ( ( 48.5518083 12.0905901, 48.5518083 18.8590853, 51.0200296 18.8590853, 51.0200296 12.0905901, 48.5518083 12.0905901))'), center = GeomFromText('POINT (49.785918949999996 15.4748377)'), polygon = GeomFromText('MULTIPOLYGON ( ( ( 50.2523986 12.0905901, 48.5518083 14.3332353, 49.5476933 18.8590853, 51.0200296 15.1718505, 50.2523986 12.0905901)))') where abbreviation = 'CZ';** Residence of the Ambassador of the Czech Republic, Ayisi Close, Airport Residential Area, Accra, Accra Metropolitan, Greater Accra Region, 2335, Ghana: UPDATE state SET bounds = GeomFromText('POINT (5.6079968 -0.1777092)'), center = GeomFromText('POINT (5.6079968 -0.1777092)'), polygon = GeomFromText('POINT (5.6079968 -0.1777092)') where abbreviation = 'CZ'; Embaixada da República Tcheca, 2, Kanda Highway, Nima, Accra, Accra Metropolitan, Greater Accra Region, NE2 8AH, Ghana: UPDATE state SET bounds = GeomFromText('POLYGON ( ( 5.5757393 -0.1981883, 5.5757393 -0.1979219, 5.5760376 -0.1979219, 5.5760376 -0.1981883, 5.5757393 -0.1981883))'), center = GeomFromText('POINT (5.575888450000001 -0.1980551)'), polygon = GeomFromText('MULTIPOLYGON ( ( ( 5.5758273 -0.1981883, 5.5757393 -0.1980224, 5.5759269 -0.1979219, 5.5760376 -0.1981306, 5.5758273 -0.1981883)))') where abbreviation = 'CZ';

You have to manually pick the update sql part just as shown above and the run the query in the db as well (which will update the geometries).
After executing this, go to fiber and verify the state is in the dropdown and that selecting it will bring you to the correct place on the map.

=== Loading new GIS data snapshot ===
The GIS data is provided by ExteNet (usually in [[https://extenetsystems-my.sharepoint.com/:f:/r/personal/klaug_extenetsystems_com/Documents/Documents/GIS_2_NOC/XML%20FILE?csf=1&e=KvR9PJ|GIS_2_NOC]] OneDrive Folder). The file should be a zip file of roughly 50-100 MB. The file can be first loaded to ExteNet QA and then to ExteNet Production.

The GIS file can be uploaded to Fiber (Fiber -> Menu -> Import Data). The UI should show no errors and the numbers should keep moving. The whole process might take up to few hours (we are importing millions of records).

Once the file is uploaded try checking the Fiber app (choosing one of the States, like IL, and verifying the data loads).

In case there is an error during the loading (or the loading gets stuck), check the fiber logs for more details.

== Analyse missing Network ID mappings ==
You will need to download the logs for the particular upload - this is in Fiber -> Menu -> Import Data -> Logs (locate the latest record and hit 'Download Import Log'). 
Then run the following script to extract the missing Network ID mappings (put in the right log filename):
<code>
cat <the-fiber-import-log> | grep 'Error: No Project ID to NOC Network ID mapping found for Project ID: ' | cut -b70-500 | sort | uniq
</code>

The resulting data can be put into excel spreadsheet and shared with ExteNet.
The used script is also stored in fiber_component repo (fiber_component/src/main/resources/usefull-scripts/import-log-analysis.sh)

=== Changing the mail list for e-mail notifications ===
Fiber 3.6 has introduced automated imports of GIS data. This process ends with sending an e-mail notification to share outcomes of the import.
The list of recipients of this notification can be changed in //application.properties//, property //fiber.mail.recipients//. This property supports multiple emails delimited by comma.


===== Methodology =====
==== Tools ====
Following table describes specific tools used on Fiber project. It doesn't cover the standard Errigal tooling (IDE, BitBucket, Gradle, etc.).

| ^ Purpose ^ Additional Information ^
^ SonarLint | Static code analysis tool | IDEA Plugin capable of performing code quality checks. It can be integrated to existing SonarQube server to delegate the analysis to it. |
^ SonarQube | Static code analysis server | More info at [[https://www.sonarqube.org/|sonarqube.org]]. Errigal installation runs at [[http://10.91.100.110:9000/sonar/|http://10.91.100.110:9000/sonar/]] (the login is as usual :)). |
^ Jenkins 2 | Build Server | More info at [[https://jenkins.io/|jenkins.io]]. Errigal installation runs at [[http://10.91.100.112:8080/job/Fiber/|http://10.91.100.112:8080/job/Fiber/]] (usual credentials as well) |
^ Postman | REST Client | More info at [[https://www.getpostman.com/|getpostman.com]]. Fiber REST Requests are available in Postman collection: [[https://www.getpostman.com/collections/d2f32d32222d056b5a7a|https://www.getpostman.com/collections/d2f32d32222d056b5a7a]] (can be imported into Postman).|

==== Before every Push ====
  - Make sure the commits are cohesive and with good commit message
  - Run SonarLint to reveal any code quality issues in your commits and try to fix them
  - Run/Fix Unit Tests
  - Run/Fix Integration Tests

==== After Push ====
  - Try to check the Jenkins pipeline if it ran well (Jenkins will be configured to send e-mail notifications soon)

==== Project Structure ====
Following figure depicts the main concepts of the application structure (rules are mentioned below).
{{:development:applications:screen_shot_2017-04-19_at_11.57.45.png?1280|}}

Every developer should follow these rules when creating new classes in the project:
  * The base package for the whole project is //com.errigal.fiber//
  * Each application modules (//entity//, //spatial//, //pm//) has it's own package (we are not going to utilise Gradle modules at this moment) with following structure:
    * //api// - definition of interfaces
    * //api.model// -definition of data objects used in interfaces
    * //rest// - REST Controllers (TODO: add link to section with description how to implement them)
    * //rest.request// - Request objects of methods in REST controllers
    * //rest.response// - Response objects of methods in REST controllers
    * //<root-of-the-module-package>// - implementations of module interfaces

==== Use of Spring ====
The classes in //src// folder of the project make use of Spring. This replaces the use of Grails Holders, but practically does the same thing. The main reason to do that is to let all the Spring Controllers, Tests and others to work well.

=== IoC - Instantiation and wiring of beans ===
It is encourage to use Spring annotations to allow Spring to handle class instantiation of any service defined by us and it's injection to other classes. The Spring annotation for bean injection can be used in following way:
{{:development:applications:screen_shot_2017-04-19_at_12.14.38.png?1280|}}

There is the //StateService// interface, which is implemented by the //InMemoryStatesService//. The implementation needs to be annotated with **@Component** annotation in order to be injected somewhere else. The mentioned //StateService// is needed in //CodeTablesRestController// and it is injected there using the **Autowired** annotation. At that point Spring will look if there is any class annotated with //@Component//, which implements the //StateService// interface. If it finds it, it will inject the instance of that class to the field of the Controller. It is good practice to //@Autowire// interfaces instead of particular classes.

=== Injecting values from configuration ===

Example: There is configuration property //frontendConfigurationFolder// in //Config.groovy//. In order to inject the value of that property to Spring beans, we have to use **@Value** annotation as depicted on following figure:

{{:development:applications:screen_shot_2017-04-19_at_12.31.05.png?1280|}}

==== Implementing Controllers ====
Use Spring Controllers and take inspiration from existing Controllers.

==== Implementing Unit Tests ====


==== Implementing Integration Tests ====

==== Logging ====
  * Avoid using println and use //@Slf4j// where possible
  * Use the log levels appropriately
  * It is good to make the values in logs easy to parse (e.g. RequestIds, request params, etc.)

==== Running Integration Tests ====

==== Security related problems ====
If you face any security related exceptions while running the project, then you are most likely missing the required certificate.

<code>
/PATH_TO_JAVA/bin/keytool -importcert -file ~/<PATH_TO_CERT_FILE>/<qalb1>.crt -keystore 
/PATH_TO_JAVA/jre/lib/security/cacerts -alias <qalb1>
</code>



===== Improvement Candidates =====
A place to note all the possible improvements that we decide not to implement, but could be done in future.
We might also put them as a Backlog to JIRA.
Sidebar

Internal Errigal Collaboration Wiki

User Tools

Site Tools

Page Tools
