Fiber Project

TODO: Introduction, description, links to documents on drive, etc.

There are following google drive documents, which describe the whole project in a detail: Fiber - Requirements & Analysis. Fiber Architecture/Design

Fiber is a Google Map Client which draws custom elements over the Map. These custom elements along with their attributes and latitude-longitudes are loaded from a XML file provided by ExteNet GIS team.

Fiber is deployed from Jenkins - you have to pick one of the deploy jobs for the desired branch and choose the environment when running the build. You can then watch the build logs to see the deployment progress (should take approx. 2 minutes).

Fiber is installed as an init.d service in linux:

[scotty@blade12(sfqaapps1) ~]$ls -la /etc/init.d/ | grep fiber
lrwxrwxrwx.  1 scotty root    32 Sep 28 01:47 fiber -> /var/spring-boot/fiber/fiber.war

This means there is no startup/shutdown script and it can be stopped/started/restarted using the standard service command:

[scotty@blade12(sfqaapps1) ~]$service fiber stop
Stopped [22697]
[scotty@blade12(sfqaapps1) ~]$service fiber start
Started [20705]
[scotty@blade12(sfqaapps1) ~]$service fiber restart
Stopped [20705]
Started [7755]

Please check the fiber.conf below to see how to set the JAVA_OPTS like heap etc.

All the applications artefacts reside in the /var/spring-boot/fiber/ folder.

[scotty@blade12(sfqaapps1) ~]$ls -la /var/spring-boot/fiber/
total 149700
drwxr-xr-x. 2 scotty root       4096 Dec 15 08:33 .
drwxr-xr-x. 4 scotty root       4096 Sep 28 01:45 ..
-rw-r--r--. 1 scotty root       3455 Dec  5 05:05 application.properties
-rw-r--r--. 1 scotty root        342 Oct 11 04:45 fiber.conf
-rw-r--r--. 1 scotty scotty      753 Dec 15 08:43 fiber-json.log
-rw-r--r--. 1 scotty scotty   168318 Dec  5 07:58 fiber-json.log.2017-12-05
-rw-r--r--. 1 scotty scotty    45773 Dec  6 07:33 fiber-json.log.2017-12-06
-rw-r--r--. 1 scotty scotty     1844 Dec  7 03:25 fiber-json.log.2017-12-07
-rw-r--r--. 1 scotty scotty     5480 Dec 11 09:20 fiber-json.log.2017-12-11
-rw-r--r--. 1 scotty scotty    21765 Dec 12 09:44 fiber-json.log.2017-12-12
-rw-r--r--. 1 scotty scotty     5308 Dec 13 06:38 fiber-json.log.2017-12-13
-rw-r--r--. 1 scotty scotty      552 Dec 14 09:20 fiber-json.log.2017-12-14
-rw-r--r--. 1 scotty scotty  4278844 Dec 15 08:43 fiber.log
-rw-r--r--. 1 scotty scotty 10651370 Dec  5 05:05 fiber.log.1
-rw-r--r--. 1 scotty scotty 11009850 Dec  5 05:05 fiber.log.2
-rw-r--r--. 1 scotty root   50237211 Dec  5 05:07 fiber.log.3
-rw-r--r--. 1 scotty root          6 Dec  5 07:46 fiber.pid
-rwxr--r--. 1 scotty root   76798493 Dec  5 07:46 fiber.war
-rw-r--r--. 1 scotty root        777 Dec  5 07:46 logback-spring.xml

The fiber.conf contains the basic java options for the application (location of the java, JAVA_OPTS, heap, etc.). Please check Spring Documentation for more details on these options.

This is equivalent to our Groovy.configs - all the DB connection string, URLs and other settings are there.

The application needs to be restarted in order to any changes to this config take effect.

Fiber is not integrated with Global Admin tool and there is no UI for creating the users (e.g. in case there is a new NOC team member). Following SQL script can be used in order to create the user manually (please modify the username to the desired one). It assumes that the same username is already in the snmp_manager (so that the SSO will check password against it and we don't have to specify the user's password in the fiber db, but the password value cannot be null, otherwise it can cause issues with CAS authentication for that user in ALL applications).

use fiber;
insert into fiber.user(id, username,password) values((select next_val from fiber.user_seq), 'another_user1','AnyHashedPasswordValue');
update fiber.user_seq set next_val = next_val + 1 LIMIT 1;
commit;

Since version 3.6 Fiber states are stored in the fiber.state table along with all the geometries (center, bounds and polygon). Inserting the geometries manually is tricky so there is a utility to support that. First you have to create the state record (please change the abbreviation and name for something real):

use fiber;
insert into fiber.state(id, abbreviation, name) values((select next_val from fiber.state_seq), 'CZ','Czech Republic');
update fiber.state_seq set next_val = next_val + 1 LIMIT 1;
commit;

Then you have to populate the geometries, there is a Fiber REST Endpoint to support this (which utilises OpenStreetMap API to get the actual polygon). You have to login to a fiber in browser (e.g. sfqa) and then use following URL in the browser (please change the params per your state):

http://localhost:8080/fiber/api/0.1/codes/stateGeometryUpdateQuery?abbreviation=CZ&query=Czech%20Republic&tolerance=0.05

The query is a fulltext query to OpenStreetMaps and the results are update statements for whatever is returned. It is possible that results will contain multiple records, just like this:

Česko: UPDATE state SET bounds = GeomFromText('POLYGON ( ( 48.5518083 12.0905901, 48.5518083 18.8590853, 51.0200296 18.8590853, 51.0200296 12.0905901, 48.5518083 12.0905901))'), center = GeomFromText('POINT (49.785918949999996 15.4748377)'), polygon = GeomFromText('MULTIPOLYGON ( ( ( 50.2523986 12.0905901, 48.5518083 14.3332353, 49.5476933 18.8590853, 51.0200296 15.1718505, 50.2523986 12.0905901)))') where abbreviation = 'CZ'; Residence of the Ambassador of the Czech Republic, Ayisi Close, Airport Residential Area, Accra, Accra Metropolitan, Greater Accra Region, 2335, Ghana: UPDATE state SET bounds = GeomFromText('POINT (5.6079968 -0.1777092)'), center = GeomFromText('POINT (5.6079968 -0.1777092)'), polygon = GeomFromText('POINT (5.6079968 -0.1777092)') where abbreviation = 'CZ'; Embaixada da República Tcheca, 2, Kanda Highway, Nima, Accra, Accra Metropolitan, Greater Accra Region, NE2 8AH, Ghana: UPDATE state SET bounds = GeomFromText('POLYGON ( ( 5.5757393 -0.1981883, 5.5757393 -0.1979219, 5.5760376 -0.1979219, 5.5760376 -0.1981883, 5.5757393 -0.1981883))'), center = GeomFromText('POINT (5.575888450000001 -0.1980551)'), polygon = GeomFromText('MULTIPOLYGON ( ( ( 5.5758273 -0.1981883, 5.5757393 -0.1980224, 5.5759269 -0.1979219, 5.5760376 -0.1981306, 5.5758273 -0.1981883)))') where abbreviation = 'CZ';

You have to manually pick the update sql part just as shown above and the run the query in the db as well (which will update the geometries). After executing this, go to fiber and verify the state is in the dropdown and that selecting it will bring you to the correct place on the map.

The GIS data is provided by ExteNet (usually in GIS_2_NOC OneDrive Folder). The file should be a zip file of roughly 50-100 MB. The file can be first loaded to ExteNet QA and then to ExteNet Production.

The GIS file can be uploaded to Fiber (Fiber → Menu → Import Data). The UI should show no errors and the numbers should keep moving. The whole process might take up to few hours (we are importing millions of records).

Once the file is uploaded try checking the Fiber app (choosing one of the States, like IL, and verifying the data loads).

In case there is an error during the loading (or the loading gets stuck), check the fiber logs for more details.

You will need to download the logs for the particular upload - this is in Fiber → Menu → Import Data → Logs (locate the latest record and hit 'Download Import Log'). Then run the following script to extract the missing Network ID mappings (put in the right log filename):

cat <the-fiber-import-log> | grep 'Error: No Project ID to NOC Network ID mapping found for Project ID: ' | cut -b70-500 | sort | uniq

The resulting data can be put into excel spreadsheet and shared with ExteNet. The used script is also stored in fiber_component repo (fiber_component/src/main/resources/usefull-scripts/import-log-analysis.sh)

Fiber 3.6 has introduced automated imports of GIS data. This process ends with sending an e-mail notification to share outcomes of the import. The list of recipients of this notification can be changed in application.properties, property fiber.mail.recipients. This property supports multiple emails delimited by comma.

Following table describes specific tools used on Fiber project. It doesn't cover the standard Errigal tooling (IDE, BitBucket, Gradle, etc.).

1. Make sure the commits are cohesive and with good commit message
2. Run SonarLint to reveal any code quality issues in your commits and try to fix them
3. Run/Fix Unit Tests
4. Run/Fix Integration Tests

1. Try to check the Jenkins pipeline if it ran well (Jenkins will be configured to send e-mail notifications soon)

Following figure depicts the main concepts of the application structure (rules are mentioned below).

Every developer should follow these rules when creating new classes in the project:

• The base package for the whole project is com.errigal.fiber
• Each application modules (entity, spatial, pm) has it's own package (we are not going to utilise Gradle modules at this moment) with following structure:
  • api - definition of interfaces
  • api.model -definition of data objects used in interfaces
  • rest - REST Controllers (TODO: add link to section with description how to implement them)
  • rest.request - Request objects of methods in REST controllers
  • rest.response - Response objects of methods in REST controllers
  • <root-of-the-module-package> - implementations of module interfaces

The classes in src folder of the project make use of Spring. This replaces the use of Grails Holders, but practically does the same thing. The main reason to do that is to let all the Spring Controllers, Tests and others to work well.

It is encourage to use Spring annotations to allow Spring to handle class instantiation of any service defined by us and it's injection to other classes. The Spring annotation for bean injection can be used in following way:

There is the StateService interface, which is implemented by the InMemoryStatesService. The implementation needs to be annotated with @Component annotation in order to be injected somewhere else. The mentioned StateService is needed in CodeTablesRestController and it is injected there using the Autowired annotation. At that point Spring will look if there is any class annotated with @Component, which implements the StateService interface. If it finds it, it will inject the instance of that class to the field of the Controller. It is good practice to @Autowire interfaces instead of particular classes.

Example: There is configuration property frontendConfigurationFolder in Config.groovy. In order to inject the value of that property to Spring beans, we have to use @Value annotation as depicted on following figure:

Use Spring Controllers and take inspiration from existing Controllers.

• Avoid using println and use @Slf4j where possible
• Use the log levels appropriately
• It is good to make the values in logs easy to parse (e.g. RequestIds, request params, etc.)

If you face any security related exceptions while running the project, then you are most likely missing the required certificate.

/PATH_TO_JAVA/bin/keytool -importcert -file ~/<PATH_TO_CERT_FILE>/<qalb1>.crt -keystore 
/PATH_TO_JAVA/jre/lib/security/cacerts -alias <qalb1>

A place to note all the possible improvements that we decide not to implement, but could be done in future. We might also put them as a Backlog to JIRA.