Differences

This shows you the differences between two versions of the page.

--- onboarding:elastic-replicator:start [2022/10/04 11:47] – 10.91.110.100
+++ onboarding:elastic-replicator:start [2024/08/29 11:28] (current) – 10.91.129.119
@@ Line 13: / Line 13: @@
 Repo: https://bitbucket.org/errigal/elastic-replicator/branch/master
+Project setup README: https://bitbucket.org/errigal/elastic-replicator/src/master/README.md
 ===== Local Dev App Urls =====
@@ Line 22: / Line 24: @@
   * https://drive.google.com/drive/u/0/folders/1HtHWiQMPSpIhOBnmPDFtKRPipdIHVtlb
-===== Apis / Data Accessed =====
+===== Data Replication into ElasticSearch =====
+Data is replicated via sql queries which can be customised according to the customer or use case. These queries are stored in /var/springboot/elastic-replicator/queries.
+The selected fields used in the queries use a unified naming convention in order to provide consistency for the elastic search record indexes.
+The queries are currently stored for deployment in the deployment-playbooks project which has subfolder structure allowing for customer specific query overrides:
+{{ :onboarding:elastic-replicator:screenshot_2022-10-04_at_12.46.59.png?600 |}}
+===== Scheduled Imports =====
+The bulk import is configured with quartz to run on a schedule (see BulkRouter.java for logic details). This refreshes the data by re-running the queries for all datatypes and updating the existing elastic search indexes. The timing of the cron is configured in the application properties as follows:
 <code>
+bulk.import.quartz.schedule=0 0 1 1/1 * ? *
+</code>
+{{ :onboarding:elastic-replicator:screenshot_2022-10-04_at_14.22.22.png?600 |}}
+===== On Demand Imports =====
+The data can be imported either via individual or bulk mechanisms as follows:
+**Individual:**
+  - GET http://10.91.140.218:8100/elastic-replicator/bulk?routes=trap
+  - GET http://10.91.140.218:8100/elastic-replicator/bulk?routes=networkElement
+  - GET http://10.91.140.218:8100/elastic-replicator/bulk?routes=ticket
+  - GET http://10.91.140.218:8100/elastic-replicator/bulk?routes=activeAlarm
+  - GET http://10.91.140.218:8100/elastic-replicator/bulk?routes=uptime
+**Limited individual**
+  - GET "http://10.91.140.218:8100/bulk?routes=uptime&start=1"
+to limit to 1 month back.
+**Bulk:**
+  -  http://10.91.140.218:8100/bulk
+**Playbook:**
+  -  initialise-nova-es.yml (deployment-playbooks)
+**Classes to note for import process:**
+<code>
+BulkRouter.java - handles bulk import for records
+SyncRouter.java - handles real time updates for records
 </code>
@@ Line 33: / Line 73: @@
 ===== Architecture =====
 {{ :onboarding:elastic-replicator:ems_nova_tech._design-architecture.drawio.png?600 |}}
+===== SSE(Server Sent Events Process) =====
+The SSE subscription drives real time update receipt in the angular ems-nova front end application.
+Server-Sent Events (SSE) is a server push technology enabling a client to receive automatic updates from a server via an HTTP connection, and describes how servers can initiate data transmission towards clients once an initial client connection has been established.
+In this case nova subscribes to the idms-proxy sse endpoint and listens for updates:
+<code>
+/idms-proxy/api/v0/replicator
+</code>
+The SSE touchpoints for  processing are as follows(see also diagram below):
+  - Gorm listener events for ticket, alarm, element inserts or updates are pushed from Ticketer and SnmpManager onto rabbitmq.
+  - Elastic Replicator consumes messages from these queues, processes them using the sql queries and updates or inserts new records into elastic search. It also pushes the events onto a queue which is consumed by idms-proxy.
+  - idms-proxy consumes a queue which receives event messages from elastic-replicator.
+  - Nova front end is subscribed to proxies sse endpoint /replicator and receives messages for real time page updates.
+{{ :onboarding:elastic-replicator:eventnotificationprocess.drawio.png?600 |}}
+  * Analysis document: https://docs.google.com/document/d/1nUBZtjKfTnopua9quoIhOpeugYVlCYHB8udhRcXwbS4/edit#heading=h.e1lpa8mdes4i
+  * Requirements document: https://docs.google.com/document/d/14mxFTS-B7BaPDNNERJVtzFEZlsCRDIQTFLbgOLaVQxo/edit#heading=h.euwbsr2vf8os
 ===== Development Configuration =====
-Project can be pointed at local or server deployed version of all api endpoints listed above.
+Project can be pointed at local or server deployed version of all /bulk api endpoints listed above in the import section.
 See src/main/resources/application.properties for environment configuration for local dev.
@@ Line 114: / Line 175: @@
 </code>
-===== Data Replication into Elastic Search =====
-=== Scheduled Imports ===
-=== On Demand Imports ===
-===== SSE =====
-Analysis document: https://docs.google.com/document/d/1nUBZtjKfTnopua9quoIhOpeugYVlCYHB8udhRcXwbS4/edit#heading=h.e1lpa8mdes4i
 === Jenkins Build ===
 https://jenkins.errigal.com:8443/job/elastic-replicator/

Sidebar

Internal Errigal Collaboration Wiki

Differences

Sidebar

Internal Errigal Collaboration Wiki

User Tools

Site Tools

Differences

Page Tools