Internal Errigal Collaboration Wiki

User Tools

Site Tools

Trace: • res-p1115 • elastic-replicator
onboarding:elastic-replicator:start

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
onboarding:elastic-replicator:start [2022/10/04 12:58] – 10.91.110.100onboarding:elastic-replicator:start [2024/08/29 11:28] (current) – 10.91.129.119
Line 24: Line 24:
   * https://drive.google.com/drive/u/0/folders/1HtHWiQMPSpIhOBnmPDFtKRPipdIHVtlb   * https://drive.google.com/drive/u/0/folders/1HtHWiQMPSpIhOBnmPDFtKRPipdIHVtlb
  
-===== Data Replication =====+===== Data Replication into ElasticSearch =====
  
 Data is replicated via sql queries which can be customised according to the customer or use case. These queries are stored in /var/springboot/elastic-replicator/queries. Data is replicated via sql queries which can be customised according to the customer or use case. These queries are stored in /var/springboot/elastic-replicator/queries.
Line 33: Line 33:
 {{ :onboarding:elastic-replicator:screenshot_2022-10-04_at_12.46.59.png?600 |}} {{ :onboarding:elastic-replicator:screenshot_2022-10-04_at_12.46.59.png?600 |}}
  
 +===== Scheduled Imports =====
 +The bulk import is configured with quartz to run on a schedule (see BulkRouter.java for logic details). This refreshes the data by re-running the queries for all datatypes and updating the existing elastic search indexes. The timing of the cron is configured in the application properties as follows:
  
-The date can be imported either via individual or bulk mechanisms:+<code> 
 +bulk.import.quartz.schedule=0 0 1 1/1 * ? * 
 +</code> 
 + 
 +{{ :onboarding:elastic-replicator:screenshot_2022-10-04_at_14.22.22.png?600 |}} 
 + 
 +===== On Demand Imports ===== 
 +The data can be imported either via individual or bulk mechanisms as follows:
  
 **Individual:** **Individual:**
-  - GET http://10.91.140.218:8100/bulk?routes=trap +  - GET http://10.91.140.218:8100/elastic-replicator/bulk?routes=trap 
-  - GET http://10.91.140.218:8100/bulk?routes=networkElement +  - GET http://10.91.140.218:8100/elastic-replicator/bulk?routes=networkElement 
-  - GET http://10.91.140.218:8100/bulk?routes=ticket +  - GET http://10.91.140.218:8100/elastic-replicator/bulk?routes=ticket 
-  - GET http://10.91.140.218:8100/bulk?routes=alarm +  - GET http://10.91.140.218:8100/elastic-replicator/bulk?routes=activeAlarm 
-  - GET http://10.91.140.218:8100/bulk?routes=uptime+  - GET http://10.91.140.218:8100/elastic-replicator/bulk?routes=uptime 
 + 
 +**Limited individual** 
 +  - GET "http://10.91.140.218:8100/bulk?routes=uptime&start=1" 
 + 
 +to limit to 1 month back.
  
 **Bulk:** **Bulk:**
Line 60: Line 74:
 {{ :onboarding:elastic-replicator:ems_nova_tech._design-architecture.drawio.png?600 |}} {{ :onboarding:elastic-replicator:ems_nova_tech._design-architecture.drawio.png?600 |}}
  
-===== Data Replication into Elastic Search ===== + 
-=== Scheduled Imports === +===== SSE(Server Sent Events Process) ===== 
-=== On Demand Imports ===+The SSE subscription drives real time update receipt in the angular ems-nova front end application.  
 +Server-Sent Events (SSE) is a server push technology enabling a client to receive automatic updates from a server via an HTTP connection, and describes how servers can initiate data transmission towards clients once an initial client connection has been established. 
 + 
 +In this case nova subscribes to the idms-proxy sse endpoint and listens for updates: 
 +<code> 
 +/idms-proxy/api/v0/replicator 
 +</code> 
 + 
 +The SSE touchpoints for  processing are as follows(see also diagram below): 
 +  - Gorm listener events for ticket, alarm, element inserts or updates are pushed from Ticketer and SnmpManager onto rabbitmq. 
 +  - Elastic Replicator consumes messages from these queues, processes them using the sql queries and updates or inserts new records into elastic search. It also pushes the events onto a queue which is consumed by idms-proxy. 
 +  - idms-proxy consumes a queue which receives event messages from elastic-replicator. 
 +  - Nova front end is subscribed to proxies sse endpoint /replicator and receives messages for real time page updates.
  
  
-===== SSE =====+{{ :onboarding:elastic-replicator:eventnotificationprocess.drawio.png?600 |}}
   * Analysis document: https://docs.google.com/document/d/1nUBZtjKfTnopua9quoIhOpeugYVlCYHB8udhRcXwbS4/edit#heading=h.e1lpa8mdes4i   * Analysis document: https://docs.google.com/document/d/1nUBZtjKfTnopua9quoIhOpeugYVlCYHB8udhRcXwbS4/edit#heading=h.e1lpa8mdes4i
   * Requirements document: https://docs.google.com/document/d/14mxFTS-B7BaPDNNERJVtzFEZlsCRDIQTFLbgOLaVQxo/edit#heading=h.euwbsr2vf8os   * Requirements document: https://docs.google.com/document/d/14mxFTS-B7BaPDNNERJVtzFEZlsCRDIQTFLbgOLaVQxo/edit#heading=h.euwbsr2vf8os
  
 ===== Development Configuration ===== ===== Development Configuration =====
-Project can be pointed at local or server deployed version of all api endpoints listed above. +Project can be pointed at local or server deployed version of all /bulk api endpoints listed above in the import section. 
 See src/main/resources/application.properties for environment configuration for local dev. See src/main/resources/application.properties for environment configuration for local dev.
  
onboarding/elastic-replicator/start.1664884703.txt.gz · Last modified: 2022/10/04 12:58 by 10.91.110.100

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki