User Tools
onboarding:elastic-replicator:start
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| onboarding:elastic-replicator:start [2022/10/04 12:55] – 10.91.110.100 | onboarding:elastic-replicator:start [2024/08/29 11:28] (current) – 10.91.129.119 | ||
|---|---|---|---|
| Line 24: | Line 24: | ||
| * https:// | * https:// | ||
| - | ===== Data Replication ===== | + | ===== Data Replication |
| Data is replicated via sql queries which can be customised according to the customer or use case. These queries are stored in / | Data is replicated via sql queries which can be customised according to the customer or use case. These queries are stored in / | ||
| Line 33: | Line 33: | ||
| {{ : | {{ : | ||
| + | ===== Scheduled Imports ===== | ||
| + | The bulk import is configured with quartz to run on a schedule (see BulkRouter.java for logic details). This refreshes the data by re-running the queries for all datatypes and updating the existing elastic search indexes. The timing of the cron is configured in the application properties as follows: | ||
| - | The date can be imported either via individual or bulk mechanisms: | + | < |
| + | bulk.import.quartz.schedule=0 0 1 1/1 * ? * | ||
| + | </ | ||
| - | Individual: | + | {{ :onboarding:elastic-replicator:screenshot_2022-10-04_at_14.22.22.png?600 |}} |
| - | - GET http:// | + | |
| - | | + | |
| - | | + | |
| - | | + | |
| - | - GET http:// | + | |
| - | Bulk: | + | ===== On Demand Imports ===== |
| + | The data can be imported either via individual or bulk mechanisms as follows: | ||
| + | |||
| + | **Individual: | ||
| + | - GET http:// | ||
| + | - GET http:// | ||
| + | - GET http:// | ||
| + | - GET http:// | ||
| + | - GET http:// | ||
| + | |||
| + | **Limited individual** | ||
| + | - GET " | ||
| + | |||
| + | to limit to 1 month back. | ||
| + | |||
| + | **Bulk:** | ||
| - http:// | - http:// | ||
| - | Playbook: | + | **Playbook:** |
| - initialise-nova-es.yml (deployment-playbooks) | - initialise-nova-es.yml (deployment-playbooks) | ||
| + | **Classes to note for import process:** | ||
| < | < | ||
| + | BulkRouter.java - handles bulk import for records | ||
| + | SyncRouter.java - handles real time updates for records | ||
| </ | </ | ||
| Line 58: | Line 73: | ||
| ===== Architecture ===== | ===== Architecture ===== | ||
| {{ : | {{ : | ||
| + | |||
| + | |||
| + | ===== SSE(Server Sent Events Process) ===== | ||
| + | The SSE subscription drives real time update receipt in the angular ems-nova front end application. | ||
| + | Server-Sent Events (SSE) is a server push technology enabling a client to receive automatic updates from a server via an HTTP connection, and describes how servers can initiate data transmission towards clients once an initial client connection has been established. | ||
| + | |||
| + | In this case nova subscribes to the idms-proxy sse endpoint and listens for updates: | ||
| + | < | ||
| + | / | ||
| + | </ | ||
| + | |||
| + | The SSE touchpoints for processing are as follows(see also diagram below): | ||
| + | - Gorm listener events for ticket, alarm, element inserts or updates are pushed from Ticketer and SnmpManager onto rabbitmq. | ||
| + | - Elastic Replicator consumes messages from these queues, processes them using the sql queries and updates or inserts new records into elastic search. It also pushes the events onto a queue which is consumed by idms-proxy. | ||
| + | - idms-proxy consumes a queue which receives event messages from elastic-replicator. | ||
| + | - Nova front end is subscribed to proxies sse endpoint /replicator and receives messages for real time page updates. | ||
| + | |||
| + | |||
| + | {{ : | ||
| + | * Analysis document: https:// | ||
| + | * Requirements document: https:// | ||
| ===== Development Configuration ===== | ===== Development Configuration ===== | ||
| - | Project can be pointed at local or server deployed version of all api endpoints listed above. | + | Project can be pointed at local or server deployed version of all /bulk api endpoints listed above in the import section. |
| See src/ | See src/ | ||
| Line 139: | Line 175: | ||
| </ | </ | ||
| - | ===== Data Replication into Elastic Search ===== | ||
| - | === Scheduled Imports === | ||
| - | === On Demand Imports === | ||
| - | |||
| - | |||
| - | ===== SSE ===== | ||
| - | * Analysis document: https:// | ||
| - | * Requirements document: https:// | ||
| === Jenkins Build === | === Jenkins Build === | ||
| https:// | https:// | ||
onboarding/elastic-replicator/start.1664884506.txt.gz · Last modified: 2022/10/04 12:55 by 10.91.110.100
Page Tools
