Table of Contents
Elastic Replicator
Updated by Anna Dowling 04/10/2022
Overview
The elastic-replicator application is utilised as a data importer / syncing tool for elastic-search. This data is then leveraged in ems-nova front end for fast loading times via apis setup in idms-proxy to query elastic search. The solution was chosen to have a separate Replica (using ElasticSearch) for the reads from nova. The Replica is kept in sync with the main SQL DB and the index structure is optimised for the use-cases such as tree level building / enhanced reporting(usually slicing and aggregation). This results in lowering the SQL load and reaching much better performance for the reads and front end rendering response times. Elastic Replicator also handles on demand inserts and updates and ties into the SSE (server sent events) process for updating the front end in real time.
Repo: https://bitbucket.org/errigal/elastic-replicator/branch/master
Project setup README: https://bitbucket.org/errigal/elastic-replicator/src/master/README.md
Local Dev App Urls
Project Documents
Data Replication into ElasticSearch
Data is replicated via sql queries which can be customised according to the customer or use case. These queries are stored in /var/springboot/elastic-replicator/queries. The selected fields used in the queries use a unified naming convention in order to provide consistency for the elastic search record indexes.
The queries are currently stored for deployment in the deployment-playbooks project which has subfolder structure allowing for customer specific query overrides:
Scheduled Imports
The bulk import is configured with quartz to run on a schedule (see BulkRouter.java for logic details). This refreshes the data by re-running the queries for all datatypes and updating the existing elastic search indexes. The timing of the cron is configured in the application properties as follows:
bulk.import.quartz.schedule=0 0 1 1/1 * ? *
On Demand Imports
The data can be imported either via individual or bulk mechanisms as follows:
Individual:
Limited individual
to limit to 1 month back.
Bulk:
Playbook:
- initialise-nova-es.yml (deployment-playbooks)
Classes to note for import process:
BulkRouter.java - handles bulk import for records SyncRouter.java - handles real time updates for records
Architecture
SSE(Server Sent Events Process)
The SSE subscription drives real time update receipt in the angular ems-nova front end application. Server-Sent Events (SSE) is a server push technology enabling a client to receive automatic updates from a server via an HTTP connection, and describes how servers can initiate data transmission towards clients once an initial client connection has been established.
In this case nova subscribes to the idms-proxy sse endpoint and listens for updates:
/idms-proxy/api/v0/replicator
The SSE touchpoints for processing are as follows(see also diagram below):
- Gorm listener events for ticket, alarm, element inserts or updates are pushed from Ticketer and SnmpManager onto rabbitmq.
- Elastic Replicator consumes messages from these queues, processes them using the sql queries and updates or inserts new records into elastic search. It also pushes the events onto a queue which is consumed by idms-proxy.
- idms-proxy consumes a queue which receives event messages from elastic-replicator.
- Nova front end is subscribed to proxies sse endpoint /replicator and receives messages for real time page updates.
Development Configuration
Project can be pointed at local or server deployed version of all /bulk api endpoints listed above in the import section. See src/main/resources/application.properties for environment configuration for local dev.
Deployment
The elastic-replicator is deployed to the server handlers in /var/spring-boot/ and is load balanced. Its environment specific variables are passed via the application.properties file to the app at runtime.
Current Config Fields Are:
# Build Info
jenkins.build.identifier=elastic-replicator » elastic-replicator-feature-branches » REL-4.2.0 REL-4.2.0.5
jenkins.build.branch=REL-4.2.0
jenkins.build.number=5
jenkins.build.time=2022-08-20 00:16:34
deployment.time=2022-09-06 10:10:49
# Spring-boot
server.port=8100
server.servlet.context-path=/elastic-replicator
# RabbitMQ settings - for factory (deprecated since DNT-beta)
spring.rabbitmq.host=scolb1
spring.rabbitmq.username=rabbit
spring.rabbitmq.password=#rabbit321!#
spring.rabbitmq.port=5672
spring.rabbitmq.ticketerVHost=admin
spring.rabbitmq.snmpVHost=admin
# RabbitMQ settings - used by camel directly (without factory)
camel.component.rabbitmq.hostname=scolb1
camel.component.rabbitmq.username=rabbit
camel.component.rabbitmq.password=#rabbit321!#
camel.component.rabbitmq.port-number=5672
camel.component.rabbitmq.vhost=admin
rabbitmq.snmp.gorm.vhost=ems
rabbitmq.consumers.start=true
# DB settings
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url = jdbc:mysql://scolb1.err:3306/snmp_manager?autoReconnect=true&useSSL=false&zeroDateTimeBehavior=convertToNull&serverTimezone=UTC
spring.datasource.username=root
spring.datasource.password=ozzrules
spring.jpa.database-platform=org.hibernate.dialect.MySQL5InnoDBDialect
spring.datasource.hikari.connection-test-query=SELECT 1
# ElasticSearch settings
spring.elasticsearch.rest.uris=scolb1:9200
elastic.index.prefix=nova.
# Apache Camel settings
camel.springboot.tracing = false
# Bulk Import
bulk.import.quartz.schedule = 0 0 1 1/1 * ? *
# General logging settings
spring.application.name=elastic-replicator
logging.file.max-history=5
logging.file.max-size=50MB
logging.path=/export/home/scotty/logs/spring-boot
logging.file.name=${logging.path}/elastic-replicator.log
# Data obfuscation
obfuscation.enabled=false
# Data Queries Location
queries.directory.path=/var/spring-boot/elastic-replicator/queries
Ansible Playbooks
ansible-playbook -i ../env-configuration/nova/hosts.ini --diff --vault-id @prompt elastic-replicator.yml --extra-vars "jenkins_build_name='elastic-replicator-feature-branches/job/branchName' appFileExtension='.jar' jenkins_job_build_num=buildNumber actions='upload,stop,deploy,configure,start'"
ansible-playbook -i ../env-configuration/nova/hosts.ini --diff --vault-id @prompt initialise-nova-es.yml