User Tools
This is an old revision of the document!
Table of Contents
Elastic Replicator
Updated by Anna Dowling 04/10/2022
Overview
The elastic-replicator application is utilised as a data importer / syncing tool for elastic-search. This data is then leveraged in ems-nova front end for fast loading times via apis setup in idms-proxy to query elastic search. The solution was chosen to have a separate Replica (using ElasticSearch) for the reads from nova. The Replica is kept in sync with the main SQL DB and the index structure is optimised for the use-cases such as tree level building / enhanced reporting(usually slicing and aggregation). This results in lowering the SQL load and reaching much better performance for the reads and front end rendering response times. Elastic Replicator also handles on demand inserts and updates and ties into the SSE (server sent events) process for updating the front end in real time.
Repo: https://bitbucket.org/errigal/elastic-replicator/branch/master
Project setup README: https://bitbucket.org/errigal/elastic-replicator/src/master/README.md
Local Dev App Urls
Project Documents
Data Replication into ElasticSearch
Data is replicated via sql queries which can be customised according to the customer or use case. These queries are stored in /var/springboot/elastic-replicator/queries. The selected fields used in the queries use a unified naming convention in order to provide consistency for the elastic search record indexes.
The queries are currently stored for deployment in the deployment-playbooks project which has subfolder structure allowing for customer specific query overrides:
Scheduled Imports
The bulk import is configured with quartz to run on a schedule (see BulkRouter.java for logic details). This refreshes the data by re-running the queries for all datatypes and updating the existing elastic search indexes. The timing of the cron is configured in the application properties as follows:
bulk.import.quartz.schedule=0 0 1 1/1 * ? *
On Demand Imports
The data can be imported either via individual or bulk mechanisms as follows:
Individual:
Bulk:
Playbook:
- initialise-nova-es.yml (deployment-playbooks)
Classes to note for import process:
BulkRouter.java - handles bulk import for records SyncRouter.java - handles real time updates for records
Architecture
SSE(Server Sent Events Process)
Development Configuration
Project can be pointed at local or server deployed version of all /bulk api endpoints listed above in the import section. See src/main/resources/application.properties for environment configuration for local dev.
Deployment
The elastic-replicator is deployed to the server handlers in /var/spring-boot/ and is load balanced. Its environment specific variables are passed via the application.properties file to the app at runtime.
Current Config Fields Are:
# Build Info
jenkins.build.identifier=elastic-replicator » elastic-replicator-feature-branches » REL-4.2.0 REL-4.2.0.5
jenkins.build.branch=REL-4.2.0
jenkins.build.number=5
jenkins.build.time=2022-08-20 00:16:34
deployment.time=2022-09-06 10:10:49
# Spring-boot
server.port=8100
server.servlet.context-path=/elastic-replicator
# RabbitMQ settings - for factory (deprecated since DNT-beta)
spring.rabbitmq.host=scolb1
spring.rabbitmq.username=rabbit
spring.rabbitmq.password=#rabbit321!#
spring.rabbitmq.port=5672
spring.rabbitmq.ticketerVHost=admin
spring.rabbitmq.snmpVHost=admin
# RabbitMQ settings - used by camel directly (without factory)
camel.component.rabbitmq.hostname=scolb1
camel.component.rabbitmq.username=rabbit
camel.component.rabbitmq.password=#rabbit321!#
camel.component.rabbitmq.port-number=5672
camel.component.rabbitmq.vhost=admin
rabbitmq.snmp.gorm.vhost=ems
rabbitmq.consumers.start=true
# DB settings
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url = jdbc:mysql://scolb1.err:3306/snmp_manager?autoReconnect=true&useSSL=false&zeroDateTimeBehavior=convertToNull&serverTimezone=UTC
spring.datasource.username=root
spring.datasource.password=ozzrules
spring.jpa.database-platform=org.hibernate.dialect.MySQL5InnoDBDialect
spring.datasource.hikari.connection-test-query=SELECT 1
# ElasticSearch settings
spring.elasticsearch.rest.uris=scolb1:9200
elastic.index.prefix=nova.
# Apache Camel settings
camel.springboot.tracing = false
# Bulk Import
bulk.import.quartz.schedule = 0 0 1 1/1 * ? *
# General logging settings
spring.application.name=elastic-replicator
logging.file.max-history=5
logging.file.max-size=50MB
logging.path=/export/home/scotty/logs/spring-boot
logging.file.name=${logging.path}/elastic-replicator.log
# Data obfuscation
obfuscation.enabled=false
# Data Queries Location
queries.directory.path=/var/spring-boot/elastic-replicator/queries
Ansible Playbooks
ansible-playbook -i ../env-configuration/nova/hosts.ini --diff --vault-id @prompt elastic-replicator.yml --extra-vars "jenkins_build_name='elastic-replicator-feature-branches/job/branchName' appFileExtension='.jar' jenkins_job_build_num=buildNumber actions='upload,stop,deploy,configure,start'"
ansible-playbook -i ../env-configuration/nova/hosts.ini --diff --vault-id @prompt initialise-nova-es.yml
Jenkins Build
