An MDC environment has atleast the following servers:
Orchestrator, Tracker, Agent are jars which usually go on the same server.
All the project sources are listed below:
https://bitbucket.org/errigal/rdf_agent https://bitbucket.org/errigal/rdf_orchestrator https://bitbucket.org/errigal/rdf_tracker https://bitbucket.org/errigal/rdf-agent-supervisor
The dependancy required there is JDK 13 (which the playbooks handle themselves)
if not, use sdkman to install the required version of java on server that requires a jar to run.
These servers need the following application servers for sync and storage. The playbooks to install these services are in the Server Configuration project.
This can be done via the following playbooks.
Install rabbitmq (solidmmw example)
ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-rabbitmq.yml --extra-vars "host_to_install=rdf-rabbitmq install_plugins=true nodename=solidmmwlbdbrmq" --limit=rdf-rabbitmq
Setup user
ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-rabbitmq-user.yml --limit=rdf-rabbitmq
The playbooks might not take care of this next step so it may need to be done via the rabbitmq management UI.
You might usually have an external volume(for Mysql8 and elasticsearch), to format use the following commands, replace vdb with whatever disk you have.
sudo mkfs.ext4 /dev/vdb Once this returns done, the volume is formatted. Check if it worked with sudo lsblk -f. You will see the volume, but no mountpoint yet. sudo mkdir /data sudo chmod -R 0777 /data sudo mount /dev/vdb /data sudo chmod -R 0777 /data Make an entry in /etc/fstab file for permanent mount at boot time. /dev/vdb /data ext4 defaults 0 0
(Install elasticsearch: lookup official website for installing the rpm version) Follow the instruction in this guide.
sudo mkdir /data/elasticsearch && sudo chmod -R 0777 /data elastic search install config in (/etc/elasticsearch/elasticsearch.yml: path.data: /data/elasticsearch (this is the mounted volume) network.host: 0.0.0.0 discovery.type: single-node (for single node) #configure higher heap based on recommendations for current version of ElasticSearch
sudo systemctl stop elasticsearch.service sudo systemctl start elasticsearch.service
make sure elastic search starts on reboot.
sudo chkconfig --add elasticsearch
(Install Kibana: lookup official website for installing the rpm version) Follow the guide here. Update /etc/kibana/kibana.yml with this config
server.host: "0.0.0.0"
make sure Kibana starts on reboot.
sudo chkconfig --add kibana sudo -i service kibana start
Ensure that a higher field limit template is present for elasticsearch, you may fire this query below in Kibana → devtools (Wrench)
go to esk ip address:5601/app/dev_tools#/console in browser
PUT /_template/rdftemplate
{
"order" : 1,
"index_patterns" : "rdf*",
"settings" : {
"index" : {
"mapping.total_fields.limit" : "15000"
}
}}
Create a DNS entry in cloudflare for the orchestrator UI using the load balancer public IP address.
Install Nginx - point the LB to whatever orchestrators are installed and configure SSL if required
sudo yum install epel-release sudo yum install nginx sudo vim /etc/nginx/nginx.conf
This is a sample config from solid using SSL
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 300;
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
upstream orchestrators {
server 10.91.140.28:8079;
server 10.91.140.181:8079;
}
server {
server_name solidmmwrdflb.errigal.com;
#root /usr/share/nginx/html;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
proxy_pass http://orchestrators;
}
location /ws {
proxy_pass http://orchestrators;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
listen 443 ssl;
ssl_certificate /etc/nginx/solidmmwrdflb.errigal.com/sitecert.fullchain.pem;
ssl_certificate_key /etc/nginx/solidmmwrdflb.errigal.com/private-key.pem;
}
server {
if ($host = solidmmwrdflb.errigal.com) {
return 301 https://$host$request_uri;
}
server_name solidmmwrdflb.errigal.com;
listen 80;
return 404;
}
}
Next you need to run this command to allow connectivity
sudo setsebool -P httpd_can_network_connect 1 sudo service nginx restart
make sure the data dir is pointing to the bigger mounted volume
https://dev.mysql.com/doc/refman/8.0/en/linux-installation-yum-repo.html
ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-mysql8.yml
you need minimum three schemas: orchestrator, tracker, <name_of_cust_in_tracker>, <name_of_cust_in_tracker> schema and connection details must be mapped in tracker.customer_data_source
ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-etcd.yml ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-mqtt.yml
In order for the ETCDCTL command line tool to work you will need to execute this command on the MDC loadbalancer, oat1 and oat2 servers.
export ETCDCTL_ENDPOINTS=http://$server_IP:2379
Example Jar Deployment Ansible Command
ansible-playbook -i ../env-configuration/towerqa/hosts --diff --vault-id @prompt redeploy-orchestrator.yml --extra-vars "jenkins_build_name='rdf-orchestrator-feature-branches/job/master' jenkins_job_build_num=330" ansible-playbook -i ../env-configuration/towerqa/hosts --diff --vault-id @prompt redeploy-tracker.yml --extra-vars "jenkins_build_name='rdf-tracker-feature-branches/job/master' jenkins_job_build_num=107"
Generating Access Token:
curl --location --request GET 'https://<orchestratorlb>/api/v1/admin/agent/token/generate?agentUsername=<agent_username>' \ --header 'Authorization: Bearer <TOKEN>' \ --header 'Content-Type: application/json' where do you get the above token in the first place? Login to the orchestrator UI in your browser, look in the cookies section of your browser and you should see it, else you could also use basic auth instead of using the admin token.
This access token should be added to the env configuration for the orchestrator. Also the snmp_manager.orchestrator_access_credentials table should be updated with this token.
Agent deployment KLA example
Note: Do not forget to update the monitoring config for whatever environment you are deploying to https://bitbucket.org/errigal/prometheus-monitoring-config
Monitoring rules to be updated for the following
Orchestrator, Tracker RabbitMQ ElasticSearch + Kibana MySQL8 etcd MQTT
This may vary from env to env. and upto us on how we configure ElasticSearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html
MySQL8 - same as standard MySQL5 backup (again varies from env to env)
mini walkthrough: https://drive.google.com/file/d/1VO5aL1n6JFVZbgr1y-3rtA0UOkn6LJk7/view?usp=sharing The walkthrough video might be horribly out of date.