An MDC environment has atleast the following servers:

Orchestrator, Tracker, Agent are jars which usually go on the same server.

All the project sources are listed below:

https://bitbucket.org/errigal/rdf_agent
https://bitbucket.org/errigal/rdf_orchestrator
https://bitbucket.org/errigal/rdf_tracker
https://bitbucket.org/errigal/rdf-agent-supervisor

The dependancy required there is JDK 13 (which the playbooks handle themselves)

if not, use sdkman to install the required version of java on server that requires a jar to run.

These servers need the following application servers for sync and storage. The playbooks to install these services are in the Server Configuration project.

1. Load Balancer
   1. RabbitMQ
   2. etcd
   3. MQTT
2. OAT Servers
   1. etcd
3. ESK Server
   1. ElasticSearch + Kibana
   2. MySQL8

• create a vhost called rdf
• user called rdf with an associated password from env-configuration for that env.
• the rdf user will have full admin access to the virtual host: rdf

This can be done via the following playbooks.

Install rabbitmq (solidmmw example)

ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-rabbitmq.yml --extra-vars "host_to_install=rdf-rabbitmq install_plugins=true nodename=solidmmwlbdbrmq" --limit=rdf-rabbitmq

Setup user

ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-rabbitmq-user.yml --limit=rdf-rabbitmq

The playbooks might not take care of this next step so it may need to be done via the rabbitmq management UI.

• Create a vhost called rdf_out and make sure the rdf user has permissions on it

You might usually have an external volume(for Mysql8 and elasticsearch), to format use the following commands, replace vdb with whatever disk you have.

sudo mkfs.ext4 /dev/vdb
Once this returns done, the volume is formatted. Check if it worked with sudo lsblk -f. You will see the volume, but no mountpoint yet.
sudo mkdir /data
sudo chmod -R 0777 /data
sudo mount /dev/vdb /data
sudo chmod -R 0777 /data
Make an entry in /etc/fstab file for permanent mount at boot time.
/dev/vdb	/data	ext4	defaults     0   0

(Install elasticsearch: lookup official website for installing the rpm version) Follow the instruction in this guide.

sudo mkdir /data/elasticsearch && sudo chmod -R 0777 /data
elastic search install config in (/etc/elasticsearch/elasticsearch.yml:
path.data: /data/elasticsearch  (this is the mounted volume)
network.host: 0.0.0.0
discovery.type: single-node  (for single node)

#configure higher heap based on recommendations for current version of ElasticSearch

sudo systemctl stop elasticsearch.service
sudo systemctl start elasticsearch.service

make sure elastic search starts on reboot.

sudo chkconfig --add elasticsearch

(Install Kibana: lookup official website for installing the rpm version) Follow the guide here. Update /etc/kibana/kibana.yml with this config

server.host: "0.0.0.0"

make sure Kibana starts on reboot.

sudo chkconfig --add kibana
sudo -i service kibana start

Ensure that a higher field limit template is present for elasticsearch, you may fire this query below in Kibana → devtools (Wrench)

go to esk ip address:5601/app/dev_tools#/console in browser

PUT /_template/rdftemplate
{
"order" : 1,
"index_patterns" : "rdf*",
"settings" : {
"index" : {
"mapping.total_fields.limit" : "15000"
 }
}}

Create a DNS entry in cloudflare for the orchestrator UI using the load balancer public IP address.

Install Nginx - point the LB to whatever orchestrators are installed and configure SSL if required

sudo yum install epel-release
sudo yum install nginx
sudo vim /etc/nginx/nginx.conf

This is a sample config from solid using SSL

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile              on;
    tcp_nopush            on;
    tcp_nodelay           on;
    keepalive_timeout     300;
    proxy_connect_timeout 300;
    proxy_send_timeout    300;
    proxy_read_timeout    300;
    send_timeout          300;

    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    upstream orchestrators {
       server 10.91.140.28:8079;
       server 10.91.140.181:8079;
    }

    server {
        server_name  solidmmwrdflb.errigal.com;
        #root         /usr/share/nginx/html;
        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
                    proxy_pass http://orchestrators;
        }
        
        location /ws {
            proxy_pass http://orchestrators;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    listen 443 ssl;
    ssl_certificate /etc/nginx/solidmmwrdflb.errigal.com/sitecert.fullchain.pem;
    ssl_certificate_key /etc/nginx/solidmmwrdflb.errigal.com/private-key.pem;
}

server {
    if ($host = solidmmwrdflb.errigal.com) {
        return 301 https://$host$request_uri;
    }


        server_name  solidmmwrdflb.errigal.com;
    listen 80;
    return 404;


}
}

Next you need to run this command to allow connectivity

sudo setsebool -P httpd_can_network_connect 1
sudo service nginx restart

make sure the data dir is pointing to the bigger mounted volume

https://dev.mysql.com/doc/refman/8.0/en/linux-installation-yum-repo.html

ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-mysql8.yml   

you need minimum three schemas: orchestrator, tracker, <name_of_cust_in_tracker>, <name_of_cust_in_tracker> schema and connection details must be mapped in tracker.customer_data_source

ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-etcd.yml 
ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-mqtt.yml  

Example Jar Deployment Ansible Command

ansible-playbook -i ../env-configuration/towerqa/hosts --diff --vault-id @prompt redeploy-orchestrator.yml --extra-vars "jenkins_build_name='rdf-orchestrator-feature-branches/job/master' jenkins_job_build_num=330"

ansible-playbook -i ../env-configuration/towerqa/hosts --diff --vault-id @prompt redeploy-tracker.yml --extra-vars "jenkins_build_name='rdf-tracker-feature-branches/job/master' jenkins_job_build_num=107"

Generating Access Token:

curl --location --request GET 'https://<orchestratorlb>/api/v1/admin/agent/token/generate?agentUsername=<agent_username>' \
--header 'Authorization: Bearer <TOKEN>' \
--header 'Content-Type: application/json'


where do you get the above token in the first place?
Login to the orchestrator UI in your browser, look in the cookies section of your browser and you should see it,
else you could also use basic auth instead of using the admin token.

This access token should be added to the env configuration for the orchestrator. Also the snmp_manager.orchestrator_access_credentials table should be updated with this token.

Agent deployment KLA example

1. Go to https://klardflb.errigal.com/rdf_public/dashboard/layout.html#/login in your browser and log into the orchestrator UI. Credentials in password safe.
2. Navigate to Agent Versions
3. For the new Agent version populate with this info.
   1. Version - 1.8.3
   2. Location - https://s3.amazonaws.com/rdf.agent.downloads/v1/rdf_agent_master.772.jar
   3. Hash - can copy another versions hash
4. Navigate to Customers - KLA_FORD - kla_ford_crash_barrier
5. Fill in version 1.8.3 into the current version field and save. That’s it!
6. Refresh after 5 mins to check if the reportedVersion matches the current version.

Note: Do not forget to update the monitoring config for whatever environment you are deploying to https://bitbucket.org/errigal/prometheus-monitoring-config

Monitoring rules to be updated for the following

Orchestrator, Tracker
RabbitMQ
ElasticSearch + Kibana
MySQL8
etcd 
MQTT 

This may vary from env to env. and upto us on how we configure ElasticSearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html

MySQL8 - same as standard MySQL5 backup (again varies from env to env)

mini walkthrough: https://drive.google.com/file/d/1VO5aL1n6JFVZbgr1y-3rtA0UOkn6LJk7/view?usp=sharing The walkthrough video might be horribly out of date.