Internal Errigal Collaboration Wiki

User Tools

Site Tools

Trace: • rdf_server_setup • supportrequestpageticket_management
Writing /app/www/public/data/meta/toolsandtechnologies/rdf_server_setup.meta failed
toolsandtechnologies:rdf_server_setup

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
toolsandtechnologies:rdf_server_setup [2020/07/17 12:43] – adsilvatoolsandtechnologies:rdf_server_setup [2021/09/13 14:43] (current) – 10.91.120.28
Line 1: Line 1:
 +====== Overview ======
  
 +
 +
 +An MDC environment has atleast the following servers:
 +
 +
 +Orchestrator, Tracker, Agent are jars which usually go on the same server.
 +
 +All the project sources are listed below:
 +
 +<code>
 +https://bitbucket.org/errigal/rdf_agent
 +https://bitbucket.org/errigal/rdf_orchestrator
 +https://bitbucket.org/errigal/rdf_tracker
 +https://bitbucket.org/errigal/rdf-agent-supervisor
 +</code>
 +
 +
 +The dependancy required there is JDK 13 (which the playbooks handle themselves)
 +
 +if not, use sdkman to install the required version of java on server that requires a jar to run.
 +
 +
 +These servers need the following application servers for sync and storage. The playbooks to install these services are in the [[https://bitbucket.org/errigal/server-configuration/src/master/|Server Configuration]] project.
 +  - Load Balancer
 +    - RabbitMQ
 +    - etcd
 +    - MQTT
 +  - OAT Servers
 +    - etcd
 +  - ESK Server 
 +    - ElasticSearch + Kibana
 +    - MySQL8
 +
 +====== RabbitMQ ======
 +
 +  * create a vhost called rdf
 +  * user called rdf with an associated password from env-configuration for that env.
 +  * the rdf user will have full admin access to the virtual host: rdf
 +
 +This can be done via the following playbooks.
 +
 +Install rabbitmq (solidmmw example)
 +<code>
 +ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-rabbitmq.yml --extra-vars "host_to_install=rdf-rabbitmq install_plugins=true nodename=solidmmwlbdbrmq" --limit=rdf-rabbitmq
 +</code>
 +
 +Setup user
 +<code>
 +ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-rabbitmq-user.yml --limit=rdf-rabbitmq
 +</code>
 +
 +The playbooks might not take care of this next step so it may need to be done via the rabbitmq management UI.
 +  * Create a vhost called rdf_out and make sure the rdf user has permissions on it
 +
 +====== External Volumes ======
 +
 +You might usually have an external volume(for Mysql8 and elasticsearch), to format use the following commands,
 +replace vdb with whatever disk you have.
 +
 +<code>
 +sudo mkfs.ext4 /dev/vdb
 +Once this returns done, the volume is formatted. Check if it worked with sudo lsblk -f. You will see the volume, but no mountpoint yet.
 +sudo mkdir /data
 +sudo chmod -R 0777 /data
 +sudo mount /dev/vdb /data
 +sudo chmod -R 0777 /data
 +Make an entry in /etc/fstab file for permanent mount at boot time.
 +/dev/vdb /data ext4 defaults     0   0
 +</code>
 +
 +====== ElasticSearch ======
 +
 +(Install elasticsearch: lookup official website for installing the rpm version)
 +Follow the instruction in this [[https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html|guide]].
 +
 +
 +<code>
 +sudo mkdir /data/elasticsearch && sudo chmod -R 0777 /data
 +elastic search install config in (/etc/elasticsearch/elasticsearch.yml:
 +path.data: /data/elasticsearch  (this is the mounted volume)
 +network.host: 0.0.0.0
 +discovery.type: single-node  (for single node)
 +
 +#configure higher heap based on recommendations for current version of ElasticSearch
 +</code>
 +
 +
 +<code>
 +sudo systemctl stop elasticsearch.service
 +sudo systemctl start elasticsearch.service
 +</code>
 +
 +make sure elastic search starts on reboot.
 +<code>
 +sudo chkconfig --add elasticsearch
 +</code>
 +
 +====== Kibana ======
 +
 +(Install Kibana: lookup official website for installing the rpm version)
 +Follow the guide [[https://www.elastic.co/guide/en/kibana/current/rpm.html|here]].
 +Update /etc/kibana/kibana.yml with this config
 +<code>
 +server.host: "0.0.0.0"
 +</code>
 +
 +make sure Kibana starts on reboot.
 +
 +<code>
 +sudo chkconfig --add kibana
 +sudo -i service kibana start
 +</code>
 +
 +Ensure that a higher field limit template is present for elasticsearch, you may fire this query below in Kibana -> devtools (Wrench)
 +
 +go to esk ip address:5601/app/dev_tools#/console in browser
 +
 +<code>
 +PUT /_template/rdftemplate
 +{
 +"order" : 1,
 +"index_patterns" : "rdf*",
 +"settings" : {
 +"index" : {
 +"mapping.total_fields.limit" : "15000"
 + }
 +}}
 +</code>
 +
 +====== Cloudflare ======
 +Create a DNS entry in cloudflare for the orchestrator UI using the load balancer public IP address.
 +
 +====== NGINX ======
 +
 +Install Nginx
 + - point the LB to whatever orchestrators are installed and configure SSL if required
 +
 +<code>
 +sudo yum install epel-release
 +sudo yum install nginx
 +sudo vim /etc/nginx/nginx.conf
 +</code>
 +
 +This is a sample config from solid using SSL
 +<code>
 +# For more information on configuration, see:
 +#   * Official English Documentation: http://nginx.org/en/docs/
 +#   * Official Russian Documentation: http://nginx.org/ru/docs/
 +
 +user nginx;
 +worker_processes auto;
 +error_log /var/log/nginx/error.log;
 +pid /run/nginx.pid;
 +
 +# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
 +include /usr/share/nginx/modules/*.conf;
 +
 +events {
 +    worker_connections 1024;
 +}
 +
 +http {
 +    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
 +                      '$status $body_bytes_sent "$http_referer" '
 +                      '"$http_user_agent" "$http_x_forwarded_for"';
 +
 +    access_log  /var/log/nginx/access.log  main;
 +
 +    sendfile              on;
 +    tcp_nopush            on;
 +    tcp_nodelay           on;
 +    keepalive_timeout     300;
 +    proxy_connect_timeout 300;
 +    proxy_send_timeout    300;
 +    proxy_read_timeout    300;
 +    send_timeout          300;
 +
 +    types_hash_max_size 2048;
 +
 +    include             /etc/nginx/mime.types;
 +    default_type        application/octet-stream;
 +
 +    # Load modular configuration files from the /etc/nginx/conf.d directory.
 +    # See http://nginx.org/en/docs/ngx_core_module.html#include
 +    # for more information.
 +    include /etc/nginx/conf.d/*.conf;
 +
 +    upstream orchestrators {
 +       server 10.91.140.28:8079;
 +       server 10.91.140.181:8079;
 +    }
 +
 +    server {
 +        server_name  solidmmwrdflb.errigal.com;
 +        #root         /usr/share/nginx/html;
 +        # Load configuration files for the default server block.
 +        include /etc/nginx/default.d/*.conf;
 +
 +        location / {
 +                    proxy_pass http://orchestrators;
 +        }
 +        
 +        location /ws {
 +            proxy_pass http://orchestrators;
 +            proxy_http_version 1.1;
 +            proxy_set_header Upgrade $http_upgrade;
 +            proxy_set_header Connection "Upgrade";
 +            proxy_set_header Host $host;
 +        }
 +
 +        error_page 404 /404.html;
 +            location = /40x.html {
 +        }
 +
 +        error_page 500 502 503 504 /50x.html;
 +            location = /50x.html {
 +        }
 +    listen 443 ssl;
 +    ssl_certificate /etc/nginx/solidmmwrdflb.errigal.com/sitecert.fullchain.pem;
 +    ssl_certificate_key /etc/nginx/solidmmwrdflb.errigal.com/private-key.pem;
 +}
 +
 +server {
 +    if ($host = solidmmwrdflb.errigal.com) {
 +        return 301 https://$host$request_uri;
 +    }
 +
 +
 +        server_name  solidmmwrdflb.errigal.com;
 +    listen 80;
 +    return 404;
 +
 +
 +}
 +}
 +</code>
 +
 +Next you need to run this command to allow connectivity
 +<code>
 +sudo setsebool -P httpd_can_network_connect 1
 +sudo service nginx restart
 +</code>
 + 
 +====== Mysql8 ======
 +
 +
 +make sure the data dir is pointing to the bigger mounted volume
 +
 +https://dev.mysql.com/doc/refman/8.0/en/linux-installation-yum-repo.html
 +
 +<code>
 +ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-mysql8.yml   
 +</code>
 +
 +you need minimum three schemas: orchestrator, tracker, <name_of_cust_in_tracker>, <name_of_cust_in_tracker> schema and connection details must be mapped in tracker.customer_data_source
 +
 +====== ETCD and MQTT ======
 +
 +<code>
 +ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-etcd.yml 
 +ansible-playbook -i ../env-configuration/solidmmw/hosts.ini --diff --vault-id @prompt setup-mqtt.yml  
 +</code>
 +
 +In order for the ETCDCTL command line tool to work you will need to execute this command on the MDC loadbalancer, oat1 and oat2 servers.
 +
 +<code>
 +export ETCDCTL_ENDPOINTS=http://$server_IP:2379
 +</code>
 +
 +====== Deploy Orchestrator & Tracker ======
 +
 +** Example Jar Deployment Ansible Command **
 +
 +<code>
 +ansible-playbook -i ../env-configuration/towerqa/hosts --diff --vault-id @prompt redeploy-orchestrator.yml --extra-vars "jenkins_build_name='rdf-orchestrator-feature-branches/job/master' jenkins_job_build_num=330"
 +
 +ansible-playbook -i ../env-configuration/towerqa/hosts --diff --vault-id @prompt redeploy-tracker.yml --extra-vars "jenkins_build_name='rdf-tracker-feature-branches/job/master' jenkins_job_build_num=107"
 +
 +</code>
 +
 +Generating Access Token:
 +<code>
 +curl --location --request GET 'https://<orchestratorlb>/api/v1/admin/agent/token/generate?agentUsername=<agent_username>' \
 +--header 'Authorization: Bearer <TOKEN>' \
 +--header 'Content-Type: application/json'
 +
 +
 +where do you get the above token in the first place?
 +Login to the orchestrator UI in your browser, look in the cookies section of your browser and you should see it,
 +else you could also use basic auth instead of using the admin token.
 +
 +</code>
 +
 +This access token should be added to the env configuration for the orchestrator.
 +Also the snmp_manager.orchestrator_access_credentials table should be updated with this token.
 +
 +====== Deploy Agent ======
 +
 +Agent deployment KLA example
 +
 +  - Go to https://klardflb.errigal.com/rdf_public/dashboard/layout.html#/login in your browser and log into the orchestrator UI. Credentials in password safe.
 +  - Navigate to Agent Versions
 +  - For the new Agent version populate with this info.
 +    - Version - 1.8.3
 +    - Location - https://s3.amazonaws.com/rdf.agent.downloads/v1/rdf_agent_master.772.jar
 +    - Hash - can copy another versions hash
 +  - Navigate to Customers - KLA_FORD - kla_ford_crash_barrier
 +  - Fill in version 1.8.3 into the current version field and save. That’s it!
 +  - Refresh after 5 mins to check if the reportedVersion matches the current version.
 +
 +
 +====== Monitoring ======
 +
 +**Note:** Do not forget to update the monitoring config for whatever environment you are deploying to https://bitbucket.org/errigal/prometheus-monitoring-config 
 +
 +Monitoring rules to be updated for the following
 +<code>
 +Orchestrator, Tracker
 +RabbitMQ
 +ElasticSearch + Kibana
 +MySQL8
 +etcd 
 +MQTT 
 +</code>
 +
 +====== Backups ======
 +This may vary from env to env. and upto us on how we configure
 +ElasticSearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html
 +
 +MySQL8 - same as standard MySQL5 backup (again varies from env to env)
 +
 +
 +mini walkthrough: https://drive.google.com/file/d/1VO5aL1n6JFVZbgr1y-3rtA0UOkn6LJk7/view?usp=sharing 
 +The walkthrough video might be horribly out of date.

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki