SSL Configuration

Author: Anna Dowling

The following wiki entry will step through the loading of a customers CA cert and bundle cert file into apache and java.

Before you can order an SSL Certificate, you must first generate a CSR (Certificate Signing Request) for your server.

A CSR is an encoded file that provides you with a standardized way to send your public key along with some information that identifies your company and domain name. When you generate a CSR, most server software asks for the following information: common name (i.e. www.example.com), organization name and location (country, state/province, city/town), key type (typically RSA), and key size (2048 bit minimum). You should also set the SAN (subject alternate name) to be the chosen domain name, as this is now required by chrome as a directive. The easiest way to specify this information and keep it consistent is to set up a san.conf file with all of these parameters. (see examples on crown or extenet qa in /etc/httpd/conf/ssl/san.conf)

The command to generate a server key and csr file is as follows:

  * openssl req -new -newkey rsa:2048 -nodes -keyout yourdomain.key -out yourdomain.csr -config san.conf

This will generate a new key so should only be performed on a new server that doesn't have a private key. If you have access to a current private key, you can generate a CSR from that key using:

  * $ openssl req -out yourdomain.csr -key private.key -new

Checking the csr file contains the configuration you set in san.conf (DNS example):

openssl req -noout -text -in yourdomain.csr | grep DNS

Root certificate: A Root SSL certificate is a certificate issued by a trusted certificate authority (CA). In the SSL ecosystem, anyone can generate a signing key and sign a new certificate with that signature. … A trusted certificate authority is an entity that has been entitled to verify that someone is effectively who it declares to be.

Intermediate certificate: Intermediate certificates are used as a stand-in for our root certificate. We use intermediate certificates as a proxy because we must keep our root certificate behind numerous layers of security, ensuring its keys are absolutely inaccessible. However, because the root certificate itself signed the intermediate certificate, the intermediate certificate can be used to sign the SSLs our customers install and maintain the “Chain of Trust.”

1. Server Certificate Chain(bundle) file - CA bundle is a file that contains root and intermediate certificates. The certificate issued for your domain constitutes the certificates’ chain with a CA bundle. (This is provided by the CA authority)
2. Server Certificate file - Often referred to as the sub CA, this is the certificate file containing the certified domain details for use with securing your site. (This is provided by the CA authority)
3. Server Private key - The private key is generated simultaneously with the CSR (certificate signing request), containing the domain name, public key and additional contact information. The CSR is to be sent to the certificate authority for validation and signing immediately after the certificate activation typically. The private key must be kept secret, ideally on the same server the certificate will be installed on. (This is generated on the server)

Pull the master branch of the server configuration repo. Run the deploy playbook to reconfigure the SSL cert for a given environment. The playbook will do all the below steps:

• Load the cert into Java
• Load the cert into Apache
• Restart Apache

ansible-playbook -i ../env-configuration/$ENV/hosts.ini reconfigure-ssl.yml --vault-id @prompt --diff 

There is no need to shut down all applications anymore.

For AWS based environments, an extra step is required. The cert will need to be updated on the loadbalancer instance. This is the case for ATC, QAATC, SCO etc.

• Sign into AWS console
• Navigate to Home Page
• Load Balancers → Listeners → View/Edit certificates ACM
• https://console.aws.amazon.com/acm/home?region=us-east-1#/
  • Selecting certiciate relevant to environment → Actions → Reimport Certificate
• Paste in the contents of the three certificate files from env-configuration (already PEM encoded) into the corresponding fields.
  • Certificate private key
  • Certificate body
  • Certificate chain

In order for the applications to obtain the certificate it must be loaded into javas list of certs via keytool. The bundle and sub cert files should both be loaded at this point. This step needs to be done on each server that is running any applications using java. The cert files should be transferred to all application servers.

If an old certificate was present on the server these should be deleted before loading the new ones to ensure that the correct cert gets used. Any java versions in use by the applications should also be taken into account when loading the certs. We may have multiple versions running apart from jdk1.7.0_80.

Examples of java default certs to not delete:

• GIAG2.crt
• aspmx.l.google.com.cert
• smpt.gmail.com.cert
• smtp-relay.gmail.com.cert

  * /usr/java/jdk1.7.0_80/bin/keytool -delete -alias tomcat -keystore /usr/java/jdk1.7.0_80/jre/lib/security/cacerts
  * /usr/java/jdk1.7.0_80/bin/keytool -delete -alias root -keystore /usr/java/jdk1.7.0_80/jre/lib/security/cacerts

Load in new certs (files mentioned are examples ):

  * /usr/java/jdk1.7.0_80/bin/keytool -import -alias root -keystore /usr/java/jdk1.7.0_80/jre/lib/security/cacerts -trustcacerts -file gd_bundle-g2-g1.crt
  * /usr/java/jdk1.7.0_80/bin/keytool -import -alias tomcat -keystore /usr/java/jdk1.7.0_80/jre/lib/security/cacerts -trustcacerts -file 8568c97197a66a9e.crt

  * /usr/java/jdk1.7.0_80/jre/bin/keytool -list -keystore /usr/java/jdk1.7.0_80/jre/lib/security/cacerts -alias tomcat
  * /usr/java/jdk1.7.0_80/jre/bin/keytool -list -keystore /usr/java/jdk1.7.0_80/jre/lib/security/cacerts -alias root

Transfer the cert files(bundle file, cert and server key) to the load balancer into the following directory:

  * /etc/httpd/conf/ssl

Edit the ssl.conf file located in:

  * /etc/httpd/conf.d

Update the following sections to point to the files that you transferred to the server (files mentioned are examples ):

  * SSLCertificateFile /etc/httpd/conf/ssl/8568c97197a66a9e.crt
  * SSLCertificateKeyFile /etc/httpd/conf/ssl/idmsqa.errigal.com.key
  * SSLCertificateChainFile /etc/httpd/conf/ssl/gd_bundle-g2-g1.crt

• If a new domain name has been provided for a system by the customer, this name should be reflected in the following places before doing a restart:
  • Grails Application Config files
  • cas.properties file
  • Database configuration domains

Applications do not need a restart anymore when using the reconfigure_ssl playbook, only the httpd service needs a restart

1. Shutdown all applications
2. Restart apache service on the load balancer:

  * sudo service httpd restart

1. Restart all applications (This is in order to load the new certs from keystore)

Check for other keystores present on the servers and delete if necessary:

  * cd /
  * sudo find . -name .keystore

The correct keystore should point to the java cacerts file:

  * /usr/java/jdk1.7.0_80/jre/lib/security/cacerts **(depends on java version in use)**

Ensure the correct JAVA_HOME is set to the correct version as this can impact the keystore:

  * readlink -f /usr/bin/java **(check the symlink of /usr/bin/java)**
  * sudo unlink /usr/bin/java **(unlink this dir if incorrect java version appears)**
  * sudo ln -s /usr/java/jdk1.7.0_80/bin/java /usr/bin/java **(set the correct symlink)**

Clear out any unused certs from the java keystore,not including default certs used by java :

  * /usr/java/jdk1.7.0_80/bin/keytool -delete -aliasUnusedCertAlias -keystore /usr/java/jdk1.7.0_80/jre/lib/security/cacerts

Check the ssl error logs on the server:

  * sudo vi /var/log/httpd/ssl_error_log

Check that the domain name and public ip address resolve to the correct cert:

  * openssl s_client -connect 12.39.4.41:443 -showcerts
  * openssl s_client -connect extenetqa.errigal.com:443 -showcerts
  * If either of these evaluate to a previous cert it is possible that the cert is being cached by the firewall and may need to be reset by a sysadmin.

Check that the private key matches the cert using the following tool: https://decoder.link/matcher/

— Anna Dowling 2017/11/15 07:17