TLS/SSL for applications : Right and Wrong ways to fix it.

Many applications use TLS/SSL to connect their components together. During development, unless you have lots of time and/or money you won’t have a signed certificate. Even if you have a free CA server, the chances are, your development instance was thrown up from Vagrant or a fresh AWS instance that doesn’t trust your CA. (Unless you work for Verisign or Symantec or …)

If you have a new instance and want to import your “company CA” then you’re probably going to want to follow some instructions like this as well.

So, your application is failing to connect to a server because it won’t accept the server’s certificate, how can you fix it? There are 2 main choices here.

Disable SSL

One option is to disable either SSL or certificate validation. That sounds like the easy option, it removes a whole layer of pain, no worrying about SSL any more. Unfortunately, the mechanism to do that varies hugely depending on your application.

You may simply be able to connect to a port that offers an insecure version of what you want. LDAP (389) instead of LDAPS (636) or HTTP (80) instead of HTTPS (443) etc…

If you’re using curl for testing an API then there is an option (“-k” or “–insecure”) to make curl ignore certificate issues.

If you’re using Java, you may have some luck with setting options like

However, disabling SSL makes things insecure, so don’t do it on live! And please don’t ask for instructions on how to do it (unless you want to pay me a £1K day rate for figuring things out). If you can’t figure out yourself how to go insecure, you probably shouldn’t be doing it.

Import the certificate

Importing the certificate is a much better way to work, and it is normally acceptable to use it in production too. It does still ensure the server is the one you think it is (no MITM attack) and will ensure your traffic is encrypted (as long as the server doesn’t allow the NULL ciphers).

There are plenty of ways to import a certificate. Some better than others.

The bad ways

You can create a local keystore and import the certificate to it. Then tell your application to use that keystore. If you can tell the application to use ONLY your keystore, this can be a very secure solution, because it will only allow your app to contact the one certificate it has. But generally, this causes problems in future when you do need to connect to other servers or other people have to work on your app and can’t remember where your keystore is, or what the password is etc… Or you have other apps that need to connect to the same server, they all need a keystore too.

If you don’t want to create a local keystore (for example you have multiple processes talking to the same server) then you can directly modify the system keystore.

This is a very bad thing to do. However, there are lots (and I mean “most of them”) of web sites that recommend this solution. However, it is doomed to failure.

For Java command will look something like :

keytool  -importcert -keystore /etc/pki/java/cacerts -storepass changeit -file server.crt

For other OS commands (like curl) you can add the cert to /etc/pki/tls/certs/ca-bundle.crt

Seems reasonable doesn’t it. You’ve added the certificate to the system’s keystore and any application can use it. EXCEPT, what you’ve done is mess with a file that is managed by your OS’s package manager. Next time you do an update, you will likely find your changes are destroyed. If the updates don’t involve a reboot, then your process may continue to run for another few years (you do have uptime of over a year too, don’t you?) and when it next restarts, someone will get a 4AM callout because something isn’t working and nobody can quite figure out why.

The good way

Someone realised that the PKI directory may need things adding to it. So, in /etc/pki/ca-trust sub-directories you can add certificate files for servers you want to connect to, and run the command :

update-ca-trust extract

when you’re ready to import the new files into your system keystores. And, when the OS updates and resets the system keystores, it does that command for you.

I suggest you have a look at the readme files you’ll find in the ca-trust directory and the man page for update-ca-trust.

How do I know what to add?

If you’re trying to connect to some remote server and your application is just throwing loads of weird java errors about PKIX path building or even just quietly failing, how do you know what to add to the ca-trust directory?

If you have access to the server and it’s config, you can likely search around and find a “servername.crt” file somewhere (often in the /etc/pki tree somewhere). But often, the server is outside of your control and the person who has control has no idea what you are talking about.

openssl is your friend here. Unfortunately, I recently described openssl as the Swiss Army Knife of ssl. You want to get a stone out of horse’s hoof, but first you open the knife, then cut your finger on the blade, then trap your other finger in a sharp thing that has no discernible purpose. Here is the simple command you need :

openssl s_client -connect {server}:{port} -starttls {protocol}

Where {server} is the server you’re connecting to, {port} is the port etc… That should give you something like :

openssl s_client -connect localhost:5222 -starttls xmpp
depth=0 C = --, ST = SomeState, L = SomeCity, O = SomeOrganization, OU = SomeOrganizationalUnit, CN = server, emailAddress = root@server
verify return:1
Certificate chain
 0 s:/C=--/ST=SomeState/L=SomeCity/O=SomeOrganization/OU=SomeOrganizationalUnit/CN=server/emailAddress=root@server
Server certificate

You just need to take the lines between BEGIN and END (including the BEGIN and END lines), save them to a file in the right place and run the update-ca-trust command.