How to Resolve Java HTTPS Exceptions
By Pete Freitag
TLDR: Most java HTTPS connection problems can be fixed by updating the JVM. Don't import into cacerts unless you really need to (eg you have an internal CA within your organization). Test other http clients to make sure it is really a java problem.
Before we get into all the details I'll start off by saying that the old advice to import the domain's certificate into cacerts is almost always the wrong way to fix this problem. Just because it may work doesn't mean it is a good solution.
Why is importing into cacerts
usually bad advice?
By importing a certificate into the cacerts
keystore file you are telling java that this certificate is a trusted certificate authority. A certificate authority is allowed to sign certificates for any domain and java may then trust those certificates. Further as trusted CA certs become compromised they are revoked and should be removed from the cacerts file, the entire system of trust gets eroded if this file is not kept up to date.
Test Other HTTP Clients
Before you get too far down the rabbit hole you should make sure that you are indeed dealing with a java problem. Try connecting to the server using other http clients such as a web browser, curl, wget, powershell, etc. Here's a curl example which is installed by default on most linux, and mac operating systems:
curl --verbose https://example.com/
If you are on windows you can open powershell and run:
Invoke-WebRequest -URI https://example.com/
PKIX Path Validation Failed
The PKIX path validation failed exception or sun.security.validator.ValidatorException: PKIX path validation failed
is a pretty common java exception you may get when attempting to connect to a HTTPS server or some other protocol that uses TLS (formerly known as SSL).
Here are some common causes of a PKIX Path Validation Failed exception:
Certificate is Expired
The certificate for the domain you are trying to connect to has expired. There is not much you can unless you operate the domain - the certificate needs to be renewed. Also make sure your server clock has the correct time.
I recently helped someone that was having this issue and it was due to a hosts file entry pointing to an old server, once the certificate expired it started causing a problem, but it was not clear because it worked everywhere except for the server in question.
Invalid Certificate Chain, Missing Intermediate Certificate
It is pretty common for site operators to forget to specify the intermediate certificate when they setup HTTPS. Several years ago Certificate authorities (CA) would sign certificates directly off of their root certificate. These days must CA's use an intermediate certificate, so they sign a sub CA certificate which then signs certificates for their customers. This approach allows the CA to revoke an intermediate certificate if it becomes compromised but they can just generate a new intermediate off the very valuable root certificate.
This site: What is my cert chain? is really good at debugging, and explaining in more detail.
This problem can be frustrating because the site will still work on most browsers, but will fail when you try to connect to it. This is because browsers cache intermediate trusted certificates and trust them for future requests.
You might get unable to find valid certification path to requested target in your exception with this issue.
Domain uses a new Certificate Authority Cert
The Certificate Authority may have a new certificate that is not in the cacerts
file. You will find hundreds of articles online telling you to just import the certificate into Java's cacerts keystore file. The best way to fix this is to update the jvm, when new versions of the jvm are released the cacerts file is often updated with the latest trusted certs. You might be able to grab the cacerts file from the latest jvm and use that if you don't want to update the JVM, but updating the JVM should be something you do frequently to stay up to date with security patches (security updates for the JVM are usually released on a quarterly basis). You can also generate a cacerts file using Mozilla's Certificate Authority List.
A good example of this is servers using Let's Encrypt Certificates. Java, Apache Commons HttpClient, ColdFusion / Lucee (cfhttp), etc are all able to connect to a server using Let's Encrypt but if your JVM is really old it will fail. This is because it is a newer certificate authority and it might not have existed when the version of Java you are using was released. Update Java and you should be good here.
Now there is a small chance that the domain is using a new certificate authority that was not yet imported into the cacerts
file that ships with the latest version of java. In practice this was common 10 or more years ago, but I haven't seen this problem in the past several years. Oracle / OpenJDK have done a good job keeping it current. In this case you might actually need to import a certificate into the cacerts file, but you should import the CA certificate, not the leaf certificate (eg don't import the cert for example.com, import the ca cert for the certificate authority that signed the cert).
Internal Certificates (self signed)
If you are using an internal certificate authority then you should import your internal ca cert into the cacerts file. If you have to connect to a server using a self signed certificate you basically have three options:
- Write a custom
javax.net.ssl.X509TrustManager
to allow the self signed cert - Import the self signed cert into cacerts (not ideal)
- Create a self signed certificate authority cert, and sign your self signed cert with that. You can use something like mkcert for this, but you have to now be really careful about the cacert that you've created - you are now a certificate authority. You might rather consider using something like: AWS Certificate Manager Private Certificate Authority for this, which will help you manage the private keys and add a lot of operational security.
Handshake Exceptions (SSLHandshakeException)
Another common type of HTTPS connection exception is: javax.net.ssl.SSLHandshakeException, this is usually due to an inability for the client (java) and the server to find a SSL / TLS protocol and cipher that they agree on.
Updating Java is usually the best resolution for this type of exception. For example if you are running on Java 1.6 or below TLS 1.2 is not supported. If you are on Java 7 (1.7), TLS 1.2 is supported but not enabled by default. For TLS 1.3 you'll need at least Java 1.8u261 or Java 11+.
You can tell Java which protocols to use by default by setting the java system property https.protocols=TLSv1.2
for example. You may find however that this doesn't always work, some java code or libraries may be coded to use a specific protocol that the server doesn't support (eg TLS 1.0). In that case you need to update the library or code.
For example you might have code like this, which was intended to force TLSv1.2:
SSLContext.getInstance("TLSv1.2");
or
sslSocket.setEnabledProtocols(new String[] {"TLSv1.2"});
or
sslParameters.setProtocols(new String[] {"TLSv1.2"});
In all those cases if a server you were trying to connect to was updated to support only TLSv1.3
then you might get a SSL Handshake Exception: javax.net.ssl.SSLHandshakeException
.
How to Resolve Java HTTPS Exceptions was first published on November 21, 2018.
If you like reading about java, https, tls, or ssl then you might also like:
- Self Signed Certificates in Edge on Windows 2022
- Java versions supporting TLS 1.3
- TLSv1 and TLSv1.1 Disabled by Default in Java
- Development SSL / TLS with CommandBox
Weekly Security Advisories Email
Advisory Week is a new weekly email containing security advisories published by major software vendors (Adobe, Apple, Microsoft, etc).
Comments
https://coldfusion.adobe.com/2019/06/error-calling-cf-via-https-solved-updating-jvm/
I will add a comment there pointing to this newer post of yours.
These articles got me doing more searching and I found a setting in the /bin/java.security file that seemed to fix our problem. We changed the line to 'keystore.type=jks' it was set to PKCS12.
Tim
I press the point because if everyone just did this, I'm wondering what the implications would be.
It is indeed very interesting that you say your problem was that https requests would "coma and go", breaking for "no apparent reason", if by that you mean the same URL in the same CF tag or feature would sometimes work and then sometimes fail, like within minutes with no other change.
I have indeed heard of some other people reporting that, and have not been able to help them solve that (since it's such an odd situation). I have suspected that jvm debug output of the ssl/tls handshake would help, but people are often either unable to do that (a jvm argument, requiring a restart), or the find the debug output to be voluminous and hard to get value from.
So if your proposal here may well really be a solution that "about anyone should do", even if NOT pointing to some keystore other than the built-in lib/security/cacerts that the JVM uses by default, that is VERY interesting to hear.
And I will look forward very much to any more that you, Pete, or others may have to offer about this. Thanks.
This was just an cfhttp (https://production.shippingapis.com). This URL was the only URL that was causing issue. We could replicate it on multiple machines. It wouldn't work periodically, it would stop working until we reboot, restart CF, change JVM versions. But it would always stop working.
The keystore in the /bin is coming back as type JKS so I changed the keystore.type to match. There is a setting for keystore.type.compat which is set to 'true' that looks like it will translate either JKS or PKCS12. This may be where the issue is with CF2021.
Again, I am not pointing to any other keystore than the on listed in the 'Java Virtual Machine Path'
If you want, tell me how to get the JVM debug you are looking for and I will get you that info.
So has it really been that changing that keystore.type to jks was alone the solution for you? If so, that will be very interesting to hear, as I will explain. If it has not, I have another idea, in addition to the JVM debug logging option I'd mentioned.
First, let's note for any following along that there has been variation over time (and over JVM versions) regarding that keystore.type (and its default) as found in that java.security file. And the availability of that keystore.type.compat option seems to have been added in about 1.8.0_60. (Note also that the location of that java.security file has also changed. In Java 8 it was indeed jre/lib/security, but in Java 11 it's now jre/conf/security).
FWIW, CF2021 and 2018 come with Java 11 by default, and it's had those defaulted to pkcs12 and true since 11.0.1. And I see that for CF2016, and the JVM it came with by default (1.8.0_112), it has the default keystore.type set to jks and the compat=true, while CF11's default jvm (1.0.0_25) also had the keystore.type default of jks and no keystore.type.compat.
Of course, if anyone changed the jvm that CF uses (which is indeed wise, as is the focus of this very post), then they need to pay attention to what version they are using, of course, and what their settings are.
But second, you say that you had keystore.type.compat set to true, and in that case it seems it shouldn't have mattered what the keystore.type was, given how that compat works (from the comments in that java.security file, "When set to 'true', both JKS and PKCS12 keystore types support loading keystore files in either JKS or PKCS12 format.")
So it's just very interesting to hear that changing it made any difference at all. If that was indeed really the solution for you, it seems there is more to this than is indicated by the comments in that file (and the java resources I have found so far).
Now, moving on, what if you might confirm that changing it in fact did not really help (and it was just a matter of time before it started failing again)?
I will say, third, if it is still happening for you and since you refer to how the problem had been coming and going, that is itself a key point. And I would wonder if it could be instead that somehow the ip address for which that domain is responding may be changing over time--but the JVM may be caching the DNS lookup (even if only breifly), and thus having it wrong. What's not clear is if that would lead to what seems a java security error, rather than just a general failure to connect. Pete may be able to confirm, if he's observed anything about this.
And finally, if you or anyone ever needs to do debugging of the ssl/tls handshake that goes on between CF (java) and any outgoing https calls (whether from cfhttp or otherwise), that can be enabled via a jvm arg like this: -Djavax.net.debug=all
There are other options besides all. One could set that (carefully) in the CF jvm.config file (in cfusion/bin, or your instancename/bin), but do be sure to save a copy in case CF won't start due to a mistake you may make. Then this https handshake debug loggind will appear either in the coldfusion-out.log (if you run CF as a service) or on the console (if you start CF from the console).
Hope that's heplful.
NOTE: CFX_EXEC is another product and it performs lightning fast DNS lookups that honor TTL. It can also run processes using specified Windows accounts (versus the account that the service was started with.)