Recently I was helping a customer who was having some SSL issues. They were on V6R1 and all of a sudden the SSL APIs used by our application started reporting a return code of -99 which is SSL_ERROR_UNKNOWN. After a long bout with IBM support trying to get to someone that could help, they were offered a couple of PTFs to install that should fix the problem (that has yet to be seen).
During the testing we noticed that the Intermediate CA certificate was expired and assumed that was the problem. Using openSSL and a series of documentation we have put together for SSL we retrieved the latest certificate and installed the new Intermediate CA which didn't expire until the end of 2015.
We still had the same issue.
So we then removed the Intermediate CA as well as the Root CA from the *SYSTEM store. Now we were back to a more familiar return code error of -23 which means NOT_TRUSTED_ROOT. Installing the Root Certificate again resulted in the return code of -99 again. We hadn't even installed the Intermediate CA yet! What was going on?
Some SSL History
Back in the V5Rx days (and even some V6Rx and V7Rx systems) when a server used a certificate with a certificate chain I found that you were required to import all of the CAs in the chain, the Root CA as well as any Intermediate CAs. If not, the return code of -23 (Root not trusted) would be issued.
But, in this case it seemed that the IBM i OS was ignoring any Intermediate CAs and only relying on the Root CA for Trust. Is this right? I don't know... in my searching I've found that it seems to be up to each application whether or not to trust just the Root CA, or the Root CA and any Intermediate CAs.
I did find that Apple seems to rely on the entire certificate chain. On April 4th, 2015 Gmail's Intermediate CA expired (without them knowing... again...) causing issues on some devices.
This isn't the first time Google has let an Intermediate CA expire...this happened a couple of years ago as well. It caused issues with many email clients until they finally issued a new certificate with a new Intermediate CA that wasn't expired. Then, a year or so later (I set up a reminder in my calendar to let me know when it expired again) it was going to happen again. I started trying to get in contact with someone at Google to report it. When I finally got the right person they said "Oh wow! Thanks for letting us know!" and issued a new certificate.
Which SSL Implementation is Correct?
So, we know that some implementations of SSL do check the entire Certificate Chain. Others don't. Are they both working correctly?
To do some more testing I set up a V7R2 partition using IBM's Virtual Loaner Program. I removed all CAs from the *SYSTEM store and tested my application that used SSL. Yep, RC -23... not trusted. That's as expected.
Then I installed the Root CA. Things worked fine even though there was an Intermediate CA in the Certificate Chain.
I then removed the Root CA and tried to import the Intermediate CA. I was issued an error (as expected) that told me I couldn't import the Intermediate CA without first importing the Root CA.
So, is this happening because of how IBM is now coding their SSL routines?
Either way, it's inconsistent and if the Intermediate Certificates are suddenly started to be validated again I can see a few SSL application starting to fail.
IBM i (AS/400) and the Wonderful World that is Secure Sockets Layer (SSL) - A Rant
The State of IBM i Support - Isn't What it Used to Be.... or Just Me?