As most know, BVSTools offers email solutions for the IBM i. We offer solutions that work with virtually any SMTP server as well as the APIs available from Google and Microsoft. All of these servers and services use SSL/TLS, which on the IBM i can mean a little extra work importing Certificate Authorities (CAs). This needs to be done so that we don't get the SSL_ERROR_NOT_TRUSTED_ROOT (-23) return code and our applications will play nice with others that require SSL/TLS.
The IBM i seems to have the most difficult way trusting certificates, but we have not only put together documentation for that, but also are happy to personally help customers, especially with the task of exporting CAs from an SSL certificate so they can be imported into the IBM i using Digital Certificate Manager (DCM).
The frustrating journey that started late last week involved what I can only surmise was an update to the SSL certificates on Microsoft's Office 365 SMTP and RESTful email servers.
When this happens it's normally as simple as using OpenSSL to connect to the server, retrieve their Certificate (or CAs) and then import the CAs into the *SYSTEM store to establish a "trust" with those servers and their SSL certificates.
But I noticed the last few days that customers using both the Office 365 SMTP server as well as the Office 365 RESTful email APIs were receiving the SSL_ERROR_NOT_TRUSTED_ROOT error. But it was happening randomly, and not all the time. And for both of the servers.
I started playing around and was able (thankfully) to get both the SSL certificates that the Office 365 RESTful email API was presenting to it's clients. I exported the CAs from each certificate for a total of 4 CAs and had customers install them. Things were looking good and a crisis was averted... or was it?
Not but a few minutes later more support emails started coming in with similar errors. Upon closer inspection I noticed these were customers who were using the Office 365 SMTP server instead of the RESTful API for sending email from their IBM i. I knew that the RESTful server had been presenting at least 2 different SSL certificates randomly, so I figured this had to be the same problem.
I tried and tried but just couldn't get openSSL to connect to an SMTP server so I could get the additional CAs. So I decided to go hard core brute force and create an application that would connect to Microsoft's Office 365 SMTP server every 5 seconds to retrieve the SSL certificate until I found one that was different.
Now, keep in mind I had asked for support for this at Stack Overlflow, Microsoft's Office 365 Developers Forums as well as Microsoft's MSDN Office Support Forums with little to no response. I even had one customer call Microsoft support and explain the issue. "Sorry, we can't help with that." was the response. I know Microsoft is a big company, but you would think these folks that are telling you they can't help or where else to post your problem could stand up from their cubicle and say "Hey, who can help with this?" (Yes, I realize this is unrealistic, but there has to be some way to find the proper support personnel.)
My last resort was what I named my Brute Force SSL Server Requesti-nator (tm) application and process. It involved the use of DOS batch files, OpenSSL calls, and finally, a quick program using RPG and the IFS.
While Felicity Smoak and Cisco Ramon (Vibe) would have been less than impressed and probably could have done it all in 2 seconds with their hardware, I have a feeling that Roy and Moss from Reynholm Industries may have been slightly amused. And because those crazy British IT boys make me laugh to no end, I value their opinion more. (Please note, I do not want any Arrow or Flash spoilers! My wife and I are only on season 2 of each!)
First I needed to make a script that would run and OpenSSL command to try and retrieve the elusive 2nd SSL certificate from Microsoft's Office 365 SMTP server. So I made a batch file as follows named openssl.bat:
for /f "skip=1" %%x in ('wmic os get localdatetime') do if not defined mydate set mydate=%%x cd "c:\my programs\openssl\v1.0.2d\" openssl s_client -connect smtp.office365.com:587 -starttls smtp > "c:\temp\cas\%mydate%certfile.txt"
What this does is create a unique filename using a timestamp. It then calls the s_client feature of OpenSSL to connect to the Office 365 SMTP server and retrieve their SSL Certificate and store it in a unique file.
The results of this call look like the following:
CONNECTED(000001D8) --- Certificate chain 0 s:/C=US/ST=WASHINGTON/L=Redmond/O=Microsoft Corporation/CN=outlook.com i:/C=US/O=DigiCert Inc/CN=DigiCert Cloud Services CA-1 1 s:/C=US/O=DigiCert Inc/CN=DigiCert Cloud Services CA-1 i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA --- Server certificate -----BEGIN CERTIFICATE----- MIIGKDCCBRCgAwIBAgIQDzMzs7xGVulbI0qZMRNoezANBgkqhkiG9w0BAQsFADBL MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSUwIwYDVQQDExxE aWdpQ2VydCBDbG91ZCBTZXJ2aWNlcyBDQS0xMB4XDTE2MTExNzAwMDAwMFoXDTE3 MTExNzEyMDAwMFowajELMAkGA1UEBhMCVVMxEzARBgNVBAgTCldBU0hJTkdUT04x EDAOBgNVBAcTB1JlZG1vbmQxHjAcBgNVBAoTFU1pY3Jvc29mdCBDb3Jwb3JhdGlv bjEUMBIGA1UEAxMLb3V0bG9vay5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQCqIm5jtfxTLIQuJcxlTTUqJLar2qlYWXlmJyJFmErwtFyWgDgkpWqK C4Hnk9bpoLdYvwQ7typORxXd2r8X6eDx8yoevxaWfo6NF9njWerNonCihVy87w8u WoN3/BGZCJMp/0QuFyVe1MJuOj3R3quyozLvDwtJD92gUs5HJtngMkwJ2pxIkqbT sSMBbcEKczHY5FRejLEnYqKx4u65GE3zwUUffyuwL7bShOPCfARp57Mjvr07Bmng wLiYdKs74j/XBcwe30WxaSs4dUVhaG9LfTuqW/vOJXjZPW/n0axuRvA8nMr/5rKt awWIdQ9/7K9Fwa+Urq6L/I2xMachB32fAgMBAAGjggLnMIIC4zAfBgNVHSMEGDAW gBTdUdCiMXOpc66PtAF+XYxXy5/w9zAdBgNVHQ4EFgQUJsShNYS5psfb493ebQTy BMZV25YwggEGBgNVHREEgf4wgfuCHWNjcy5sb2dpbi5taWNyb3NvZnRvbmxpbmUu Y29tggtvdXRsb29rLmNvbYINKi5vdXRsb29rLmNvbYINb2ZmaWNlMzY1LmNvbYIP Ki5vZmZpY2UzNjUuY29tggoqLmxpdmUuY29tghYqLmludGVybmFsLm91dGxvb2su Y29tghcqLm91dGxvb2sub2ZmaWNlMzY1LmNvbYISb3V0bG9vay5vZmZpY2UuY29t gh1hdHRhY2htZW50Lm91dGxvb2sub2ZmaWNlLm5ldIIgYXR0YWNobWVudC5vdXRs b29rLm9mZmljZXBwZS5uZXSCDCoub2ZmaWNlLmNvbTAOBgNVHQ8BAf8EBAMCBaAw HQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMIGNBgNVHR8EgYUwgYIwP6A9 oDuGOWh0dHA6Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydENsb3VkU2Vydmlj ZXNDQS0xLWcxLmNybDA/oD2gO4Y5aHR0cDovL2NybDQuZGlnaWNlcnQuY29tL0Rp Z2lDZXJ0Q2xvdWRTZXJ2aWNlc0NBLTEtZzEuY3JsMEwGA1UdIARFMEMwNwYJYIZI AYb9bAEBMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LmRpZ2ljZXJ0LmNvbS9D UFMwCAYGZ4EMAQICMHwGCCsGAQUFBwEBBHAwbjAlBggrBgEFBQcwAYYZaHR0cDov L29jc3B4LmRpZ2ljZXJ0LmNvbTBFBggrBgEFBQcwAoY5aHR0cDovL2NhY2VydHMu ZGlnaWNlcnQuY29tL0RpZ2lDZXJ0Q2xvdWRTZXJ2aWNlc0NBLTEuY3J0MAwGA1Ud EwEB/wQCMAAwDQYJKoZIhvcNAQELBQADggEBAEdYMggwa+BCKgVrlnVjubpBNhEl r7kD001Bg1qdtt8lmX7hW/AUHsNZV6CkJqdMIQxwDzQdBvHCVTennu9YYKFMFoz8 wmwvbk2vnQn5KeX7WEUVuUbU9YJ+4A6ixh8In5mLpjWZBT201m3APPD8IWmRd3CM 0fJEzyoEu64XQqoH0TRo2Og2NVWJd7LhnCwpgIUhYH8VKFNjoQtoNxDrKy7db0BC +3js2TXoK9mqVYkGrGtx/rTWoyBS5bWsaqVktU6YHMg+EiLVK7oH0etx6N4N3XKI s1Lv+eMHXGTj7UlG9gKzOHlNQKsNNOZ0eJhEoHFaAWXpKbXZT2lrG1m/hjc= -----END CERTIFICATE----- subject=/C=US/ST=WASHINGTON/L=Redmond/O=Microsoft Corporation/CN=outlook.com issuer=/C=US/O=DigiCert Inc/CN=DigiCert Cloud Services CA-1 --- No client certificate CA names sent Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: RSA+SHA512:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1 Shared Requested Signature Algorithms: RSA+SHA512:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1 Peer signing digest: SHA1 Server Temp Key: ECDH, P-384, 384 bits --- SSL handshake has read 3704 bytes and written 578 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-SHA384 Session-ID: BB440000CE711379208DB25AEEA4443C0D901E6B31575D35E6EC6796D04D1BE1 Session-ID-ctx: Master-Key: 13DB5B2554FF25E08DC793E6C3EF2E4C445598A626423D522630518519D54D83290245C0AF430D835D5A5E17411E59C4 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None Start Time: 1481670275 Timeout : 300 (sec) Verify return code: 20 (unable to get local issuer certificate) ---
The certificate used by the server can be extracted by copying everything between BEGIN CERTIFICATE and END CERTIFICATE (including these tags) into a .cer file.
So the next step was to create a batch file that would loop calling this OpenSSL command. Not to bore you with details, the first thing I did was make openssl.bat an .exe file with a free utility. This is so in another batch file I could use the DOS Start operation so once it was called, control would return to this looping batch file. (I couldn't find a way using DOS to send a Ctl-C or Quit command to the OpenSSL terminal to shut it down)
So, my next DOS batch file, which I called runit.bat, is as follows:
@echo off :start start /min c:\temp\openssl.exe timeout 5 taskkill /f /im openssl.exe goto start
This new batch file starts the openssl.exe I created from the openssl.bat file, rests for 5 seconds, then kills the OpenSSL process. Each time this is done the SSL certificate is retrieved from the Microsoft Office 365 SMTP server and stored on my PC. The hope is that we will get lucky and download the oddball certificate.
At first I didn't want to look like I was causing a DoS attack on Microsoft's server, so I ran it for 15 minutes. I now had a huge list of SSL certificates which I moved to the IFS on my IBM i into a directory named /tmp/cas. I took one of them, that I knew was the SSL certificate that I already had, and place it in the /tmp directory. This file would be the one I would compare all the others against, hoping to find the elusive second SSL certificate.
The quickie program (with a surprising amount of error checking for a quickie program) I created is as follows:
H DFTACTGRP(*NO) BNDDIR('BVSTOOLS') **************************************************************** * Other Prototypes **************************************************************** /COPY QCOPYSRC,P.IFS /COPY QCOPYSRC,P.IFSSTD /COPY QCOPYSRC,P.SNDPM * D #QCmdExc PR ExtPgm('QCMDEXC') D Cmd 32000 Const D CmdLen 15 5 Const * D beginString C CONST('-----BEGIN CERTIFICATE-----') D endString C CONST('-----END CERTIFICATE-----') D dir@ S * D data1 S 66535 D data2 S 66535 D fd1 S 10i 0 D fd2 S 10i 0 D x S 10i 0 D y S 10i 0 D match S 10i 0 D dontMatch S 10i 0 D UD_NAME S 256 D TempFullName S 256 D fileType S LIKE(st_objtype) * D QCmdCmd S 1024 INZ D QCmdLength S 15 5 INZ(%size(QCmdCmd)) * D rc S 10i 0 *--------------------------------------------------------------* /free fd1 = #openStmf('/tmp/originalca.txt'); if (fd1 >= 0); rc = #readStmf(fd1:%addr(data1):%size(data1)); endif; if (rc < 0); exsr $Return; endif; x = %scan(beginString:data1); if (x <= 0); exsr $Return; endif; y = %scan(endString:data1); if (y <= x); exsr $Return; endif; data1 = %subst(data1:x:y-x); #closeStmf(fd1); if (data1 <> ' '); dir@ = opendir('/tmp/cas'); if (dir@ = *NULL); #sndpm('Directory /tmp/css not found.':'D'); EXSR $Return; endif; p_dirent = readdir(dir@); dow (p_dirent <> *NULL); // %subst(d_name:1:d_namelen) = filename UD_NAME = %subst(d_name:1:d_namelen); TempFullName = '/tmp/cas/' + UD_NAME; fileType = #stmf_getAtt(TempFullName:'type'); if (fileType = '*STMF'); fd2 = #openStmf(%trim(TempFullName)); if (fd2 >= 0); rc = #readStmf(fd2:%addr(data2):%size(data2)); if (rc > 0); x = %scan(beginString:data2); if (x > 0); y = %scan(endString:data2); if (y > x); data2 = %subst(data2:x:y-x); if (data1 <> ' ') and (data2 <> ' '); if (data1 = data2); match+=1; else; dontMatch+=1; #SndPM('File ' + %trimr(TempFullName) + ' is DIFFERENT!':'D'); QCmdCmd = 'CPY OBJ(''' + %trim(TempFullName) + ''') TODIR(''/tmp/cas/different'')'; callp(e) #QCmdExc(QCmdCmd:%len(%trimr(QCmdCmd))); endif; //data1 = data2 endif; //data1 <> ' ' and data2 <> ' ' endif; // y > x endif; // x > 0 endif; // rc > 0 #closeStmf(fd2); endif; // fd2 >= 0 endif; // fileType = '*STMF' p_dirent = readdir(dir@); enddo; closedir(dir@); #SndPM('Match:' + %char(match) + ' - Not: ' + %char(dontMatch):'C'); endif; exsr $Return; //**************************************************************** // Return //***************************************************************** BegSr $Return; *INLR = *ON; return; EndSr;
Not the prettiest program, but it sure got the job done.
What it did first was open the certificate file in the /tmp directory and read it in. I then found everything between the BEGIN CERTIFICATE and END CERTIFICATE tags and put that into a variable (data1).
Next, I read through the /tmp/cas directory. This is where the large list of SSL certificates was. For each *STMF in the directory I again grabbed everything between BEGIN CERTIFICATE and END CERTIFICATE and compared it to the data from the source SSL certificate. If they were the same, we incremented one counter. If they were different, we incremented another counter. If a different certificate was found it was copied into the /tmp/cas/different directory.
Finally, at the end of all the files we would display how many matched an how many didn't.
So, at first I did the 200 or so SSL certificates I had first retrieved. Surprisingly the program actually ran in less than a second! But, every SSL certificate was the same.
A few more times I continued this process, letting it grab CA for about 5 minutes and then comparing them and still not finding any differences.
I had thought that maybe the SSL certificates were all done replicating on Microsoft's servers (or that they had fixed the issue). So I contacted a couple customers. One said no problems, the other said yes, it was still happening.
Well, by this time it was 5:30 and I needed to go pick up my daughter from gymnastics. So I fired up the program and let it run while I was gone for a good hour.
When I got back I had over 900 certificates to test against,. I copied them to the IFS and ran my program. I was happy to finally see:
File /tmp/cas/20161213170505.497000-360certfile.txt is DIFFERENT!
File /tmp/cas/20161213170500.510000-360certfile.txt is DIFFERENT!
File /tmp/cas/20161213170455.494000-360certfile.txt is DIFFERENT!
File /tmp/cas/20161213170450.531000-360certfile.txt is DIFFERENT!
File /tmp/cas/20161213170445.561000-360certfile.txt is DIFFERENT!
File /tmp/cas/20161213170440.527000-360certfile.txt is DIFFERENT!
Match:805 - Not: 73
Thank goodness! I looked in the /tmp/cas/different directory and noticed they were all the same. These all had to be the elusive second SSL certificate that customers were randomly getting causing the error. I exported the CAs from it, sent them to my customers and prayed.
That night a few customers reported that they had zero problems since installing the new CAs. And this morning, even more. So, I believe this is crisis averted for now.
What's the moral of this story? I don't know. What I DO know is the next time I have a customer ask which I prefer or recommend when it comes to a cloud mail provider, it will be GMail over Microsoft Office 365 for sure. This won't have much bearing on most Microsoft Shops, but it may help others.
I also know that if this is how Microsoft treats paying customers (ie, ignoring them), that's not something that I, as a small business owner who values his customers, will never want to parrot.