UPDATE: For updated code please see The Brute Force SSL Server Request-inator (tm) - Version 2.0

As most know, BVSTools offers email solutions for the IBM i.  We offer solutions that work with virtually any SMTP server as well as the APIs available from Google and Microsoft.  All of these servers and services use SSL/TLS, which on the IBM i can mean a little extra work importing Certificate Authorities (CAs).  This needs to be done so that we don't get the SSL_ERROR_NOT_TRUSTED_ROOT (-23) return code and our applications will play nice with others that require SSL/TLS.

The IBM i seems to have the most difficult way trusting certificates, but we have not only put together documentation for that, but also are happy to personally help customers, especially with the task of exporting CAs from an SSL certificate so they can be imported into the IBM i using Digital Certificate Manager (DCM).

The Frustrating Microsoft Office 365 SSL Journey 

The frustrating journey that started late last week involved what I can only surmise was an update to the SSL certificates on Microsoft's Office 365 SMTP and RESTful email servers.  

When this happens it's normally as simple as using OpenSSL to connect to the server, retrieve their Certificate (or CAs) and then import the CAs into the *SYSTEM store to establish a "trust" with those servers and their SSL certificates.

But I noticed the last few days that customers using both the Office 365 SMTP server as well as the Office 365 RESTful email APIs were receiving the SSL_ERROR_NOT_TRUSTED_ROOT error.  But it was happening randomly, and not all the time.  And for both of the servers.

I started playing around and was able (thankfully) to get both the SSL certificates that the Office 365 RESTful email API was presenting to it's clients.  I exported the CAs from each certificate for a total of 4 CAs and had customers install them.  Things were looking good and a crisis was averted... or was it?

Not but a few minutes later more support emails started coming in with similar errors.  Upon closer inspection I noticed these were customers who were using the Office 365 SMTP server instead of the RESTful API for sending email from their IBM i.  I knew that the RESTful server had been presenting at least 2 different SSL certificates randomly, so I figured this had to be the same problem.

I tried and tried but just couldn't get openSSL to connect to an SMTP server so I could get the additional CAs.  So I decided to go hard core brute force and create an application that would connect to Microsoft's Office 365 SMTP server every 5 seconds to retrieve the SSL certificate until I found one that was different.

Now, keep in mind I had asked for support for this at Stack Overlflow, Microsoft's Office 365 Developers Forums as well as Microsoft's MSDN Office Support Forums with little to no response.  I even had one customer call Microsoft support and explain the issue.  "Sorry, we can't help with that." was the response.  I know Microsoft is a big company, but you would think these folks that are telling you they can't help or where else to post your problem could stand up from their cubicle and say "Hey, who can help with this?"  (Yes, I realize this is unrealistic, but there has to be some way to find the proper support personnel.)

The Solution - The Brute Force SSL Server Request-inator (tm)

My last resort was what I named my Brute Force SSL Server Requesti-nator (tm) application and process.  It involved the use of DOS batch files, OpenSSL calls, and finally, a quick program using RPG and the IFS.  

While Felicity Smoak and Cisco Ramon (Vibe) would have been less than impressed and probably could have done it all in 2 seconds with their hardware, I have a feeling that Roy and Moss from Reynholm Industries may have been slightly amused.  And because those crazy British IT boys make me laugh to no end, I value their opinion more.  (Please note, I do not want any Arrow or Flash spoilers!  My wife and I are only on season 2 of each!)

First I needed to make a script that would run and OpenSSL command to try and retrieve the elusive 2nd SSL certificate from Microsoft's Office 365 SMTP server.  So I made a batch file as follows named openssl.bat:

for /f "skip=1" %%x in ('wmic os get localdatetime') do if not defined mydate set mydate=%%x

cd "c:\my programs\openssl\v1.0.2d\"
openssl s_client -connect smtp.office365.com:587 -starttls smtp > "c:\temp\cas\%mydate%certfile.txt" 

What this does is create a unique filename using a timestamp.  It then calls the s_client feature of OpenSSL to connect to the Office 365 SMTP server and retrieve their SSL Certificate and store it in a unique file.

The results of this call look like the following:

CONNECTED(000001D8)
---
Certificate chain
 0 s:/C=US/ST=WASHINGTON/L=Redmond/O=Microsoft Corporation/CN=outlook.com
   i:/C=US/O=DigiCert Inc/CN=DigiCert Cloud Services CA-1
 1 s:/C=US/O=DigiCert Inc/CN=DigiCert Cloud Services CA-1
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGKDCCBRCgAwIBAgIQDzMzs7xGVulbI0qZMRNoezANBgkqhkiG9w0BAQsFADBL
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSUwIwYDVQQDExxE
aWdpQ2VydCBDbG91ZCBTZXJ2aWNlcyBDQS0xMB4XDTE2MTExNzAwMDAwMFoXDTE3
MTExNzEyMDAwMFowajELMAkGA1UEBhMCVVMxEzARBgNVBAgTCldBU0hJTkdUT04x
EDAOBgNVBAcTB1JlZG1vbmQxHjAcBgNVBAoTFU1pY3Jvc29mdCBDb3Jwb3JhdGlv
bjEUMBIGA1UEAxMLb3V0bG9vay5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCqIm5jtfxTLIQuJcxlTTUqJLar2qlYWXlmJyJFmErwtFyWgDgkpWqK
C4Hnk9bpoLdYvwQ7typORxXd2r8X6eDx8yoevxaWfo6NF9njWerNonCihVy87w8u
WoN3/BGZCJMp/0QuFyVe1MJuOj3R3quyozLvDwtJD92gUs5HJtngMkwJ2pxIkqbT
sSMBbcEKczHY5FRejLEnYqKx4u65GE3zwUUffyuwL7bShOPCfARp57Mjvr07Bmng
wLiYdKs74j/XBcwe30WxaSs4dUVhaG9LfTuqW/vOJXjZPW/n0axuRvA8nMr/5rKt
awWIdQ9/7K9Fwa+Urq6L/I2xMachB32fAgMBAAGjggLnMIIC4zAfBgNVHSMEGDAW
gBTdUdCiMXOpc66PtAF+XYxXy5/w9zAdBgNVHQ4EFgQUJsShNYS5psfb493ebQTy
BMZV25YwggEGBgNVHREEgf4wgfuCHWNjcy5sb2dpbi5taWNyb3NvZnRvbmxpbmUu
Y29tggtvdXRsb29rLmNvbYINKi5vdXRsb29rLmNvbYINb2ZmaWNlMzY1LmNvbYIP
Ki5vZmZpY2UzNjUuY29tggoqLmxpdmUuY29tghYqLmludGVybmFsLm91dGxvb2su
Y29tghcqLm91dGxvb2sub2ZmaWNlMzY1LmNvbYISb3V0bG9vay5vZmZpY2UuY29t
gh1hdHRhY2htZW50Lm91dGxvb2sub2ZmaWNlLm5ldIIgYXR0YWNobWVudC5vdXRs
b29rLm9mZmljZXBwZS5uZXSCDCoub2ZmaWNlLmNvbTAOBgNVHQ8BAf8EBAMCBaAw
HQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMIGNBgNVHR8EgYUwgYIwP6A9
oDuGOWh0dHA6Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydENsb3VkU2Vydmlj
ZXNDQS0xLWcxLmNybDA/oD2gO4Y5aHR0cDovL2NybDQuZGlnaWNlcnQuY29tL0Rp
Z2lDZXJ0Q2xvdWRTZXJ2aWNlc0NBLTEtZzEuY3JsMEwGA1UdIARFMEMwNwYJYIZI
AYb9bAEBMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LmRpZ2ljZXJ0LmNvbS9D
UFMwCAYGZ4EMAQICMHwGCCsGAQUFBwEBBHAwbjAlBggrBgEFBQcwAYYZaHR0cDov
L29jc3B4LmRpZ2ljZXJ0LmNvbTBFBggrBgEFBQcwAoY5aHR0cDovL2NhY2VydHMu
ZGlnaWNlcnQuY29tL0RpZ2lDZXJ0Q2xvdWRTZXJ2aWNlc0NBLTEuY3J0MAwGA1Ud
EwEB/wQCMAAwDQYJKoZIhvcNAQELBQADggEBAEdYMggwa+BCKgVrlnVjubpBNhEl
r7kD001Bg1qdtt8lmX7hW/AUHsNZV6CkJqdMIQxwDzQdBvHCVTennu9YYKFMFoz8
wmwvbk2vnQn5KeX7WEUVuUbU9YJ+4A6ixh8In5mLpjWZBT201m3APPD8IWmRd3CM
0fJEzyoEu64XQqoH0TRo2Og2NVWJd7LhnCwpgIUhYH8VKFNjoQtoNxDrKy7db0BC
+3js2TXoK9mqVYkGrGtx/rTWoyBS5bWsaqVktU6YHMg+EiLVK7oH0etx6N4N3XKI
s1Lv+eMHXGTj7UlG9gKzOHlNQKsNNOZ0eJhEoHFaAWXpKbXZT2lrG1m/hjc=
-----END CERTIFICATE-----
subject=/C=US/ST=WASHINGTON/L=Redmond/O=Microsoft Corporation/CN=outlook.com
issuer=/C=US/O=DigiCert Inc/CN=DigiCert Cloud Services CA-1
---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA512:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA512:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1
Peer signing digest: SHA1
Server Temp Key: ECDH, P-384, 384 bits
---
SSL handshake has read 3704 bytes and written 578 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-SHA384
    Session-ID: BB440000CE711379208DB25AEEA4443C0D901E6B31575D35E6EC6796D04D1BE1
    Session-ID-ctx: 
    Master-Key: 13DB5B2554FF25E08DC793E6C3EF2E4C445598A626423D522630518519D54D83290245C0AF430D835D5A5E17411E59C4
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1481670275
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

The certificate used by the server can be extracted by copying everything between BEGIN CERTIFICATE and END CERTIFICATE (including these tags) into a .cer file.

So the next step was to create a batch file that would loop calling this OpenSSL command.  Not to bore you with details, the first thing I did was make openssl.bat an .exe file with a free utility.  This is so in another batch file I could use the DOS Start operation so once it was called, control would return to this looping batch file.  (I couldn't find a way using DOS to send a Ctl-C or Quit command to the OpenSSL terminal to shut it down)

So, my next DOS batch file, which I called runit.bat, is as follows:

@echo off
:start
start /min c:\temp\openssl.exe
timeout 5
taskkill /f /im openssl.exe
goto start

This new batch file starts the openssl.exe I created from the openssl.bat file, rests for 5 seconds, then kills the OpenSSL process.  Each time this is done the SSL certificate is retrieved from the Microsoft Office 365 SMTP server and stored on my PC.  The hope is that we will get lucky and download the oddball certificate.

At first I didn't want to look like I was causing a DoS attack on Microsoft's server, so I ran it for 15 minutes.  I now had a huge list of SSL certificates which I moved to the IFS on my IBM i into a directory named /tmp/cas.  I took one of them, that I knew was the SSL certificate that I already had, and place it in the /tmp directory.  This file would be the one I would compare all the others against, hoping to find the elusive second SSL certificate.

The quickie program (with a surprising amount of error checking for a quickie program) I created is as follows:

     H DFTACTGRP(*NO) BNDDIR('BVSTOOLS')
      ****************************************************************
      * Other Prototypes
      ****************************************************************
      /COPY QCOPYSRC,P.IFS
      /COPY QCOPYSRC,P.IFSSTD
      /COPY QCOPYSRC,P.SNDPM
      *
     D #QCmdExc        PR                  ExtPgm('QCMDEXC')
     D  Cmd                       32000    Const
     D  CmdLen                       15  5 Const
      *
     D beginString     C                   CONST('-----BEGIN CERTIFICATE-----')
     D endString       C                   CONST('-----END CERTIFICATE-----')
     D dir@            S               *
     D data1           S          66535
     D data2           S          66535
     D fd1             S             10i 0
     D fd2             S             10i 0
     D x               S             10i 0
     D y               S             10i 0
     D match           S             10i 0
     D dontMatch       S             10i 0
     D UD_NAME         S            256
     D TempFullName    S            256
     D fileType        S                   LIKE(st_objtype)
      *
     D QCmdCmd         S           1024    INZ
     D QCmdLength      S             15  5 INZ(%size(QCmdCmd))
      *
     D rc              S             10i 0
      *--------------------------------------------------------------*
      /free

       fd1 = #openStmf('/tmp/originalca.txt');

       if (fd1 >= 0);
         rc = #readStmf(fd1:%addr(data1):%size(data1));
       endif;

       if (rc < 0);
         exsr $Return;
       endif;

       x = %scan(beginString:data1);

       if (x <=  0);
         exsr $Return;
       endif;

       y = %scan(endString:data1);

       if (y <= x);
         exsr $Return;
       endif;

       data1 = %subst(data1:x:y-x);

       #closeStmf(fd1);

       if (data1 <> ' ');
         dir@ = opendir('/tmp/cas');

         if (dir@ = *NULL);
           #sndpm('Directory /tmp/css not found.':'D');
           EXSR $Return;
         endif;

         p_dirent = readdir(dir@);

         dow (p_dirent <> *NULL);
           // %subst(d_name:1:d_namelen) = filename
           UD_NAME = %subst(d_name:1:d_namelen);

           TempFullName = '/tmp/cas/' + UD_NAME;
           fileType = #stmf_getAtt(TempFullName:'type');

           if  (fileType = '*STMF');
             fd2 = #openStmf(%trim(TempFullName));

             if (fd2 >= 0);
               rc = #readStmf(fd2:%addr(data2):%size(data2));

               if (rc > 0);
                 x = %scan(beginString:data2);

                 if (x > 0);
                   y = %scan(endString:data2);

                   if (y > x);
                     data2 = %subst(data2:x:y-x);

                     if (data1 <> ' ') and (data2 <> ' ');

                       if (data1 = data2);
                         match+=1;
                       else;
                         dontMatch+=1;
                         #SndPM('File ' +  %trimr(TempFullName) +
                                ' is DIFFERENT!':'D');
                         QCmdCmd = 'CPY OBJ(''' +
                                   %trim(TempFullName) +
                                   ''') TODIR(''/tmp/cas/different'')';
                         callp(e) #QCmdExc(QCmdCmd:%len(%trimr(QCmdCmd)));
                       endif; //data1 = data2

                     endif; //data1 <> ' ' and data2 <> ' '

                   endif; // y > x

                 endif; // x > 0

               endif; // rc > 0

               #closeStmf(fd2);
             endif; // fd2 >= 0

           endif; // fileType = '*STMF'

           p_dirent = readdir(dir@);
         enddo;

         closedir(dir@);
         #SndPM('Match:' + %char(match) + ' - Not: ' + %char(dontMatch):'C');
       endif;

       exsr $Return;

       //****************************************************************
       // Return
       //*****************************************************************
       BegSr $Return;

         *INLR = *ON;
         return;

       EndSr;

Not the prettiest program, but it sure got the job done.

What it did first was open the certificate file in the /tmp directory and read it in.  I then found everything between the BEGIN CERTIFICATE and END CERTIFICATE tags and put that into a variable (data1).

Next, I read through the /tmp/cas directory.  This is where the large list of SSL certificates was.  For each *STMF in the directory I again grabbed everything between BEGIN CERTIFICATE and END CERTIFICATE and compared it to the data from the source SSL certificate.  If they were the same, we incremented one counter.  If they were different, we incremented another counter.  If a different certificate was found it was copied into the /tmp/cas/different directory.

Finally, at the end of all the files we would display how many matched an how many didn't.

So, at first I did the 200 or so SSL certificates I had first retrieved.  Surprisingly the program actually ran in less than a second!  But, every SSL certificate was the same.

A few more times I continued this process, letting it grab CA for about 5 minutes and then comparing them and still not finding any differences.

I had thought that maybe the SSL certificates were all done replicating on Microsoft's servers (or that they had fixed the issue).  So I contacted a couple customers.  One said no problems, the other said yes, it was still happening.  

Well, by this time it was 5:30 and I needed to go pick up my daughter from gymnastics.  So I fired up the program and let it run while I was gone for a good hour.

When I got back I had over 900 certificates to test against,.  I copied them to the IFS and ran my program.  I was happy to finally see:

File /tmp/cas/20161213170505.497000-360certfile.txt is DIFFERENT!   
Object copied.                                                      
File /tmp/cas/20161213170500.510000-360certfile.txt is DIFFERENT!   
Object copied.                                                      
File /tmp/cas/20161213170455.494000-360certfile.txt is DIFFERENT!   
Object copied.                                                      
File /tmp/cas/20161213170450.531000-360certfile.txt is DIFFERENT!   
Object copied.                                                      
File /tmp/cas/20161213170445.561000-360certfile.txt is DIFFERENT!   
Object copied.                                                      
File /tmp/cas/20161213170440.527000-360certfile.txt is DIFFERENT!   
Object copied.                                                      
Match:805 - Not: 73                                                 

Thank goodness!  I looked in the /tmp/cas/different directory and noticed they were all the same.  These all had to be the elusive second SSL certificate that customers were randomly getting causing the error.  I exported the CAs from it, sent them to my customers and prayed.

That night a few customers reported that they had zero problems since installing the new CAs.  And this morning, even more.  So, I believe this is crisis averted for now.

What's the moral of this story?  I don't know.  What I DO know is the next time I have a customer ask which I prefer or recommend when it comes to a cloud mail provider, it will be GMail over Microsoft Office 365 for sure.  This won't have much bearing on most Microsoft Shops, but it may help others.  

I also know that if this is how Microsoft treats paying customers (ie, ignoring them), that's not something that I, as a small business owner who values his customers, will never want to parrot.