Re: adding CA certificates (for use by sendmail)

On 11/12/24 19:30, Wolfgang Agnes wrote:

It's not easy to decide where I should ask a FreeBSD group or sendmail one.  I'm posting to both, but suggesting a follow-up-to comp.mail.sendmail.

I get why you chose comp.mail.sendmail.  But my take on this is that it's more of an OS (lack of certs) issue than it is a Sendmail (configuration) issue.  But both work for me.

I'm particularly interested---due to curiosity and my education on the subject---in the following sendmail message in maillog:
 --8<-------------------------------------------------------->8---
Nov 12 08:27:39 antartida sm-mta[72775]: STARTTLS=client, cert-subject=/CN=mx.google.com, cert-issuer=/C=US/O=Google+20Trust+20Services/CN=WR2, verifymsg=unable to get issuer certificate
--8<-------------------------------------------------------->8---
 My intuition says the system doesn't have the certificate for the CN
mx.google.com.

I interpret it a little bit differently.
As is true with most Unix errors, the error is staring you in the face. But you have to have seen the error before to recognize and understand it:
--> unable to get issuer certificate <--
The issuer is:
--> cert-issuer=/C=US/O=Google+20Trust+20Services/CN=WR2 <--
Or decoded:
--> cert-issuer=/C=US/O=Google Trust Services/CN=WR2 <--
The problem is related to the issuer's (Google Trust Service) certificate, not the subject's (mx.google.com) certificate.

If my intuition is totally wrong, feel free to stop reading right here.

You're closer than you realize.

The hostname connecting to my sendmail that generated that line above is
    aspmx.l.google.com

I think this was from Sendmail /sending/ an email to Google and acting as the client to Google's server.

So I tried (as a guess) to connect to this host on port 25, saying STARTTLS and fetching the certificates that came.

:-)

Because sendmail is my primary concern, I installed these certificates in /etc/mail/certs.  I was afraid that if they were outside of /etc/mail, sendmail would refuse to read them.  And if my operation were not successful in this conservative approach, then it would be a waste of time to try to move these cerficates outside of /etc/mail, so I did not even try that because I did not succeed in avoiding the message ``unable to get issuer certificate''.

I would have to reference documentation to refresh myself on the nuances and minutia of /etc/mail/certs to know for sure if that was the proper thing to do or not.
Having fought this type of battle, and won, multiple times, I would have actually opted for the system wide certificate store under /etc/ssl or /etc/tls.

I found got three certificates with the -showcerts option by OpenSSL.

Nice.  I suspect you're now in the 1% of people that work with TLS on the command line.  Welcome to the club.  And I'm sorry.

They were GTS_Root_R1_2028.pem---this expires in 2028, but I already had
one for GTS Root R1 that expires in 2036---, GTS_WR2_RSA.pem and
GTS_WR2_EC.pem.

Those names don't /exactly/ match the issuers that I see when running tests on my side, but they are very close.

I obtained them with
    openssl s_client -connect aspmx.l.google.com:25 \
       -starttls smtp \
       -showcerts </dev/null

Nice.
Here's a modified (added STARTTLS bit) version of an excerpt from a script I wrote today to check HTTPS chains:
    FQDN=smtp.google.com
    PORT=25
    openssl s_client -showcerts -connect ${FQDN}:${PORT} -STARTTLS smtp 0</dev/null 2>/dev/null | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' > ${FQDN}.pem
    csplit -zf ${FQDN}.pem ${FQDN}.pem '/-----BEGIN CERTIFICATE-----/' '{*}' -b ".cert-%0d.pem" > /dev/null
    PARTS=($(ls -1r ${FQDN}.pem.cert-?.pem))
This leaves us with four files:
  - smtp.google.com.pem - the combined file extracted with s_client
  - smtp.google.com.pem.cert-0.pem - the 1st cert
  - smtp.google.com.pem.cert-1.pem - the 2nd cert
  - smtp.google.com.pem.cert-2.pem - the 3rd cert

I obtained the hash of each certificate with, for example,
    openssl x509 -noout -hash < GTS_Root_R1_2028.pem

:-)
I tend to prefer the `-in ${FILE}` vs redirect approach.  But both work.

Then I created the symbolic links using these hash as ls shows below.

Regarding your "I already had" comment, it looks like you used ".0" for all of your hash files.  The .# nomenclature is to allow for different files that have the same hash.  Unexpected, but not impossible either.

But that did not change those log messages.  I don't know how to debug sendmail to the point of seeing which files it is reading. Thanks for any advice you might have.

I think that it's going to depend on how Sendmail is configured.  Is it using it's own certificate store or the system wide certificate store?
What is the CACertPath set to in your sendmail.cf file?
You will need your hash links to be in the directory that Sendmail is looking for root certs in.
When I look at the three smtp.google.com.pem.cert-#.pem files generated above, I find that there is a fourth certificate that is needed.
    for ((x=${#PARTS}; x>0; x--)); do
       echo;
       echo ${PARTS[$x]};
       openssl x509 -noout -subject -subject_hash -startdate -enddate -issuer -issuer_hash -in ${PARTS[$x]};
    done
    smtp.google.com.pem.cert-0.pem
    subject= /CN=mx.google.com
    6264a34a
    notBefore=Oct  7 08:25:31 2024 GMT
    notAfter=Dec 30 08:25:30 2024 GMT
    issuer= /C=US/O=Google Trust Services/CN=WR2
    3c8b39ef
    smtp.google.com.pem.cert-1.pem
    subject= /C=US/O=Google Trust Services/CN=WR2
    3c8b39ef
    notBefore=Dec 13 09:00:00 2023 GMT
    notAfter=Feb 20 14:00:00 2029 GMT
    issuer= /C=US/O=Google Trust Services LLC/CN=GTS Root R1
    1001acf7
    smtp.google.com.pem.cert-2.pem
    subject= /C=US/O=Google Trust Services LLC/CN=GTS Root R1
    1001acf7
    notBefore=Jun 19 00:00:42 2020 GMT
    notAfter=Jan 28 00:00:42 2028 GMT
    issuer= /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
    5ad8a5d6
The fourth file that's needed is the 5ad8a5d6.0 file (in /etc/ssl/certs on my system):
    openssl x509 -noout -subject -subject_hash -startdate -enddate -issuer -issuer_hash -in /etc/ssl/certs/5ad8a5d6.0
    subject= /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
    5ad8a5d6
    notBefore=Sep  1 12:00:00 1998 GMT
    notAfter=Jan 28 12:00:00 2028 GMT
    issuer= /C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
    5ad8a5d6
So, when I ask OpenSSL to verify the chain from smtp.google.com / mx.google.com all the way back to the root certificates that my system trusts, the chain is valid:
smtp.google.com.pem.cert-0.pem (6264a34a) issued by 3c8b39ef
smtp.google.com.pem.cert-1.pem (3c8b39ef) issued by 1001acf7
smtp.google.com.pem.cert-2.pem (1001acf7) issued by 5ad8a5d6
And then the trusted root certificate:
/etc/ssl/certs/5ad8a5d6.0 -> GlobalSign_Root_CA.pem (5ad8a5d6) issued by 5ad8a5d6 (itself)
    openssl verify -verbose -untrusted smtp.google.com.pem.cert-2.pem -untrusted smtp.google.com.pem.cert-1.pem smtp.google.com.pem.cert-0.pem 2>/dev/null
    smtp.google.com.pem.cert-0.pem: OK
Thus we have a complete chain of trust from the leaf cert used by smtp.google.com 6264a34a, through two intermediary certs (3c8b39ef & 1001acf7), to a trusted root (5ad8a5d6).  Thus a complete chain of trust.
I think that you're probably on the correct track.  I speculate that it's a possible mismatch of where Sendmail is looking and where you have your files.
--
Grant. . . .