Problem with FEATURE(`sts'): bogus "not listed in SANs" rejects

I have been testing out the MTA-STS support for a while, using the
Debian sendmail package 8.17.1.9-2+deb12u2 with postfix-mta-sts-resolver
to look up policies.

This seemed to work pretty well, but yesterday I noticed a problem: Some
emails were temporarily rejected with an unexpected "not listed in SANs"
message.  This was unexpexted because most of the affected emails were
for MXes hosted by Microsoft and their "protection.outlook.com" service.

Anonymzed example log lines:

 Oct 27 19:49:47 dilbert sm-mta[1589041]: 49N8ZWuJ1287620: ruleset=tls_rcpt, arg1=redacted@hotmail.com, relay=hotmail-com.olc.protection.outlook.com., reject=450 4.7.0 <redacted@hotmail.com>... hotmail-com.olc.protection.outlook.com not listed in SANs
 Oct 27 19:54:49 dilbert sm-mta[1589264]: 493ISGKc649896: ruleset=tls_rcpt, arg1=redacted.redacted@outlook.com, relay=outlook-com.olc.protection.outlook.com., reject=450 4.7.0 <redacted.redacted@outlook.com>... outlook-com.olc.protection.outlook.com not listed in SANs


The smarthost handles only a very low volume, and most MXes don't have a
MTA-STS policy obviously.  But amond those few, there was also a similar
reject for a non-Microsoft hosted MX:


 Oct 27 19:24:48 dilbert sm-mta[1587923]: 49JAD2Eb1019895: ruleset=tls_rcpt, arg1=redacted@01-mail.org, relay=mx.01-mail.net., reject=450 4.7.0 <redacted@01-mail.org>... mx.01-mail.net not listed in SANs


There were also a few examples of MXes publishing a policy and still
working fine. This included my own MX as well as Googles:

 alt1.gmail-smtp-in.l.google.com
 alt2.gmail-smtp-in.l.google.com
 alt3.gmail-smtp-in.l.google.com
 alt4.gmail-smtp-in.l.google.com
 gmail-smtp-in.l.google.com


Looking at the certificates I see that the most obvious difference is
the use of wildcard SANs.  Both the certficate of my MX and Googles
explicitly list all their names, using no wildcards at all:

            X509v3 Subject Alternative Name:
                DNS:canardo.dyn.mork.no, DNS:canardo.mork.no, DNS:imap.mork.no, DNS:mail.mork.no, DNS:smtp.mork.no


            X509v3 Subject Alternative Name:
                DNS:mx.google.com, DNS:smtp.google.com, DNS:aspmx.l.google.com, DNS:alt1.aspmx.l.google.com, DNS:alt2.aspmx.l.google.com, DNS:alt3.aspmx.l.google.com, DNS:alt4.aspmx.l.google.com, DNS:gmail-smtp-in.l.google.com, DNS:alt1.gmail-smtp-in.l.google.com, DNS:alt2.gmail-smtp-in.l.google.com, DNS:alt3.gmail-smtp-in.l.google.com, DNS:alt4.gmail-smtp-in.l.google.com, DNS:gmr-smtp-in.l.google.com, DNS:alt1.gmr-smtp-in.l.google.com, DNS:alt2.gmr-smtp-in.l.google.com, DNS:alt3.gmr-smtp-in.l.google.com, DNS:alt4.gmr-smtp-in.l.google.com, DNS:mx1.smtp.goog, DNS:mx2.smtp.goog, DNS:mx3.smtp.goog, DNS:mx4.smtp.goog, DNS:aspmx2.googlemail.com, DNS:aspmx3.googlemail.com, DNS:aspmx4.googlemail.com, DNS:aspmx5.googlemail.com, DNS:gmr-mx.google.com


While both the failing MXes use certificates with multiple wildcard
SANs:

            X509v3 Subject Alternative Name:
                DNS:mail.protection.outlook.com, DNS:*.mail.eo.outlook.com, DNS:*.mail.protection.outlook.com, DNS:mail.messaging.microsoft.com, DNS:outlook.com, DNS:*.olc.protection.outlook.com, DNS:*.pamx1.hotmail.com, DNS:*.mail.protection.outlook.de, DNS:*.mx.microsoft, DNS:*.k-v1.mx.microsoft, DNS:*.n-v1.mx.microsoft, DNS:*.q-v1.mx.microsoft, DNS:*.y-v1.mx.microsoft, DNS:*.d-v1.mx.microsoft, DNS:*.e-v1.mx.microsoft, DNS:*.a-v1.mx.microsoft, DNS:*.r-v1.mx.microsoft, DNS:*.w-v1.mx.microsoft, DNS:*.p-v1.mx.microsoft, DNS:*.x-v1.mx.microsoft, DNS:*.j-v1.mx.microsoft, DNS:*.s-v1.mx.microsoft, DNS:*.c-v1.mx.microsoft, DNS:*.b-v1.mx.microsoft, DNS:*.f-v1.mx.microsoft, DNS:*.i-v1.mx.microsoft, DNS:*.t-v1.mx.microsoft, DNS:*.m-v1.mx.microsoft, DNS:*.o-v1.mx.microsoft, DNS:*.g-v1.mx.microsoft, DNS:*.v-v1.mx.microsoft, DNS:*.h-v1.mx.microsoft, DNS:*.l-v1.mx.microsoft, DNS:*.u-v1.mx.microsoft

            X509v3 Subject Alternative Name:
                DNS:*.01-mail.com, DNS:*.01-mail.net, DNS:*.01-mail.org, DNS:01-mail.com, DNS:01-mail.net, DNS:01-mail.org


It's dangerous to draw conclusions based on so few samples, but this is
all I've got. Could there be a problem with FEATURE(`sts') related to
wildcard SANs?  Either wildcard SANs in general, or maybe just when
there is more than one?

I see that there is code in cf/m4/proto.m4 which is supposed to do
wildcard matching, but I must admit that I am too clueless wrt m4 to
understand if this should work with the certificate examples above:

dnl check SAN for STS
SSTS_SAN
ifdef(`_STS_SAN', `dnl
R$*                     $: $&{server_name}
dnl exact match
R$={cert_altnames}      $@ ok
# strip only one level (no recursion!)
R$-.$+                  $: $2
dnl wildcard: *. or just .?
R *.$={cert_altnames}   $@ ok
dnl R .$={cert_altnames}        $@ ok
dnl always temporary error? make it an option (of the feature)?
R$*                     $#error $@ 4.7.0 $: 450 $&{server_name} not listed in SANs', `dnl')



I have not yet verified this problem on 8.18, but AFAICS the code I am
quoting above was introduced in 8.17.1 and has not been changed since.

The getaltnames() function in sendmail/tls.c, populating the
cert_altnames class has not been changed since it was introduced in
8.16.0.48.

Disabling the SAN test by doing

 FEATURE(`sts',,`NO_SAN_TST')

is verified to "solve" the problem as expected.  But obviously with the
huge downside of not actually validating the MX name.



Bjørn