Function ssl.create_default_context raises exception on Windows 10 when called with ssl.Purpose.SERVER_AUTH) attribute

[pervlad]

BPO	35665
Nosy	@tiran, @pervlad, @MDM-79, @pukkandan
Files	cacerts.pem: pub certs Untitled.png: MUPCA Root - Certificates

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-01-08.09:38:24.479>
created_at = <Date 2019-01-05.12:24:45.346>
labels = ['expert-SSL', 'type-bug', '3.9']
title = 'Function ssl.create_default_context raises exception on Windows 10  when called with  ssl.Purpose.SERVER_AUTH) attribute'
updated_at = <Date 2021-09-28.21:47:58.903>
user = 'https://github.com/pervlad'

bugs.python.org fields:

activity = <Date 2021-09-28.21:47:58.903>
actor = 'pukkandan'
assignee = 'none'
closed = True
closed_date = <Date 2019-01-08.09:38:24.479>
closer = 'pervlad'
components = ['SSL']
creation = <Date 2019-01-05.12:24:45.346>
creator = 'pervlad'
dependencies = []
files = ['48029', '50311']
hgrepos = []
issue_num = 35665
keywords = []
message_count = 16.0
messages = ['333054', '333055', '333158', '333159', '333168', '333181', '333183', '333184', '333215', '333224', '360050', '402782', '402792', '402794', '402805', '402809']
nosy_count = 5.0
nosy_names = ['christian.heimes', 'pervlad', 'pedja', 'MDM-1', 'pukkandan']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35665'
versions = ['Python 3.9']

Linked PRs

• gh-79846: Make ssl.create_default_context() ignore invalid certificates #91740
• [3.13] gh-79846: Make ssl.create_default_context() ignore invalid certificates (GH-91740) #122768
• [3.12] gh-79846: Make ssl.create_default_context() ignore invalid certificates (GH-91740) #122769

[pervlad]

In Python 3.7.1 on Windows 10 ssl library function call ssl.create_default_context(ssl.Purpose.SERVER_AUTH) raises an ssl error:

File "C:\Python37\lib\ssl.py", line 471, in _load_windows_store_certs
self.load_verify_locations(cadata=certs)
ssl.SSLError: nested asn1 error (_ssl.c:3926)

In Python 3.6.4 same function call raises no error.

[pervlad]

Same outcome in Python 3.7.2.
See first comment for detailed explanation of issue.

[vstinner]

self.load_verify_locations(cadata=certs)
...
ssl.SSLError: nested asn1 error (_ssl.c:3926)

It seems like one of your certificate is invalid.

In Python 3.6.4 same function call raises no error.

We frequently update OpenSSL in Python. You can get OpenSSL version using:

$ python3
Python 3.7.2 (default, Jan  3 2019, 09:14:01) 
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.1.1 FIPS  11 Sep 2018'
>>> ssl.OPENSSL_VERSION_INFO
(1, 1, 1, 0, 15)
>>> ssl.OPENSSL_VERSION_NUMBER
269488143
>>> hex(ssl.OPENSSL_VERSION_NUMBER)
'0x1010100f'

[vstinner]

Would it be possible to attach the certification to the issue so someone can try to reproduce the issue? (but don't attach any private key ;-))

[tiran]

The certs are coming from Windows' trust store. Could you please dump the trust store for me and attach the result to the bug tracker. The following script is untested but should work. I don't have access to a Windows machine at the moment.

ctx = ssl.SSLContext(ssl.PROTOCOL_TLS)
certs = []
for storename in ("CA", "ROOT"):
    certs.append(storename)
    for cert, encoding, trust in ssl.enum_certificates(storename):
        if encoding == "x509_asn":
            if trust is True or ssl.Purpose.SERVER_AUTH.oid in trust:
                try:
                    ctx.load_verify_locations(cadata=cert)
                except Exception as e:
                    certs.append(str(e))
                certs.append(ssl.DER_cert_to_PEM_cert(cert))

with open('cacerts.pem', 'w') as f:
    f.write('\n'.join(certs))

[pervlad]

Public Certificate file cert.pem is attached.

Version of ssl lib in pythons on my machine:
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.1.0j  20 Nov 2018'

Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.2q  20 Nov 2018'

[tiran]

Your Windows cert store contains multiple invalid certificates. The first failing certificate is the custom "MUPCA Root", which looks like a certificate from http://ca.mup.gov.rs/sertifikati.html. The serial number seems to be badly formated or padded. There is nothing we can do about erroneous and bad certificates.

$ openssl x509 -in ca.pem 
unable to load certificate
140613019477824:error:0D0E20DD:asn1 encoding routines:c2i_ibuf:illegal padding:crypto/asn1/a_int.c:187:
140613019477824:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:crypto/asn1/tasn_dec.c:627:Field=serialNumber, Type=X509_CINF
140613019477824:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:crypto/asn1/tasn_dec.c:627:Field=cert_info, Type=X509
140613019477824:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib:crypto/pem/pem_oth.c:33:

$ openssl asn1parse -in ca.pem  
    0:d=0  hl=4 l=1300 cons: SEQUENCE          
    4:d=1  hl=4 l= 764 cons: SEQUENCE          
    8:d=2  hl=2 l=   3 cons: cont [ 0 ]        
   10:d=3  hl=2 l=   1 prim: INTEGER           :02
   13:d=2  hl=2 l=   4 prim: INTEGER           :BAD INTEGER:[00000066]
   19:d=2  hl=2 l=  13 cons: SEQUENCE          
   21:d=3  hl=2 l=   9 prim: OBJECT            :sha1WithRSAEncryption
   32:d=3  hl=2 l=   0 prim: NULL              
   34:d=2  hl=2 l=  83 cons: SEQUENCE          
   36:d=3  hl=2 l=  19 cons: SET               
   38:d=4  hl=2 l=  17 cons: SEQUENCE          
   40:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   45:d=5  hl=2 l=  10 prim: UTF8STRING        :MUPCA Root
   57:d=3  hl=2 l=  29 cons: SET               
   59:d=4  hl=2 l=  27 cons: SEQUENCE          
   61:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
   66:d=5  hl=2 l=  20 prim: UTF8STRING        :MUP Republike Srbije
   88:d=3  hl=2 l=  16 cons: SET               
   90:d=4  hl=2 l=  14 cons: SEQUENCE          
   92:d=5  hl=2 l=   3 prim: OBJECT            :localityName
   97:d=5  hl=2 l=   7 prim: UTF8STRING        :Beograd
  106:d=3  hl=2 l=  11 cons: SET               
  108:d=4  hl=2 l=   9 cons: SEQUENCE          
  110:d=5  hl=2 l=   3 prim: OBJECT            :countryName
  115:d=5  hl=2 l=   2 prim: PRINTABLESTRING   :RS
  119:d=2  hl=2 l=  30 cons: SEQUENCE          
  121:d=3  hl=2 l=  13 prim: UTCTIME           :100227161918Z
  136:d=3  hl=2 l=  13 prim: UTCTIME           :200227161918Z
  ...

$ wget http://ca.mup.gov.rs/MUPCARoot.crt
$ openssl x509 -in MUPCARoot.crt -inform DER
unable to load certificate
140699773712192:error:0D0E20DD:asn1 encoding routines:c2i_ibuf:illegal padding:crypto/asn1/a_int.c:187:
140699773712192:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:crypto/asn1/tasn_dec.c:627:Field=serialNumber, Type=X509_CINF
140699773712192:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:crypto/asn1/tasn_dec.c:627:Field=cert_info, Type=X509

[tiran]

OpenSSL 1.1.0 is more strict than OpenSSL 1.0.2. That's why you don't see the issue with Python 3.6 but with 3.7. The problem is explained in https://mta.openssl.org/pipermail/openssl-dev/2016-February/005100.html

The CA has encoded the integer 102 (0x66) as "02 04 00
00 00 66", which violates the DER standard. The correct encoding is "02 01 66".

>>> from asn1crypto.core import Integer
>>> import binascii
>>> binascii.hexlify(Integer(102).dump())
b'020166'

[pervlad]

Thank you all for this expeditive help. Sorry for taking your time.
I will remove bad certificates from my machine.
Thanks again.

I will try to close this one.

[tiran]

I also checked how other implementations deal with invalid DER encoding. NSS 3.41, Firefox, and Chromium accept the certifiate.

NSS shows the serial number as "102 (0x66)"
Firefox and Chromium display the serial number as "00:00:00:66".

$ echo "password" > passwd
$ certutil -d . -f passwd -N
$ certutil -d . -f passwd -A -n ca -i ../ca.pem -t C,C,C
$ certutil -d . -L -n ca
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 102 (0x66)
        Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption
        Issuer: "C=RS,L=Beograd,O=MUP Republike Srbije,CN=MUPCA Root"
        Validity:
            Not Before: Sat Feb 27 16:19:18 2010
            Not After : Thu Feb 27 16:19:18 2020
        Subject: "C=Re...,L=Beograd,O=MUP Republike Srbije,CN=MUPCA Resursi"
        Subject Public Key Info:
            Public Key Algorithm: PKCS #1 RSA Encryption
            RSA Public Key:
                Modulus:
                    ea:69:46:bc:c7:70:00:d5:f5:32:8d:c7:4e:ad:3a:a5:
                    d3:29:7e:a2:46:12:a9:dd:57:75:b1:49:95:80:20:ed:
                    9b:68:6b:e3:c5:55:d8:64:15:68:42:ab:a3:f7:c0:96:
                    37:08:51:cb:05:ca:b5:99:f6:07:a6:8b:f2:cd:d2:f5:
                    d6:16:12:da:bf:a8:0b:9c:45:5d:ac:79:1d:a8:67:47:
                    ee:7f:83:40:f8:58:00:d5:dd:c4:c9:52:1b:d2:f4:ce:
                    e1:fa:8a:66:d3:18:86:1e:ea:fc:0a:8b:b5:ec:49:cd:
                    86:bf:8b:7e:b0:61:81:ec:ea:99:4f:64:82:96:93:9d:
                    ab:80:7d:a7:27:65:00:d4:12:26:98:45:64:7e:76:0b:
                    98:ff:16:50:49:0c:45:20:82:ce:2e:23:a2:65:3a:b7:
                    44:cd:51:00:d9:bf:e3:1f:de:23:1d:57:e9:32:c3:55:
                    f0:24:af:d4:cf:cd:9e:77:1f:19:7e:1c:03:5b:7a:e4:
                    75:84:3b:d4:1d:e9:23:d6:8c:f2:8f:b2:0d:e3:79:df:
                    9e:03:1e:0e:15:5b:7b:0c:dd:6e:4d:82:86:5a:63:79:
                    64:b5:07:79:dd:fd:08:e3:d6:cb:60:01:fd:82:11:59:
                    2c:8d:22:f8:f9:91:59:b1:cd:12:7b:39:6d:08:82:5d
                Exponent: 65537 (0x10001)
        Signed Extensions:
            Name: Certificate Basic Constraints
            Critical: True
            Data: Is a CA with no maximum path length.

        Name: Certificate Key Usage
        Critical: True
        Usages: Certificate Signing
                CRL Signing

        Name: Authority Information Access
        Method: PKIX CA issuers access method
        Location: 
            URI: "http://ca.mup.gov.rs/MUPCARoot.crt"

        Name: Certificate Subject Key ID
        Data:
            cb:f9:00:a9:b7:b6:c1:6f:44:43:d0:22:ad:fc:0e:6e:
            cc:8f:f6:0f

        Name: Certificate Authority Key Identifier
        Key ID:
            3f:66:b0:0f:66:fb:f0:10:2e:61:a4:6f:ef:2c:95:8a:
            14:72:6f:71

        Name: CRL Distribution Points
        Distribution point:
            URI: "http://ca.mup.gov.rs/MUPCARoot.crl"

Signature Algorithm: PKCS #1 SHA-1 With RSA Encryption

[pedja]

This is still an issue. Serious one. People are unable to just remove this certificate as it is needed for everyday use.

It is reasonable that application does not deal with invalid certificates.
I can understand application to breaks if one tries to use invalid certificate.

But this is not that case. Application breaks on just enumerating certificates even if it does not need it at all.

It is advisable for application just to skip and ignore invalid certificate unless it is required for application to work.

Please reconsider action on this issue to prevent application breaking when it is not necessary.

[MDM-79]

Just to ad to the issue, and to further update the importance of those certificates...

I came to this issue (still persistent with all python versions since 3.6) while using yt-dlp: yt-dlp/yt-dlp#1060

I obviously have the SAME problem than the guy in your link since I am from Serbia too, and those certificates "MUPCA Root" are (unfortunately-badly executed) crucial (issued by the ministry of interior - police 🙄) ones to be able too read ID cards and use personal signing certificates, and they're are all valid...
So the option to remove the faulty certificates, is a no go to me (or anyone in Serbia using their ID card - individuals, companies and entrepreneurs like me)...

Please help!

[tiran]

We cannot fix the issue in Python. Please report the problem to OpenSSL and to your government. Either OpenSSL needs to relax its cert parser again or your government has to replace the broken certificates with correct certificates.

[MDM-79]

Maybe better to continue in my newly opened tread https://bugs.python.org/issue45312 since I suppose I wasn't correctly specific (read I am a noob!), and pukkandan was more so.

And my government fixing their certificates?! No chance i hell, they are like this for more of a decade! 😏

[tiran]

Could you please open an OpenSSL bug on the projects bug tracker https://github.com/openssl/openssl/ and explain the issue there? They might be able to implement a workaround for the broken certificates or advise you how to handle the invalid certificates.

A workaround for Python would require a major rewrite of the Windows CA store integration. We don't have any capacity to work on that area. Even if we had capacity, a workaround would land in Python 3.11 earliest (October 2022).

[pukkandan]

A workaround for Python would require a major rewrite of the Windows CA store integration. We don't have any capacity to work on that area

In theory, the issue can be worked around by simply loading each certificate separately. See yt-dlp/yt-dlp@599ca41

[levicki]

@tiran Can you please clarify whether this issue been fixed, and if so in which Python version? I am still encountering it in Miniconda3 which bundles Python 3.9.12 with OpenSSL 1.1.1q.

I understand that the underlying issue stems from OpenSSL throwing an error on invalid DER encoding (in some cases it's the serial number which is harmless), but the end result is that HTTPS connections established using default context in Python code are failing if user has any invalid certificates in their Windows root certificate store.

I'd like to point out that:

1. Error message (stacktrace) shown by Python doesn't help the end user understand the error, much less find the offending certificate
2. Even if it did, regular users trying to run a Python application may not have local admin rights to remove those certificates from the OS root CA store
3. Even if they had local admin rights, randomly removing root CA certificates trying to figure out which one is responsible can render their operating system unusable
4. Even if they had the required knowledge of which CAs they can safely remove, they might need those offending certificates anyway and removing them would just break some other application or intranet website they are using
5. In 99% of cases those certificates are not needed for HTTPS communication and can be safely skipped

So clearly, the bare minimum that could be done by Python developers is to modify _load_windows_store_certs to load certificates one by one and skip the invalid ones:

    def _load_windows_store_certs(self, storename, purpose):
        certs = bytearray()
        try:
            for cert, encoding, trust in enum_certificates(storename):
                # CA certs are never PKCS#7 encoded
                if encoding == "x509_asn":
                    if trust is True or purpose.oid in trust:
                        try:
                            self.load_verify_locations(cadata=cert)
                            certs.extend(cert)
                        except SSLError:
                            warnings.warn("Bad certificate in Windows certificate store")
        except PermissionError:
            warnings.warn("unable to enumerate Windows certificate store")
        return certs

It would be even better if the CN or friendlyName of the offending certificate could be displayed instead of just Bad certificate in Windows certificate store, but even this change would help immensely.

Please advise.

[serhiy-storchaka]

Reopened this issue as the original problem have not been solved yet and we continue to receive bug reports.

Has anyone reported the problem to OpenSSL and to the Serbian government as was suggested in #79846 (comment)?

We can add a workaround in the Python stdlib (BTW, the code proposed by @levicki looks better than #91740), but it will only solve the part of the problem. And I am not sure that ignoring all errors will not open a security hole.

[levicki]

Has anyone reported the problem to OpenSSL and to the Serbian government as was suggested in #79846 (comment)?

I have just opened an issue with OpenSSL:
openssl/openssl#25023

I don't think they will be too willing to relax the DER validation though.

As for Serbian Interior Ministry, they certainly won't be reissuing the problematic root which is about to expire in 2030.

We can add a workaround in the Python stdlib (BTW, the code proposed by @levicki looks better than #91740), but it will only solve the part of the problem. And I am not sure that ignoring all errors will not open a security hole.

If you ignore a malformed certificate (i.e. do not add it from the Windows store into the Python trust store) then there is no security hole — if the certificate is used for securing HTTPS connection (this particular root isn't), then the communication attempt will fail.

This particular certificate isn't used to secure communication (just as a root for document signing certificates), but others might be used for communication as well so the best option would be to load and trust all certificates if possible even if they are malformed.

My rationale is that they are already in the user / system root store which assumes that the user trusts them and wants them to work.

[serhiy-storchaka]

Thank you @levicki for reviving this issue.

[levicki]

Thank you @levicki for reviving this issue.

You're welcome.

However, if the fix isn't backported to 3.10 and 3.11 as well you may keep receiving those bug reports — many popular "AI" repos (such as AUTOMATIC1111 for example) have 3.10 as a requirement. Some of them can work also with 3.11, but many can't upgrade without breaking metric ton of libraries they use.

[hugovk]

@levicki Unfortunately 3.10 and 3.11 are only accepting security fixes, see https://devguide.python.org/versions/ and https://devguide.python.org/developer-workflow/development-cycle/#security-branches.

@serhiy-storchaka The 3.12 and 3.13 backports are pending merge, keeping in mind 3.13 is in the RC phase.

[levicki]

@hugovk

@levicki Unfortunately 3.10 and 3.11 are only accepting security fixes, see https://devguide.python.org/versions/ and https://devguide.python.org/developer-workflow/development-cycle/#security-branches.

Yes, I am well aware of that. Too bad, because 3.10 is still widely used in AI community without any indication that it could be replaced soon due to all code dependencies.