Fix ERR_CERT_AUTHORITY_INVALID error python clients get while proxying. Mitmproxy

You may find out as me that after proper setting Mitmproxy up python https requests end up with ERR_CERT_AUTHORITY_INVALID error while other http clients are working with no problems. Quite unexpectable, isn't it?

Let's have a look at the example.

$ HTTPS_PROXY=127.0.0.1:8080 https github.com

https: error: SSLError: HTTPSConnectionPool(host='github.com', port=443): 
Max retries exceeded with url: / 
(Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)) 
while doing a GET request to URL: https://github.com/

It happens because httpie client internally use requests lib and one doesn't see the OS ssl system-wide settings directories. But it is customizable as described in the documentation how to specify manually a cert file which we are using for proxyfying.

Important remark here is we don't open client's code and add any settings into an http request call as we are man in the middle who technically may not have a way to edit the code.

Run command with env REQUESTS_CA_BUNDLE for requests based clients

Armed with the cert file path let's run the same command adding the env REQUESTS_CA_BUNDLE.

$ HTTPS_PROXY=127.0.0.1:8080 \
REQUESTS_CA_BUNDLE=/usr/local/share/ca-certificates/mitmproxy-ca-cert.crt \
https github.com

HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=0, private, must-revalidate
Server: GitHub.com
...

Now https client picks up the right cert file and goes through Mitmproxy.

Run Mitmproxy passing a custom cert file in

Another way is to figure out what a cert file requests based client use and substitute its path into the command mitmproxy --certs *=<cert-file>.

So what the cert file is and where? Well, we have general information requests deals with two envs REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE and relies on certifi utility to delegate it all the certs searching in a file system stuff.

Seems the requests source code has to be inspected:

  1. Locate REQUESTS_CA_BUNDLE.

    # Look for requests environment configuration and be compatible
            # with cURL.
            if verify is True or verify is None:
                verify = (os.environ.get('REQUESTS_CA_BUNDLE') or
                          os.environ.get('CURL_CA_BUNDLE'))

    A point here is CURL_CA_BUNDLE may not have a value as REQUESTS_CA_BUNDLE does, that's confusing. But what the information do we can find about the curl env?

  2. Search for CURL_CA_BUNDLE info.

    Nothing! However, at the moment we still have a bit knowledge of the requests certs flow - the certifi util.

  3. Search for certifi info.

    certs.py

    from certifi import where

    utils.py

    from . import certs

DEFAULT_CA_BUNDLE_PATH = certs.where()

    It looks like what we are looking for. Semantic of a constant name makes me want to print a value of DEFAULT_CA_BUNDLE_PATH.

  4. Print DEFAULT_CA_BUNDLE_PATH.

    In [1]: import requests
In [2]: requests.utils.DEFAULT_CA_BUNDLE_PATH
Out[2]: 'venv/lib/python3.8/site-packages/certifi/cacert.pem'

    Pretty "appropriate" file i consider. Now we see what we have to substitute in.

  5. Substitute and run Mitmproxy

    $ mitmproxy --certs "venv/lib/python3.8/site-packages/certifi/cacert.pem"

    $ HTTPS_PROXY=127.0.0.1:8080 https --headers github.com

https: error: SSLError: HTTPSConnectionPool(host='github.com', port=443):
 Max retries exceeded with url: /
(Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:852)'),))
 while doing a GET request to URL: https://github.com/

    I've failed and crying .

Afterwords

Mitmproxy them all with no pain.

Give it a star