You may find out as me that after proper setting Mitmproxy up python https requests end up with ERR_CERT_AUTHORITY_INVALID error while other http clients are working with no problems. Quite unexpectable, isn't it?
Let's have a look at the example.
$ HTTPS_PROXY=127.0.0.1:8080 https github.com https: error: SSLError: HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)) while doing a GET request to URL: https://github.com/
It happens because httpie client internally use requests lib and one doesn't see the OS ssl system-wide settings directories. But it is customizable as described in the documentation how to specify manually a cert file which we are using for proxyfying.
Important remark here is we don't open client's code and add any settings into an http request call as we are man in the middle who technically may not have a way to edit the code.
Armed with the cert file path let's run the same command adding the env
$ HTTPS_PROXY=127.0.0.1:8080 \ REQUESTS_CA_BUNDLE=/usr/local/share/ca-certificates/mitmproxy-ca-cert.crt \ https github.com HTTP/1.1 200 OK Accept-Ranges: bytes Cache-Control: max-age=0, private, must-revalidate Server: GitHub.com ...
https client picks up the right cert file and goes through Mitmproxy.
Another way is to figure out what a cert file requests based client use and substitute its path into the command
mitmproxy --certs *=<cert-file>.
So what the cert file is and where? Well, we have general information requests deals with two envs
REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE and relies on certifi utility to delegate it all the certs searching in a file system stuff.
Seems the requests source code has to be inspected:
# Look for requests environment configuration and be compatible # with cURL. if verify is True or verify is None: verify = (os.environ.get('REQUESTS_CA_BUNDLE') or os.environ.get('CURL_CA_BUNDLE'))
A point here is
CURL_CA_BUNDLEmay not have a value as
REQUESTS_CA_BUNDLEdoes, that's confusing. But what the information do we can find about the curl env?
Nothing! However, at the moment we still have a bit knowledge of the requests certs flow - the certifi util.
from certifi import where
from . import certs DEFAULT_CA_BUNDLE_PATH = certs.where()
It looks like what we are looking for. Semantic of a constant name makes me want to print a value of
In : import requests In : requests.utils.DEFAULT_CA_BUNDLE_PATH Out: 'venv/lib/python3.8/site-packages/certifi/cacert.pem'
Pretty "appropriate" file i consider. Now we see what we have to substitute in.
Substitute and run Mitmproxy
$ mitmproxy --certs "venv/lib/python3.8/site-packages/certifi/cacert.pem"
$ HTTPS_PROXY=127.0.0.1:8080 https --headers github.com https: error: SSLError: HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:852)'),)) while doing a GET request to URL: https://github.com/
I've failed and crying .
Mitmproxy them all with no pain.