The Path for Testing Path Traversal Vulnerabilities with Python

Sunday. April 12, 2020 - 2 mins

I have noticed an odd behavior in the requests module in Python, which uses urllib3. I inspected the root cause via regression testing, and I found that the root cause was a change introduced in urllib3.

import requests
requests.get("http://127.0.0.1/../../../../doing/certain/check")

Which should typically send the request to the following path: /../../../doing/certain/check. Instead, it requests /doing/certain/check and modifies the specified input.

The reason is dot segments escaping, following RFC3986.

The change to normalize URIs was introduced in this commit[1] to follow RFC3986[2]. The change is based on the “Remove Dot Segments” at the RFC.

This breaks a lot of security tools that are performing fuzzing for path traversal vulnerabilities in web applications.

I have compiled several solutions for this issue.

Solutions

Option 1: urllib.request

url = "https://example.com/../../../etc/passwd"
urllib.request.urlopen(url)

Option 2: requests.Request

Credit: Rich Warren - [3] and [4].

url = "https://example.com/../../../etc/passwd"
with requests.Session() as s:
    r = requests.Request(method='GET', url=url, headers=headers)
    prep = r.prepare()
    prep.url = url
    req = s.send(prep, verify=False)

And

url = "https://example.com/../../../etc/passwd"
s = requests.Session()
r = requests.Request(method='GET', url=url)
prep = r.prepare()
prep.url = url

Option 3: Using urllib3.HTTPConnectionPool

Credit: Quentin Pradet

import urllib3

pool = urllib3.HTTPConnectionPool("localhost", 8000)
r = pool.urlopen("GET", "/../../../../doing/certain/check")
print(r.status)

Option 4: Downgrading urllib3

Using previous versions of requests or urllib3 is possible.

The option for current tools that use requests or urllib3 is to revert to urllib3==1.24.3. However, there may be better options from an engineering perspective.

Final Additions

I started a discussion on the Python Urllib3 Issue tracker on Github: https://github.com/urllib3/urllib3/issues/1790

I wanted to thank Seth Michael Larson, Quentin Pradet, and https://github.com/hodbn for all their work on urllib3 and with the Python community!

References

Commit to support the removal of dot-segments: https://github.com/urllib3/urllib3/commit/5b047b645f5f93900d5e2fc31230848c25eb1f5f
RFC3986 - Section 5.2.4 - Remove Dot Segments: http://tools.ietf.org/html/rfc3986#section-5.2.4
https://github.com/trustedsec/cve-2019-19781/issues/13
https://github.com/trustedsec/cve-2019-19781/blob/master/citrixmash.py#L140

Mazin Ahmed

Thoughts of a hacker