The Path for Testing Path Traversal Vulnerabilities with Python- 3 mins
I have noticed an odd behavior in
requests module in Python, which uses urllib3. I inspected the root cause via regression testing and I found that the root cause of the issue is a change that was introduced in urllib3.
import requests requests.get("http://127.0.0.1/../../../../doing/certain/check")
Which should typically send the request to the following path:
/../../../doing/certain/check. Instead, it’s requesting
/doing/certain/check and modifying the specified input.
The reason is, dot segments escaping, following RFC3986.
The change to normalize URIs was introduced in this commit to follow RFC3986. The change is based on the “Remove Dot Segments” at the RFC.
This breaks a lot of security tools that are performing fuzzing for path traversal vulnerabilities in web-applications.
I have compiled a number of solutions for this issue.
Option 1: urllib.request
url = "https://example.com/../../../etc/passwd" urllib.request.urlopen(url)
Option 2: requests.Request
Credit: Rich Warren -  and .
url = "https://example.com/../../../etc/passwd" with requests.Session() as s: r = requests.Request(method='GET', url=url, headers=headers) prep = r.prepare() prep.url = url req = s.send(prep, verify=False)
url = "https://example.com/../../../etc/passwd" s = requests.Session() r = requests.Request(method='GET', url=url) prep = r.prepare() prep.url = url
Option 3: Using urllib3.HTTPConnectionPool
Credit: Quentin Pradet
import urllib3 pool = urllib3.HTTPConnectionPool("localhost", 8000) r = pool.urlopen("GET", "/../../../../doing/certain/check") print(r.status)
Option 4: Downgrading urllib3
It’s possible to use previous versions of
The option for current tools that use requests or urllib3 is to revert to
urllib3==1.24.3. However, this may not be the best option from an engineering perspective.
I started a discussion on Python Urllib3 Issue tracker on Github: https://github.com/urllib3/urllib3/issues/1790
- Commit to support the removal of dot-segments: https://github.com/urllib3/urllib3/commit/5b047b645f5f93900d5e2fc31230848c25eb1f5f
- RFC3986 - Section 5.2.4 - Remove Dot Segments: http://tools.ietf.org/html/rfc3986#section-5.2.4