The Path for Testing Path Traversal Vulnerabilities with Python

- 3 mins

I have noticed an odd behavior in requests module in Python, which uses urllib3. I inspected the root cause via regression testing and I found that the root cause of the issue is a change that was introduced in urllib3.

import requests
requests.get("http://127.0.0.1/../../../../doing/certain/check")

Which should typically send the request to the following path: /../../../doing/certain/check. Instead, it’s requesting /doing/certain/check and modifying the specified input.

The reason is, dot segments escaping, following RFC3986.

The change to normalize URIs was introduced in this commit[1] to follow RFC3986[2]. The change is based on the “Remove Dot Segments” at the RFC.

This breaks a lot of security tools that are performing fuzzing for path traversal vulnerabilities in web-applications.

I have compiled a number of solutions for this issue.


Solutions

Option 1: urllib.request

url = "https://example.com/../../../etc/passwd"
urllib.request.urlopen(url)

Option 2: requests.Request

Credit: Rich Warren - [3] and [4].

url = "https://example.com/../../../etc/passwd"
with requests.Session() as s:
    r = requests.Request(method='GET', url=url, headers=headers)
    prep = r.prepare()
    prep.url = url
    req = s.send(prep, verify=False)

And

url = "https://example.com/../../../etc/passwd"
s = requests.Session()
r = requests.Request(method='GET', url=url)
prep = r.prepare()
prep.url = url

Option 3: Using urllib3.HTTPConnectionPool

Credit: Quentin Pradet

import urllib3

pool = urllib3.HTTPConnectionPool("localhost", 8000)
r = pool.urlopen("GET", "/../../../../doing/certain/check")
print(r.status)

Option 4: Downgrading urllib3

It’s possible to use previous versions of requests or urllib3.

The option for current tools that use requests or urllib3 is to revert to urllib3==1.24.3. However, this may not be the best option from an engineering perspective.


Final Additions

I started a discussion on Python Urllib3 Issue tracker on Github: https://github.com/urllib3/urllib3/issues/1790

I wanted to thank Seth Michael Larson, Quentin Pradet, and https://github.com/hodbn, for all their work on urllib3 and with the Python community!


References

  1. Commit to support the removal of dot-segments: https://github.com/urllib3/urllib3/commit/5b047b645f5f93900d5e2fc31230848c25eb1f5f
  2. RFC3986 - Section 5.2.4 - Remove Dot Segments: http://tools.ietf.org/html/rfc3986#section-5.2.4
  3. https://github.com/trustedsec/cve-2019-19781/issues/13
  4. https://github.com/trustedsec/cve-2019-19781/blob/master/citrixmash.py#L140
Mazin Ahmed

Mazin Ahmed

Thoughts of a hacker

rss facebook twitter github gitlab youtube mail spotify lastfm instagram linkedin google google-plus pinterest medium vimeo stackoverflow reddit quora quora