[CVE-2024-11168] urlparse incorrectly retrieves IPv4 and regular name hosts from inside of brackets

Background

RFC 3986 defines a host as follows

host = IP-literal / IPv4address / reg-name

Where

IP-literal = "[" ( IPv6address / IPvFuture  ) "]"
reg-name = *( unreserved / pct-encoded / sub-delims )
IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet

WhatWG says that "A valid host string must be a valid domain string, a valid IPv4-address string, or: U+005B ([), followed by a valid IPv6-address string, followed by U+005D (])."

The Bug

This is code from Lib/urllib/parse.py:196-208 used for retrieving the hostname from the netloc

    @property
    def _hostinfo(self):
        netloc = self.netloc
        _, _, hostinfo = netloc.rpartition('@')
        _, have_open_br, bracketed = hostinfo.partition('[')
        if have_open_br:
            hostname, _, port = bracketed.partition(']')
            _, _, port = port.partition(':')
        else:
            hostname, _, port = hostinfo.partition(':')
        if not port:
            port = None
        return hostname, port

It will incorrectly retrieve IPv4 addresses and regular name hosts from inside brackets. This is in violation of both specifications.

Minimally reproducible example:

from urllib.parse import urlsplit

parsedURL = urlsplit('scheme://user@[regname]/Path')
print(parsedURL.hostname) # Prints 'regname'

Your environment

CPython versions tested on:
- 3.12a7 (23cf1e2)
- 3.10.10
Operating system and architecture:
- Arch Linux x86_64

Linked PRs

gh-103848: Adds checks to ensure that bracketed hosts found by urlspl…

pythongh-103848: Adds checks to ensure that bracketed hosts found by …

[3.11] pythongh-103848: Adds checks to ensure that bracketed hosts fo…

[3.11] gh-103848: Adds checks to ensure that bracketed hosts found by…

Merge branch 'main' into fixloadsuperspec

As implemented, the URL scheme://foobar.[::1]/ is considered correct, but results in the hostname value of ::1. The hostname portion of the netloc is invalid however, as it doesn't start with [. Shouldn't the parser at least validate that the [ bracket is the first character or directly following a @?

This change caused an issue on our system. We have brackets in a password. Since 3.11.4, urlparse tries to parse an ip address if it finds brackets. The brackets don't even have to be in the correct order.

Running this code:

from urllib.parse import urlparse
urlparse("https://user:some]password[@host.com")

In 3.11.3 it works as expected:

ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')

In 3.11.4 it throws an exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/urllib/parse.py", line 395, in urlparse
    splitresult = urlsplit(url, scheme, allow_fragments)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/urllib/parse.py", line 500, in urlsplit
    _check_bracketed_host(bracketed_host)
  File "/usr/local/lib/python3.11/urllib/parse.py", line 446, in _check_bracketed_host
    ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ipaddress.py", line 54, in ip_address
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: '@host.com' does not appear to be an IPv4 or IPv6 address

This change caused an issue on our system. We have brackets in a password. Since 3.11.4, urlparse tries to parse an ip address if it finds brackets. The brackets don't even have to be in the correct order.

Running this code:
from urllib.parse import urlparse
urlparse("https://user:some]password[@host.com")
In 3.11.3 it works as expected:
ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')

That this worked before was a bug. urlparse() was violating the various and miriad URL standards before; [ and ] are reserved characters with meaning in the netloc portion of a URL (the part between the scheme:// and /path) and must be URL encoded. See the WhatWG living URL standard percent-encoding section; for the userinfo part (username + password), you need to encode:

U+002F (/), U+003A (:), U+003B (;), U+003D (=), U+0040 (@), U+005B ([) to U+005E (^), inclusive, and U+007C (|).
U+003F (?), U+0060 (`), U+007B ({), and U+007D (}) (path percent-encode set).
U+0020 SPACE, U+0022 ("), U+0023 (#), U+003C (<), and U+003E (>) (query percent-encode set).
U+0000 NULL to U+001F INFORMATION SEPARATOR ONE, inclusive, and all codepoints past U+007E (~) (C0 control percent-encode set)

So this works:

>>> from urllib.parse import urlparse
>>> urlparse("https://user:some%5Dpassword%5B@host.com")
ParseResult(scheme='https', netloc='user:some%5Dpassword%5B@host.com', path='', params='', query='', fragment='')
>>> _.password
'some%5Dpassword%5B'

The password is still URL encoded, use urllib.parse.unquote() for that.

Further references:

Wikipedia: URL encoding
RFC 3986, section 2.2 Reserved Characters

Thank you for the clarification! This makes a lot of sense :)

Handle urlparse ValueError on invalid IP addresses in python >= 3.11

I'm also hitting "ValueError: 'xxxxx' does not appear to be an IPv4 or IPv6 address" on code that was working.

In my case I have no control over the input data, it's background noise from the internet that i'm parsing and analyzing. Some of the crap in my dataset is very much not standards compliant. None the less, as it was working and now isn't i figured i should 👍 before i start vendoring old versions of the stdlib into my project 😆

tests: Use valid IPv6 address

test_url.py: handle 3.9 and 3.10 for http://[2a01:5cc0:1:2:3:4] testcase

python3.pkgs.w3lib: fix build with python 3.9 and 3.10

@mcepl

My backport of fix for Python 3.6 (which with minimal changes applies to 3.4 as well, and with a lot of changes, including injecting ipaddress from https://github.com/phihag/ipaddress into the Python standard library, into 2.7) is at openSUSE-Python@4f2496b

Hi @mcepl. I'm currently evaluating how to fix this in older Debian dists for 2.7, and FYI it seems the version from OpenSUSE Leap 15 (150000.74.1) doesn't fully fix it:

$ PYTHONPATH=/root/rpmbuild/BUILD/Python-2.7.18/Lib \
  python2 -c 'import urlparse; print(urlparse.urlparse("https://user:some]password[@host.com"))'
ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')

(expected: ValueError)
It seems CVE-2024-11168-validation-IPv6-addrs.patch fixes the "http://" special case but not the general case (which handles "httpS://"). I didn't check the 3.6 patch closely but it seems it's the opposite situation there.

After fixing the above I also get a couple test_urlparse errors, related to passing non-unicode data to the ipaddress backport. Casting to Unicode ipaddress.ip_address(unicode(hostname)) appears to tame the testsuite.

(expected: ValueError) It seems CVE-2024-11168-validation-IPv6-addrs.patch fixes the "http://" special case but not the general case (which handles "httpS://"). I didn't check the 3.6 patch closely but it seems it's the opposite situation there.

After fixing the above I also get a couple test_urlparse errors, related to passing non-unicode data to the ipaddress backport. Casting to Unicode ipaddress.ip_address(unicode(hostname)) appears to tame the testsuite.

Do you have a corrected patch? I am all ears!

Do you have a corrected patch? I am all ears!

Now yes :)
OpenSUSE patch with corrections: https://salsa.debian.org/lts-team/packages/python2.7/-/blob/08639c6a31c4726e2862d2908a8a805be9c186df/debian/patches/CVE-2024-11168-opensuse.patch
Alternate approach, without embedding (and having to maintain) ipaddress: https://salsa.debian.org/lts-team/packages/python2.7/-/blob/c62e4e05f86c0e14ae6aa405382268d3c97137c6/debian/patches/CVE-2024-11168.patch

On Sat Mar 8, 2025 at 9:20 AM CET, Beuc wrote: Now yes :) OpenSUSE patch with corrections: https://salsa.debian.org/lts-team/packages/python2.7/-/blob/08639c6a31c4726e2862d2908a8a805be9c186df/debian/patches/CVE-2024-11168-opensuse.patch Alternate approach, without embedding (and having to maintain) ipaddress: https://salsa.debian.org/lts-team/packages/python2.7/-/blob/c62e4e05f86c0e14ae6aa405382268d3c97137c6/debian/patches/CVE-2024-11168.patch

Thank you. In the end I went just with the modification of the original patch, I like how similar the patch is on all versions of the interpreter.

[3.11] pythongh-103848: Adds checks to ensure that bracketed hosts fo…

00444: Security fix for CVE-2024-11168

Navigation Menu

Repository navigation

[CVE-2024-11168] urlparse incorrectly retrieves IPv4 and regular name hosts from inside of brackets #103848

Background

The Bug

Your environment

Linked PRs

21 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[CVE-2024-11168] urlparse incorrectly retrieves IPv4 and regular name hosts from inside of brackets #103848

Description

Background

The Bug

Your environment

Linked PRs

Activity

mjpieters commented on Jun 6, 2023

MarcoGlauser commented on Jun 12, 2023

mjpieters commented on Jun 12, 2023

MarcoGlauser commented on Jun 12, 2023

Jc2k commented on Jun 28, 2023

21 remaining items

Beuc commented on Mar 7, 2025

mcepl commented on Mar 8, 2025

Beuc commented on Mar 8, 2025

mcepl commented on Mar 13, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions