Skip to content

Add tunnel CONNECT response headers to httplib / http.client #69152

Closed
Closed
Add tunnel CONNECT response headers to httplib / http.client#69152
Labels
3.12only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@thomasbelhalfaoui
thomasbelhalfaoui
mannequin
opened on Aug 30, 2015
BPO 24964
Nosy @terryjreedy, @vadmium, @OneMoreZanuda
PRs
  • gh-69152: Add _proxy_response_headers attribute to HTTPConnection #26152
    		•
    Files
  • httplib.py: Proposed patch for httplib.py
  • http-detach.patch: HTTPConnection.detach() implementation only
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2015-08-30.17:41:50.140>
labels = ['type-feature', 'library', '3.11']
title = 'Add tunnel CONNECT response headers to httplib / http.client'
updated_at = <Date 2021-05-15.21:19:25.060>
user = 'https://bugs.python.org/thomasbelhalfaoui'

    bugs.python.org fields:

    activity = <Date 2021-05-15.21:19:25.060>
actor = 'alexey.namyotkin'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2015-08-30.17:41:50.140>
creator = 'thomas.belhalfaoui'
dependencies = []
files = ['40301', '40371']
hgrepos = []
issue_num = 24964
keywords = ['patch']
message_count = 10.0
messages = ['249361', '249373', '249741', '249803', '249897', '249906', '249996', '250085', '393729', '393730']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'martin.panter', 'thomas.belhalfaoui', 'alexey.namyotkin']
pr_nums = ['26152']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24964'
versions = ['Python 3.11']

    Linked PRs

    Activity

    1. thomasbelhalfaoui commented on Aug 30, 2015

      @thomasbelhalfaoui
      thomasbelhalfaouimannequin
      on Aug 30, 2015
      MannequinAuthor

      When using httplib / http.client to connect to an HTTPS website through a proxy (by making a tunnel with a CONNECT request), there is no way to retrieve the HTTP headers which the proxy sends back in response to that CONNECT request.

      This becomes a problem when using rotating proxy providers like ProxyMesh, who send useful information in those headers (for instance, "X-ProxyMesh-IP" contains the IP address of the proxy, which is necessary to keep the same address throughout the session).

      It would be nice to save those headers in a property of the HTTPConnection class (e.g. self._tunnel_response_headers), which would be set up inside the _tunnel method (as proposed in the attached patch, lines 748 and 827-831). This would allow to get the headers back and/or pass them to a higher-level library (such as requests).

    2. thomasbelhalfaoui
      added
      stdlibStandard Library Python modules in the Lib/ directory
      type-featureA feature request or enhancement
      on Aug 30, 2015

    3. vadmium commented on Aug 30, 2015

      @vadmium
      vadmium
      on Aug 30, 2015
      Member

      Such a change would involve adding a new API, so should go into a new version of Python.

      Thomas: a diff rather than a full copy of the changed file would be more convenient. Also, if this gets accepted, test cases and documentation would be needed.

      It is also useful to get the header of an unsuccessful CONNECT response. For example, see bpo-7291, where the Proxy-Authenticate header of the proxy’s 407 response needs to be accessible. In that issue, I started working on a patch tht may also be useful here. From memory, usage would be a bit like this:

      proxy_conn = HTTPConnection("proxy")
proxy_conn.request("CONNECT", "website:443")
proxy_resp = proxy_conn.getresponse()
if proxy_resp.status == PROXY_AUTHENTICATION_REQUIRED:
    # Handle proxy_resp.msg["Proxy-Authenticate"]
    ...
# Handle proxy_resp.msg["X-ProxyMesh-IP"]
...
tunnel = proxy_conn.detach()  # Returns socket and any buffered data
website_conn = HTTPSConnection("website", tunnel=tunnel)
website_conn.request("GET", "/")
...
website_conn.close()

      Thomas, let me know if this would be useful for you, and I can try and dig up my patch.

    4. thomasbelhalfaoui commented on Sep 4, 2015

      @thomasbelhalfaoui
      thomasbelhalfaouimannequin
      on Sep 4, 2015
      MannequinAuthor

      Martin: Thanks for your quick answer (and sorry for sending the whole file) !
      I think it is indeed a good idea to detach the proxy connection and treat it as any other connection, as you did in your patch. It would be great if you would be able to dig it up !

    5. terryjreedy commented on Sep 4, 2015

      @terryjreedy
      terryjreedy
      on Sep 4, 2015
      Member

      Thomas, please sign a contributor agreement for your patches to be considered.
      https://www.python.org/psf/contrib/
      https://www.python.org/psf/contrib/contrib-form/

    6. vadmium commented on Sep 5, 2015

      @vadmium
      vadmium
      on Sep 5, 2015
      Member

      This is the patch I had in mind. It looks like it only implements the detach() method, so we would still need to add support for passing in the tunnel details to the HTTPSConnection constructor.

      This patch would allow doing stuff at a lower level than the existing tunnel functionality. The patch includes a test case for getting the proxy’s response header fields, and another test case illustrating how a plain text HTTP 2 upgrade could work.

    7. thomasbelhalfaoui commented on Sep 5, 2015

      @thomasbelhalfaoui
      thomasbelhalfaouimannequin
      on Sep 5, 2015
      MannequinAuthor

      Terry: Thanks for the form, I just filled it.

      Martin: Thanks for sending your patch. I will dive into it, and try to figure out how to add support for passing in the tunnel details to the HTTPSConnection constructor.

    8. thomasbelhalfaoui commented on Sep 6, 2015

      @thomasbelhalfaoui
      thomasbelhalfaouimannequin
      on Sep 6, 2015
      MannequinAuthor

      Martin, I went through your patch and made some simple tests, and I have a couple of questions.

      1. When I run the following code, I get a "Bad file descriptor" :
      conn = httplib.HTTPConnection("uk.proxymesh.com", 31280)
conn.set_tunnel("www.google.com", 80)
conn.request("GET", "/")
resp = conn.getresponse()
print(resp.read())

      So I tweaked the "getresponse" function so that it does not call "self.close()" (i.e. the connection stays open after the CONNECT request) in that case, and it seems to works fine.

      1. I added "self.sock, _ = tunnel" in HTTPConnection constructor, to try your use case, but I get "http.client.RemoteDisconnected: Remote end closed connection without response".

      Do you think it makes sense or am I missing something ?

    9. vadmium commented on Sep 7, 2015

      @vadmium
      vadmium
      on Sep 7, 2015
      Member

      1. The real problem is when _tunnel() internally calls getresponse(), it notices the connection cannot be reused for another request, and closes the socket object. Perhaps I should rethink my logic; maybe move sock and detach() to HTTPResponse.

      2. With some rough experimentation, passing tunnel through the HTTPConnection (plain text HTTP) constructor seems to work for me. However if you meant HTTPSConnection (over TLS) instead, you will probably need to manually do the wrap_socket() step. Maybe that’s why your connection is being dropped.

    10. terryjreedy commented on May 15, 2021

      @terryjreedy
      terryjreedy
      on May 15, 2021
      Member

      Alexey, to repeat what I said to Thomas above: please sign a contributor agreement for your patches to be considered.
      https://www.python.org/psf/contrib/
      https://www.python.org/psf/contrib/contrib-form/

    11. terryjreedy
      added
      3.11only security fixes
      on May 15, 2021

    12. OneMoreZanuda commented on May 15, 2021

      @OneMoreZanuda
      OneMoreZanudamannequin
      on May 15, 2021
      Mannequin

      Thanks, Terry. I signed it.

    13. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    14. orsenthil
      added a commit that references this issue on May 5, 2023

      gh-69152: Add _proxy_response_headers attribute to HTTPConnection (#2…

      1afe0e0
    15. carljm
      added a commit that references this issue on May 5, 2023

      Merge branch 'main' into inlinecomp2

      baacf5f
    16. jbower-fb
      added a commit that references this issue on May 8, 2023

      pythongh-69152: Add _proxy_response_headers attribute to HTTPConnecti…

      ef989ab
    17. arhadthedev
      added
      3.13only security fixes
      and removed
      3.11only security fixes
      on May 11, 2023
    18. gpshead
      added a commit that references this issue on May 16, 2023

      gh-69152: add method get_proxy_response_headers to HTTPConnection cla…

      85ec192
    19. arhadthedev
      added
      3.12only security fixes
      and removed
      3.13only security fixes
      on May 16, 2023

    20. gpshead commented on May 16, 2023

      @gpshead
      gpshead
      on May 16, 2023
      Member

      thanks for the contribution!

    21. gpshead
      closed this as completedon May 16, 2023
    22. carljm
      added a commit that references this issue on May 16, 2023

      Merge branch 'main' into supermon

      a6dd626
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.12only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions