requests::Response::content hangs forever for Socket::read

[der-brecher]

Port, board and/or hardware

ESP32/ESP8266/Unix

MicroPython version

MicroPython v1.23.0 on 2024-06-02; ESP module with ESP8266

Reproduction

I experienced the issue with an ESP running micropython wanting to access an AhoyDTU via simple HTTP GET. After updating the AhoyDTU (latest (ahoy_v0.8.140) https://fw.ahoydtu.de/fw/release%2Fahoy_v0.8.140/) the call got stuck and the ESP hangs there forever.
As other HTTP clients do not have any issues requesting the server, I see the issue independent from AhoyDTU and request micropython to be more robust here.

Expected behaviour

No response

Observed behaviour

Depending of the server's message requests::Response::content hangs as the internal Socket::read() blocks and hangs forever. As the documentation for Socket::read() says it waits until EOF is recognized I assume this is the issue here.

Additional Information

In case a Content-Length is being transmitted Response::content should only read the amount of bytes as denoted by Content-Length. I case the message is being streamed, read() shall be used to read until EOF is recognized.

I was able to fix this locally by doing the following:
Try to read Content-Length from HTTP headers and conditionally set it at the response object. In case Response::content is requested and the socket is used to read the payload, check if Content-Length is available and conditionally only read the denoted amount of bytes via Socket::read(int) which does not wait for EOF. In case no Content-Length is available, do the Socket::read() as before.
fix-requests-hangs-on-read.zip

Code of Conduct

Yes, I agree

[dpgeorge]

Thanks for the report.

To properly investigate this we would need instructions on how to reproduce the issue. Eg how to set up the server, and what code to run on the client (what HTTP request).

You could also possibly use wireshark to trace the network traffic between the server and client and that may help debug the issue.

Do you see the socket read hanging if the client is an ESP32 board? Can you try using normal Python to run the client code?

[der-brecher]

I only experience this issue when requesting the current Ahoy-DTU running on an ESP32. The request is made from ESP32 or ESP8266 running the latest Micropython. When I interrupt the execution it gives me the stack trace pointing to read():

 Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/__init__.py", line 33, in json
  File "requests/__init__.py", line 20, in content
KeyboardInterrupt: 

The request is the following but all REST calls on ahoy-dtu behave the same:
http://ahoy-dtu/api

via:

import requests
print(requests.get('http://ahoy-dtu/api').json())

The issue does not appear when run on Python.
FYI: I thought it did not work for MicroPython v1.17+ds-1.1ubuntu2 either. I retried it now to capture some wireshark protocols but to my surprise it works for the Ubuntu version. I also double-checked the ESPs but the issue is still there. Maybe this helps

[dpgeorge]

I was able to reproduce the problem, using the following steps:

1. Get Ahoy-DTU running on an ESP32 (flashing bootloader.bin, partitions.bin, and the main firmware using esptool.py, then configuring it through it's AP HTML interface). I had an nRF23L04+ attached but I don't think that's necessary to reproduce the issue.
2. Try to do a request of the /api URL on the device. With Python it works, with unix MicroPython it hangs forever trying to read the HTTP payload.

The issue is in the Ahoy-DTU web server code: it's responding with a HTTP 1.0 header but not following the HTTP 1.0 specification. In particular it's sending Content-Length in the headers and not closing the connection after sending all the JSON payload (it's assuming the request was HTTP 1.1 when it was not). So the client is hanging forever waiting to read more data, waiting for the server to close the connection.

The possible fixes are:

1. Get Ahoy-DTU to support HTTP 1.0.
2. Get Ahoy-DTU to reject HTTP 1.0 requests (then MicroPython would at least raise an error trying to access the /api URL).
3. Use MicroPython's aiohttp library to do the request, because that supports HTTP 1.1 (this requires using asyncio in your app).
4. Write your own code to do the requests (this is not that difficult, see examples/network/http_client.py in this repo).
5. Wait for MicroPython's requests library to support HTTP 1.1. See Change HTTP/1.0 to HTTP/1.1 in requests Python module micropython-lib#861

[der-brecher]

Great - thank you for your investigations!