DEV Community

Cover image for Comparing requests, aiohttp, and httpx: Which HTTP client should you use?
Leapcell
Leapcell

Posted on

Comparing requests, aiohttp, and httpx: Which HTTP client should you use?

#python #django #flask #fastapi

Image description

Leapcell: The Best of Serverless Web Hosting

Evaluation of Python HTTP Client Libraries: requests, aiohttp and httpx

Among the rich variety of Python HTTP client libraries, the most well-known ones are requests, aiohttp and httpx. Without the help of other third-party libraries, requests can only send synchronous requests; aiohttp can only send asynchronous requests; while httpx has the ability to send both synchronous and asynchronous requests.

Concepts of Synchronous and Asynchronous Requests

  • Synchronous Requests: In code running in a single process and single thread, after initiating a request, it is impossible to initiate the next request until the return result is received.
  • Asynchronous Requests: In code running in a single process and single thread, after initiating a request, during the time waiting for the website to return the result, more requests can be sent.

Shallow Evaluation: Performance Comparison of Sending Multiple GET Requests

Although the test results are related to the network speed, testing in the same time period and on the same network can still reveal the performance differences among these libraries.

Sending Requests with requests 

import requests

url = 'https://www.leapcell.io/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

def main():
    res = requests.get(url, headers=headers)
    print(res.status_code)

if __name__ == '__main__':
    main()
Enter fullscreen mode Exit fullscreen mode

Sending Requests with httpx

Synchronous Requests 

import httpx

url = 'https://www.leapcell.io/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

def main():
    res = httpx.get(url, headers=headers)
    print(res.status_code)

if __name__ == '__main__':
    main()
Enter fullscreen mode Exit fullscreen mode

The synchronous mode of httpx has a code overlap rate of up to 99% with that of requests. Just replace requests with httpx and the code can run normally.

Asynchronous Requests 

import httpx
import asyncio

url = 'https://www.leapcell.io/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

async def main():
    async with httpx.AsyncClient() as client:
        resp = await client.get(url, headers=headers)
        print(resp.status_code)

if __name__ == '__main__':
    asyncio.run(main())
Enter fullscreen mode Exit fullscreen mode

Sending Requests with aiohttp 

import asyncio
import aiohttp

url = 'https://www.leapcell.io/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

async def main():
    async with aiohttp.ClientSession() as client:
        async with client.get(url, headers=headers) as resp:
            print(await resp.text())
            print(resp.status)

if __name__ == '__main__':
    asyncio.run(main())
Enter fullscreen mode Exit fullscreen mode

The code of aiohttp has an approximate 90% code overlap rate with the asynchronous mode code of httpx. It just replaces AsyncClient with ClientSession.

Performance Test: Time Consumption of Sending 100 Requests

requests

Without Keeping the Connection 

import time
import requests

url = 'https://www.leapcell.io/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

def make_request():
    resp = requests.get(url, headers=headers)
    print(resp.status_code)

def main():
    start = time.time()
    for _ in range(100):
        make_request()
    end = time.time()
    print(f'sent 100 requests, cost：{end - start}')

if __name__ == '__main__':
    main()
Enter fullscreen mode Exit fullscreen mode

Time consumption for sending 100 requests: 10.295854091644287

Keeping the Connection 

import time
import requests

session = requests.session()

url = 'https://www.leapcell.io/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

def make_request():
    resp = session.get(url, headers=headers)
    print(resp.status_code)

def main():
    start = time.time()
    for _ in range(100):
        make_request()
    end = time.time()
    print(f'sent 100 requests, cost：{end - start}')

if __name__ == '__main__':
    main()
Enter fullscreen mode Exit fullscreen mode

Time consumption for sending 100 requests: 4.679062128067017, which is obviously about 6 seconds faster.

httpx

Synchronous Mode 

import time
import httpx

url = 'https://www.leapcell.io/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

def make_request():
    resp = httpx.get(url, headers=headers)
    print(resp.status_code)

def main():
    start = time.time()
    for _ in range(100):
        make_request()
    end = time.time()
    print(f'sent 100 requests, cost：{end - start}')

if __name__ == '__main__':
    main()
Enter fullscreen mode Exit fullscreen mode

Time consumption for sending 100 requests: 16.60569405555725

Asynchronous Mode: Create httpx.AsyncClient() Only Once 

import httpx
import asyncio
import time

url = 'https://www.leapcell.io/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

async def make_request(client):
    resp = await client.get(url, headers=headers)
    print(resp.status_code)

async def main():
    async with httpx.AsyncClient() as client:
        start = time.time()
        tasks = [asyncio.create_task(make_request(client)) for _ in range(100)]
        await asyncio.gather(*tasks)
        end = time.time()
    print(f'sent 100 requests, cost：{end - start}')

if __name__ == '__main__':
    asyncio.run(main())
Enter fullscreen mode Exit fullscreen mode

Time consumption for sending 100 requests: 4.359861135482788

Asynchronous Mode: Create httpx.AsyncClient() Every Time 

import httpx
import asyncio
import time

url = 'https://www.leapcell.io/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

async def make_request():
    async with httpx.AsyncClient() as client:
        resp = await client.get(url, headers=headers)
        print(resp.status_code)

async def main():
    start = time.time()
    tasks = [asyncio.create_task(make_request()) for _ in range(100)]
    await asyncio.gather(*tasks)
    end = time.time()
    print(f'sent 100 requests, cost：{end - start}')

if __name__ == '__main__':
    asyncio.run(main())
Enter fullscreen mode Exit fullscreen mode

Time consumption for sending 100 requests: 6.378381013870239

aiohttp

Create aiohttp.ClientSession() Only Once 

import time
import asyncio
import aiohttp

url = 'https://www.leapcell.io/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

async def make_request(client):
    async with client.get(url, headers=headers) as resp:
        print(resp.status)

async def main():
    async with aiohttp.ClientSession() as client:
        start = time.time()
        tasks = [asyncio.create_task(make_request(client)) for _ in range(100)]
        await asyncio.gather(*tasks)
        end = time.time()
    print(f'sent 100 requests, cost：{end - start}')

if __name__ == '__main__':
    asyncio.run(main())
Enter fullscreen mode Exit fullscreen mode

Time consumption for sending 100 requests: 2.235464334487915

Create aiohttp.ClientSession() Every Time 

import time
import asyncio
import aiohttp

url = 'https://www.leapcell.io/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
}

async def make_request():
    async with aiohttp.ClientSession() as client:
        async with client.get(url, headers=headers) as resp:
            print(resp.status)

def main():
    start = time.time()
    tasks = [asyncio.ensure_future(make_request()) for _ in range(100)]
    loop = asyncio.get_event_loop()
    loop.run_until_complete(asyncio.wait(tasks))
    end = time.time()
    print(f'sent 100 requests, cost：{end - start}')

if __name__ == '__main__':
    main()
Enter fullscreen mode Exit fullscreen mode

Time consumption for sending 100 requests: 2.6662471294403076

Speed Ranking for 100 Requests

aiohttp (create client only once) > aiohttp (create client every time) > httpx asynchronous (create client only once) > requests.session > httpx asynchronous (create client every time) > requests

Conclusion

  • Small Number of Requests: If only a few requests are to be sent, using requests or the synchronous mode of httpx makes the code the simplest.
  • Connection Management of requests: Whether requests creates a session to keep the connection or not makes a big difference in speed. In the absence of anti-crawling measures, if only speed is pursued, it is recommended to use requests.session().
  • Mixed Request Requirements: If a large number of requests need to be sent, and some require synchronous requests while some require asynchronous requests, then using httpx is the most convenient.
  • High-speed Request Requirements: If a large number of requests need to be sent and the fastest speed is pursued, then using aiohttp is the best choice.

Leapcell: The Best of Serverless Web Hosting

Finally, I would like to recommend a platform that is most suitable for deploying Python services: Leapcell

Image description

🚀 Build with Your Favorite Language

Develop effortlessly in JavaScript, Python, Go, or Rust.

🌍 Deploy Unlimited Projects for Free

Only pay for what you use—no requests, no charges.

⚡ Pay-as-You-Go, No Hidden Costs

No idle fees, just seamless scalability.

Image description

📖 Explore Our Documentation

🔹 Follow us on Twitter: @LeapcellHQ

Top comments (0)