Diagnose slow Python code

Performance optimization is one of the most satisfying challenges in programming, but before you can optimize, you need to identify the bottlenecks. Recently, I set out to learn how to profile Python code and experimented with a simple project.

The Problem: A Simple Endpoint

I created a simple Python function to read a list of websites from a file, fetch their HTML content, and count occurrences of https:// and http://. Here’s the initial code:

import requests
import re
import time

def slow_endpoint_count_https():
    with open("websitesList.txt", "r", encoding="utf-8") as f:
        urls = [line.strip() for line in f.readlines()]
    htmls = []
    for url in urls:
        htmls.append(requests.get(url).text)
    count_https = 0
    count_http = 0
    for html in htmls:
        count_https += len(re.findall("https://", html))
        count_http += len(re.findall("http://", html))
    print("finished parsing")
    time.sleep(2.0)
    print(f"{count_https=}")
    print(f"{count_http=}")
    print(f"{count_https/count_http=}")

You can quickly notice the bad code in this snippet, but we need to pinpoint exactly where the time was being spent and learn how to optimize it.

Step 1: Profiling the Code

To find the bottlenecks, I used Python’s built-in cProfile module. This tool records the execution time of every function call, making it easy to see which parts of your code are slow.

Here’s how I profiled the function:

import cProfile
import pstats

def main():
    with cProfile.Profile() as pr:
        slow_endpoint_count_https()
    stats = pstats.Stats(pr)
    stats.sort_stats(pstats.SortKey.TIME)
    stats.print_stats()

if __name__ == "__main__":
    main()

After running this, I found that the majority of the time was spent in requests.get calls and regular expression matching.

Each request blocked the next, resulting in high latency for a larger list of URLs.

Step 2: Visualizing the Results with SnakeViz

While cProfile outputs detailed stats, they’re not always easy to interpret. That’s where SnakeViz comes in. It’s a visualization tool that turns profiling data into interactive flame graphs.

Get Matheus Fontana’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

To generate the flame graph, I first saved the profiling stats to a file:

stats.dump_stats(filename="profiling_stats.prof")

Then, I used SnakeViz to visualize the data:

snakeviz profiling_stats.prof

The flame graph clearly showed that the requests.get calls dominated the runtime. Here’s a snapshot of the results:

Step 3: Optimizing with Asyncio

To improve performance, I rewrote the function to use asynchronous programming with aiohttp. This allowed multiple HTTP requests to be made concurrently, drastically reducing the overall execution time.

Here’s the optimized code:

import asyncio
import aiohttp
import re

async def improved_endpoint_count_https():
    with open("websitesList.txt", "r", encoding="utf-8") as f:
        urls = [line.strip() for line in f.readlines()]
    htmls = []
    async with aiohttp.ClientSession() as session:
        tasks = [session.get(url) for url in urls]
        responses = await asyncio.gather(*tasks)
        for response in responses:
            htmls.append(await response.text())
    count_https = sum(len(re.findall("https://", html)) for html in htmls)
    count_http = sum(len(re.findall("http://", html)) for html in htmls)
    print("finished parsing")
    print(f"{count_https=}")
    print(f"{count_http=}")
    print(f"{count_https/count_http=}")

This version uses aiohttp for non-blocking HTTP requests and asyncio.gather to execute them concurrently.

Step 4: Comparing Results

I profiled the optimized code using the same cProfile and SnakeViz workflow. The flame graph for the improved code looked much better:

Execution time dropped significantly to from 14.5s to 3.22s.

Without tools like cProfile and SnakeViz, it’s hard to know where your code is slow when working with real production code. Profiling gives you the data you need to make informed optimizations.

Flame graphs make it easy to identify bottlenecks and communicate findings to others.

🧑‍💻 Check out the complete code: GitHub Repo