Building Smarter Python Workflows With Async Programming
How I Learned to Scale Python Apps Beyond Their Limits
💼 High-Paying Tech Roles — Hiring Now! 🚀
Earn $30–$120/hr working on exciting remote & full-time projects worldwide.
🔥 Roles Available:
✅ Frontend & Backend Developers
✅ Full Stack & Mobile Engineers
✅ UI/UX Designers
✅ DevOps, AI & Data Experts
Positions are limited — top companies are hiring fast!
👉 Apply Now — the fastest applicants secure the best roles!
For years, I treated Python like a polite, single-threaded coworker — reliable but slow at multitasking. Then I discovered asynchronous programming. It felt like I unlocked a hidden level in the game of Python development.
In this article, I’ll take you through how I leveled up my automation and web scraping workflows using Python’s asyncio, aiohttp, and async/await syntax. If you’ve ever hit performance walls while processing I/O-bound tasks, this one’s for you.
1) Understanding Why Sync Code Fails at Scale
Imagine you’re downloading 1000 URLs sequentially. Traditional synchronous Python would process them one at a time — which means if each takes 1 second, you’ll be done in roughly 1000 seconds. Brutal.
import requests
urls = ["https://example.com/page1", "https://example.com/page2", "https://example.com/page3"]
for url in urls:
response = requests.get(url)
print(f"Fetched {url}: {len(response.text)} characters")This works, but your CPU spends most of that time waiting. Waiting for the network. Waiting for responses. Waiting for data that could easily be fetched in parallel.
That’s where asynchronous I/O changes everything.
2) Introducing Asyncio — The Backbone of Concurrency
asyncio lets Python handle I/O operations concurrently — without needing threads or processes.
Here’s what the asynchronous version of the above example looks like:
import aiohttp
import asyncio
urls = ["https://example.com/page1", "https://example.com/page2", "https://example.com/page3"]
async def fetch(session, url):
async with session.get(url) as response:
text = await response.text()
print(f"Fetched {url}: {len(text)} characters")
async def main():
async with aiohttp.ClientSession() as session:
tasks = [fetch(session, url) for url in urls]
await asyncio.gather(*tasks)
asyncio.run(main())With this, your 1000 URLs no longer take 1000 seconds — they might finish in 10 or less.
Get Code with Margaret’s stories in your inbox
Join Medium for free to get updates from this writer.
Pro tip:
Async is not parallel. It’s cooperative multitasking — meaning each task runs until it hits an I/O wait, then yields control.
3) Mixing Sync and Async Code the Right Way
Sometimes, you’ll need to run blocking code in an async workflow (like reading from disk or processing data). To handle this gracefully, use asyncio.to_thread().
import asyncio
import time
def cpu_bound_task(n):
time.sleep(1)
return n ** 2
async def main():
results = await asyncio.gather(*[asyncio.to_thread(cpu_bound_task, i) for i in range(5)])
print(results)
asyncio.run(main())This lets your CPU-heavy function run in a separate thread pool, freeing up the event loop to continue processing.
4) Real-World Example: Asynchronous Web Scraper
Let’s combine these ideas into a real, production-style use case.
import aiohttp
import asyncio
from bs4 import BeautifulSoup
urls = [f"https://example.com/page{i}" for i in range(1, 51)]
async def fetch_and_parse(session, url):
async with session.get(url) as response:
html = await response.text()
soup = BeautifulSoup(html, "html.parser")
title = soup.find("title").text if soup.find("title") else "No title"
print(f"{url} -> {title}")
async def main():
async with aiohttp.ClientSession() as session:
tasks = [fetch_and_parse(session, url) for url in urls]
await asyncio.gather(*tasks)
asyncio.run(main())What used to take minutes can now run in seconds.
5) Handling Errors and Timeouts Gracefully
Async code runs fast — sometimes too fast — and you’ll want to handle edge cases properly.
import aiohttp
import asyncio
async def fetch_safe(session, url):
try:
async with session.get(url, timeout=5) as response:
return await response.text()
except asyncio.TimeoutError:
print(f"Timeout: {url}")
except aiohttp.ClientError as e:
print(f"Request failed: {url} -> {e}")
async def main():
urls = ["https://example.com" for _ in range(10)]
async with aiohttp.ClientSession() as session:
await asyncio.gather(*[fetch_safe(session, u) for u in urls])
asyncio.run(main())This keeps your script resilient — no crashes, no surprises.
6) Async Database Access
Most developers stop at networking, but databases can benefit from async patterns too. Libraries like databases and asyncpg make PostgreSQL access blazing fast.
import asyncio
import asyncpg
async def main():
conn = await asyncpg.connect(user='postgres', password='12345', database='testdb', host='127.0.0.1')
await conn.execute('''CREATE TABLE IF NOT EXISTS users(id SERIAL PRIMARY KEY, name TEXT)''')
await conn.execute('''INSERT INTO users(name) VALUES($1)''', 'Alice')
rows = await conn.fetch('SELECT * FROM users')
print(rows)
await conn.close()
asyncio.run(main())This approach can handle thousands of concurrent queries efficiently without blocking.
7) Async Task Scheduling with Background Jobs
You can use aiojobs to schedule background tasks and offload work to async workers.
import asyncio
from aiojobs import create_scheduler
async def background_task(name):
await asyncio.sleep(2)
print(f"Completed {name}")
async def main():
scheduler = await create_scheduler()
for i in range(5):
await scheduler.spawn(background_task(f"Task {i}"))
await asyncio.sleep(3)
asyncio.run(main())Perfect for microservices that need to juggle multiple jobs simultaneously.
8) Integrating Async with FastAPI
FastAPI is designed to work seamlessly with async functions — making it the perfect web framework for high-performance Python apps.
from fastapi import FastAPI
import httpx
app = FastAPI()
@app.get("/fetch")
async def fetch_data():
async with httpx.AsyncClient() as client:
response = await client.get("https://api.github.com")
return response.json()This allows for lightning-fast APIs that handle thousands of concurrent requests without breaking a sweat.
9) Final Thoughts: Async as a Superpower
Async isn’t just about speed — it’s about efficiency. It’s about writing code that doesn’t waste time waiting. Once you grasp the event loop and await, your Python applications stop being “good enough” and start being scalable.
“Premature optimization is the root of all evil,” Knuth once said. True — but asynchronous programming isn’t premature. It’s strategic optimization.
If you want to build tools that run faster, scale better, and feel professional, async programming is your next big unlock.
Thank you for being a part of the community
Before you go:
👉 Be sure to clap and follow the writer ️👏️️
👉 CodeToDeploy Tech Community is live on Discord — Join now!
👉 Follow our publication, CodeToDeploy
Note: This Post may contain affiliate links.
