Build an AI Agent Harness from Scratch
A hands-on Python course for understanding what agent frameworks actually do
AI agents can seem mysterious when introduced through large frameworks. Behind the abstractions, however, an agent is built from a small set of understandable components: an LLM call, a loop, conversation history, tools, and rules for deciding what happens next.
Agent Harness from Scratch is a hands-on course that builds these components incrementally using Python and the OpenAI-compatible API.
https://github.com/jiminelee/agent-harness-course
Get Jimin Lee’s stories in your inbox
Join Medium for free to get updates from this writer.
The course starts with a single LLM request and gradually adds:
- An agent loop and conversation history
- Function calling and tool registries
- Memory compression
- Planning and task decomposition
- Error recovery and reflection
- Concurrent tool execution with
asyncio - Guardrails and human approval
- Structured tracing and observability
- Multi-agent orchestration
- Packaging, caching, cost tracking, and prompt versioning
Each module is a standalone Python program. The code intentionally avoids frameworks such as LangChain or AutoGen so you can see the mechanics directly. Implementations favor clarity over production-level error handling and abstraction.
The default setup uses Ollama and gemma4:e4b, so the course can run locally. Because it uses the OpenAI Python SDK and an OpenAI-compatible request format, it can also be adapted to OpenAI and other compatible providers.
This course is intended for developers who are already comfortable with Python and basic LLM API calls. Following the full project should take roughly 10–15 hours, depending on how many experiments you try.
The final module also provides a roadmap explaining what would need to change before using these ideas in a production system.
Fun fact: the course itself was developed with help from an AI agent. AI accelerated coding, review, and documentation — but the goals, curriculum, architecture, constraints, and evaluation still required human judgment.
That is also the central lesson of the project: a capable model is only one component. Reliable agent behavior comes from the system designed around it.
