Build Your First AI Agent From Scratch (Without Any Framework)

Frameworks like LangChain, LangGraph, CrewAI, and AutoGen are powerful, but they can also hide what’s really happening behind the scenes. If you truly want to master AI agents, you should first build one from scratch. In this article, we’ll create a simple AI agent using pure Python and the OpenAI API (or any compatible LLM API). By the end, you’ll understand the core execution loop that powers nearly every modern AI agent.

Introduction

In the previous article, we explored popular AI agent frameworks and learned how they simplify development.

But here’s an important observation:

Every framework is built on the same fundamental ideas.

Whether you’re using LangChain, CrewAI, LangGraph, or the OpenAI Agents SDK, the underlying workflow is remarkably similar.

User Input
 ↓
Reason
 ↓
Choose Tool
 ↓
Execute Tool
 ↓
Observe Result
 ↓
Reason Again
 ↓
Final Answer

Today, we’ll implement this workflow ourselves.

No framework.

Just Python.

What Are We Building?

Our first AI agent will be able to:

• Receive a user request
• Decide whether it needs a tool
• Execute that tool
• Read the tool’s output
• Generate the final response

It won’t have long-term memory or multiple agents yet.

We’ll add those later.

The Basic Agent Loop

Every AI agent follows the same cycle.

Receive Goal
 ↓
Think
 ↓
Choose Action
 ↓
Execute Action
 ↓
Observe
 ↓
Repeat
 ↓
Finish

Everything else is an enhancement of this loop.

Step 1: Define the Goal

Suppose the user asks:

“What is 245 × 876?”

The agent first receives the request.

user_input = "What is 245 * 876?"

At this point, the agent has only one piece of information:

the user’s goal.

Step 2: Think

The LLM analyzes the request.

Instead of immediately answering, it reasons internally.

Conceptually, the model decides:

This looks like a mathematical calculation.

Using a calculator would be more reliable.

Notice that the agent is deciding what to do, not what to answer.

Step 3: Select a Tool

Suppose we have two tools.

TOOLS = {
    "calculator": calculator,
    "weather": weather_api
}

The model selects one.

Chosen Tool: calculator

This is exactly how modern tool-calling works.

Step 4: Execute the Tool

Now the Python program not the LLM calls the function.

result = calculator("245 * 876")

The calculator returns

214620

The model never performs the arithmetic itself.

It simply receives the answer.

Step 5: Observe

The tool returns information.

Get MD Soyeb Hoque’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The agent updates its understanding.

Tool Result

  ↓

214620

Now the agent has enough information to respond confidently.

Step 6: Generate the Final Response

The LLM combines:

• the original question
• the tool output

to produce:

“245 × 876 equals 214,620.”

Notice that reasoning and execution were separated.

Putting Everything Together

The complete workflow now looks like this.

User
↓
LLM
↓
Select Tool
↓
Python Function
↓
Tool Result
↓
LLM
↓
Final Response

This simple loop forms the foundation of almost every AI agent.

A Simple Python Skeleton

Without worrying about API details, an agent loop conceptually looks like this.

while True:
   user_request = get_user_input()
   decision = llm(user_request)

   if decision.requires_tool:
       tool_result = run_tool(decision.tool)
       answer = llm(user_request, tool_result)
   else:
       answer = decision.response
   print(answer)

Although simplified, this captures the essence of agent execution.

Why This Works

The language model is not responsible for everything.

Instead:

The LLM specializes in:

• understanding language
• reasoning
• planning
• selecting tools

Python specializes in:

• executing code
• calling APIs
• accessing databases
• reading files
• performing calculations

Each component focuses on what it does best.

What Happens Internally?

When you chat with an AI agent, the interaction may look like this.

User
↓
LLM
↓
Tool Request
↓
Python
↓
API
↓
Result
↓
LLM
↓
Final Answer

Most of these intermediate steps happen automatically.

The user only sees the final response.

Limitations of Our First Agent

Our simple agent still has several limitations.

It cannot:

• remember previous conversations
• search documents
• plan long workflows
• recover from failures
• collaborate with other agents
• execute multiple tools in sequence

Fortunately, each limitation can be solved by adding one new capability at a time.

The Road Ahead

Over the next few articles, we’ll gradually transform this simple agent into a production-ready AI system.

We’ll add:

• Memory
• Retrieval-Augmented Generation (RAG)
• Multiple tools
• Planning
• Reflection
• Multi-agent collaboration
• Streaming responses
• Human approval
• Production deployment

Each feature builds naturally on today’s basic loop.

Key Takeaways

• Every AI agent is built around a simple execution loop.
• The LLM decides what to do; external tools perform the actual work.
• Tool execution happens outside the language model.
• Separating reasoning from execution makes agents more reliable.
• Modern frameworks automate this process but do not fundamentally change it.
• Understanding this loop is the first step toward building advanced AI systems.

What’s Next?

Our first agent can now reason and use a single tool.

But real-world agents rarely use just one tool.

A software engineering agent might need to:

• search documentation,
• execute Python,
• query a database,
• read local files,
• call external APIs,
• and generate images,

all within the same conversation.

In the next article, we’ll extend our agent to support multiple tools, build a simple tool registry, and implement dynamic tool selection the same idea used by modern production AI agents.