Beyond LLMs: The Rise of AI Agents
Agency has been a hot topic for a long time. Researchers are still debating whether we, as humans, truly have self-agency or free will. Some recent research even suggests we don’t¹. While not comparable to comprehending the human brain, understanding the concept of AI Agency presents its own set of challenges. With the current focus on AI Agents, I’d like to share my perspective on them.
Throughout human history, we’ve seen numerous revolutionary optimizations, such as the industrial revolution, assembly lines, and scripting-based automation. However, we have only recently begun to consider delegating our thinking process to a non-human entity. This is where AI Agents come into play.
How Did We Get Here ?
Machine Learning and AI have been part of our lives for a long time, but their audience was mostly limited to technical people, such as data scientists. This changed with ChatGPT. Nowadays, anyone can interact with AI systems using the power of LLMs. This sparked widespread interest in various applications of LLMs, with one of the most notable being the so-called AI Agents.
An AI Agent is a piece of software that can autonomously work toward an objective. It can take actions using the available tools. There are many ways to categorize AI Agents, yet I prefer categorizing them based on their capabilities:
- Retrieval-based AI Agents: These simpler agents are built solely around LLMs, without the need for other interactions. One exception to this is Retrieval-Augmented Generation (RAG), where the knowledge base of an LLM is externalized using vector search. These agents are great for data retrieval and search. Since I have a particular interest in retrieval and search, I will dive into their details in another post.
- Tool-based AI Agents: These are the agents which does not only need LLMs but also other interactions such as web search, sending emails, compiling or testing a code piece. For many AI enthusiasts, these are the true AI Agents. While some LLMs natively support tool calling², the others need extra configuration for structured outputs³.
What are the main components?
The core components of an AI Agent are simply an LLM and tools. However, the devil is in the details when it comes to building the right architecture.
LLMs are used for multiple purposes in agent architectures, such as:
- Query Planning: Creating a strategy for subsequent operations.
- Aggregating Results: Combining data from various sources.
- Routing: Directing tasks to the appropriate tools or processes.
An AI Agent’s capabilities are only limited by the provided tools. Here are some example tools:
- Weather Retrieval: A Python function that retrieves up-to-date weather information, allowing the agent to provide relevant environmental data.
- Data Retrieval: A RAG-based retrieval engine that helps the agent access and synthesize information from large knowledge bases.
- Agent Collaboration: Using another agent as a tool, enabling multi-agent collaboration and more complex task handling.
How to improve the performance ?
As AI Agents are still in their infancy, there is significant room for improvement. However, there are already some proven best practices.
Choosing the Right LLM
LLMs with more parameters are generally more performant. Yet, depending on the objective and desired resource consumption, consulting relevant benchmarks⁴ can provide clarity.
Get Ersin Bayraktar’s stories in your inbox
Join Medium for free to get updates from this writer.
Moreover, recent work from DeepMind researchers suggests that improving test-time compute can be more effective than increasing the parameter count⁵. This claim was further supported by OpenAI’s o1 release, where an obvious improvement in reasoning skills was observed. The main difference between earlier OpenAI versions and o1 was the focus on train-time and test-time compute, rather than the number of parameters⁶.
Collaboration is key
What’s better than an AI agent? Multiple collaborative AI Agents. In an example hierarchical architecture, a root-level agent coordinates worker agents, which receive instructions from the coordinator. Each agent has its own objective and can work toward it independently, potentially using different LLM models. The root-level agent aggregates outputs and creates further objectives for the worker agents. This approach is known as Mixture of Agents (MoA).
In a recent study, researchers were able to outperform ChatGPT 4 by using open-source LLMs in an MoA setup, achieving a score of 65.1% compared to ChatGPT’s 57.5% in AlpacaEval 2.0 benchmark⁷.
As a software engineer, I find MoA approach very similar to recent transition from Monolith to Microservices. Just like microservices, MoA brings the benefit of isolated components which can be independently developed.
Business Impact
AI Agents have the potential to revolutionize today’s business models. Admittedly, today’s agents function more like assistants. However, as they mature, what will prevent humanity from using AI Agents to replace a role, department, or even an entire organization? There is ongoing debate about so-called “10x or 100x software engineers” in some organizations. What if we had a software engineer with infinite scalability?
For many, these questions are intriguing; for others, they are frightening. As someone with high hopes for humanity, I choose to stay positive yet educated.
References
- https://www.nature.com/articles/nrn.2017.14
- https://platform.openai.com/docs/guides/function-calling
- https://docs.llamaindex.ai/en/stable/examples/structured_outputs/structured_outputs/
- https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
- https://arxiv.org/pdf/2408.03314
- https://openai.com/index/learning-to-reason-with-llms/
- https://arxiv.org/pdf/2406.04692
