Introduction to AI Agents

Forget abstract AI concepts — let’s talk about action! That’s precisely what AI Agents are all about. Think of them as the ‘do-ers’ of the AI world, engineered to perceive, reason, and act within environments to reach specific objectives. From self-driving marvels to intelligent personal assistants, AI agents are not just prospects; they’re actively building our today and will undoubtedly shape tomorrow. Curious to understand the inner workings of these fascinating entities? I recently completed the fantastic online course on AI Agents Fundamentals by Hugging Face, which illuminated this dynamic domain for me. In this blog series, we’ll unpack the fundamental principles of AI Agents, drawing heavily on the knowledge I gained from this insightful course.

What is an Agent?

Informally, an agent is an AI model capable of reasoning, planning, and interacting with its environment. We call it Agent because it has agency, aka it can interact with the environment.

More formally, an agent is a system that leverages an AI model to interact with its environment to achieve a user-defined objective. It combines reasoning, planning, and executing actions (often via external tools) to fulfill tasks. The scope of possible actions depends on what the agent has been equipped with.

Think of the Agent as having two main parts:

• The Brain (AI Model): This is where all the thinking happens. The AI model handles reasoning and planning. It decides which Actions to take based on the situation. The most common AI model in Agents is an LLM (Large Language Model), which takes Text as an input and outputs Text. It’s also possible to use models that accept other inputs as the Agent’s core model. For example, a Vision Language Model (VLM) is like an LLM but also understands images as input.
• The Body (Capabilities and Tools): This part represents everything the Agent is equipped to do.

How does an AI take action in its environment?

Large Language Models (LLMs) are powerful text generators, but they lack the inherent ability to create visual outputs like images. However, platforms like HuggingChat and ChatGPT overcome this limitation by integrating external tools. These tools essentially provide the LLM with a “body” or an “extension” that allows it to interact with the visual world. When prompted to generate an image, the LLM doesn’t create it directly. Instead, it uses these integrated tools (like image generation APIs) as a means to “act” on its environment and produce the desired result. This demonstrates how AI agents can leverage tools to extend their capabilities beyond their core functionalities, effectively giving them a “body” to perform actions in different modalities.

What type of tasks can an Agent do?

An Agent can perform any task we implement via Tools to complete Actions. For example, if I write an Agent to act as my assistant (like Siri) on my computer, and I ask it to “send an email to my Manager asking to delay today’s meeting”, I can give it some code to send emails. This will be a new Tool the Agent can use whenever it needs to send an email. We can write it in Python.

The effectiveness of an AI agent significantly depends on the design of its tools. Carefully crafted, specialized tools are crucial for tackling complex, specific tasks, whereas general-purpose tools like web search can handle broader inquiries. It’s important to distinguish between ‘tools’ and ‘actions.’ An action might require the agent to utilize multiple tools in sequence to achieve a goal. By equipping agents with the ability to interact with their environment through these tools, we unlock practical applications for businesses and individuals alike.

Let's give three examples:

1- Personal Virtual Assistants: Virtual assistants like Siri, Alexa, or Google Assistant, work as agents when they interact on behalf of users using their digital environments. They take user queries, analyze context, retrieve information from databases, and provide responses or initiate actions (like setting reminders, sending messages, or controlling smart devices).

2– Customer Service Chatbots: Many companies deploy chatbots as agents that interact with customers in natural language. These agents can answer questions, guide users through troubleshooting steps, open issues in internal databases, or even complete transactions. Their predefined objectives might include improving user satisfaction, reducing wait times, or increasing sales conversion rates. By interacting directly with customers, learning from the dialogues, and adapting their responses over time, they demonstrate the core principles of an agent in action.

3- AI Non-Playable Character in a video game: AI agents powered by LLMs can make Non-Playable Characters (NPCs) more dynamic and unpredictable. Instead of following rigid behavior trees, they can respond contextually, adapt to player interactions, and generate more nuanced dialogue. This flexibility helps create more lifelike, engaging characters that evolve alongside the player’s actions.

What are LLMs?

Large Language Models (LLMs) are AI systems designed to comprehend and produce human language. Trained on massive text datasets, they learn linguistic patterns and nuances. Characterized by their vast number of parameters, most modern LLMs utilize the Transformer architecture, which leverages the ‘Attention’ mechanism and gained prominence after Google’s BERT in 2018.

A typical Transformer architecture with an encoder on the left and a decoder on the right.

There are 3 types of transformers :

1. Encoders: An encoder-based Transformer takes text (or other data) as input and outputs a dense representation (or embedding) of that text. BERT from Google is a good example for the encoder-only transformer. Text classification, semantic search, Named Entity Recognition are typical use cases and they include millions of parameters.
2. Decoders: A decoder-based Transformer focuses on generating new tokens to complete a sequence, one token at a time. Llama from Meta represents the Decoder-only Transformer. Text generation, chatbots, code generation are the typical use case examples and they include billions (in the US sense, i.e., 10⁹) of parameters.
3. Seq2Seq (Encoder–Decoder): A sequence-to-sequence Transformer combines an encoder and a decoder. The encoder first processes the input sequence into a context representation, and then the decoder generates an output sequence. T5 and BART use these transformers, and they provide Translation, Summarization, Paraphrasing by using millions of parameters.

LLM’s objective is to predict the next token, given a sequence of previous tokens. A “token” is the unit of information an LLM works with. You can think of a “token” as if it were a “word”, but for efficiency reasons, LLMs don’t use whole words.

• For example, while English has an estimated 600,000 words, an LLM might have a vocabulary of around 32,000 tokens (as is the case with Llama 2). Tokenization often works on sub-word units that can be combined.
• For instance, consider how the tokens “interest” and “ing” can be combined to form “interesting”, or “ed” can be appended to form “interested.”

Tokenizer Playground

Each LLM has some special tokens specific to the model. The LLM uses these tokens to open and close the structured components of its generation. For example, to indicate the start or end of a sequence, message, or response. Moreover, the input prompts that we pass to the model are also structured with special tokens. The most important of those is the End of sequence token (EOS). The forms of special tokens are highly diverse across model providers.

LLMs are said to be autoregressive, meaning that the output from one pass becomes the input for the next one. This loop continues until the model predicts the next token to be the EOS token, at which point the model can stop. In other words, an LLM will decode text until it reaches the EOS. To understand how learning agents work, let’s simplify what occurs in one decoding step, though the complete process is complex.

• Once the input text is tokenized, the model computes a representation of the sequence that captures information about the meaning and the position of each token in the input sequence.
• This representation goes into the model, which outputs scores that rank the likelihood of each token in its vocabulary as being the next one in the sequence.

Understanding next token prediction:

• Based on the scores, we have multiple strategies to select the tokens to complete the sentence. The easiest decoding strategy would be to always take the token with the maximum score.
• You can interact with the decoding process yourself with SmolLM2 in this Space (remember, it decodes until reaching an EOS token which is <|im_end|> for this model):

Attention:

• A key aspect of the Transformer architecture is Attention. When predicting the next word, not every word in a sentence is equally important; words like “Spain” and “capital” in the sentence “The capital of Spain is …” carry the most meaning. This process of identifying the most relevant words to predict the next token has proven to be incredibly effective.
• Although the basic principle of LLMs — predicting the next token — has remained consistent since GPT-2, there have been significant advancements in scaling neural networks and making the attention mechanism work for longer and longer sequences.
• Context length refers to the maximum number of tokens the LLM can process and the maximum attention span it has.

Prompting the LLM is important:

• Considering that the only job of an LLM is to predict the next token by looking at every input token and choosing which tokens are “important”, the wording of your input sequence is very important.
• The input sequence you provide an LLM is called a prompt. The careful design of the prompt makes it easier to guide the generation of the LLM toward the desired output.

How are LLMs trained?

• LLMs are trained on large datasets of text, where they learn to predict the next word in a sequence through a self-supervised or masked language modeling objective.
• From this unsupervised learning, the model learns the structure of the language and underlying patterns in text, allowing the model to generalize to unseen data.
• After this initial pre-training, LLMs can be fine-tuned to a supervised learning objective to perform specific tasks. For example, some models are trained for conversational structures or tool usage, while others focus on classification or code generation.

How can I use LLMs?: You have two main options: Run locally (if you have sufficient hardware). Use a Cloud/API (e.g., via the Hugging Face Serverless Inference API). LLMs are a key component of AI Agents, providing the foundation for understanding and generating human language. They can interpret user instructions, maintain context in conversations, define a plan, and decide which tools to use.

Messages (The Underlying System of LLMs) and Special Tokens

Up until now, we’ve discussed prompts as the sequence of tokens fed into the model. But when you chat with systems like ChatGPT or HuggingChat, you’re exchanging messages. Behind the scenes, these messages are concatenated and formatted into a prompt that the model can understand. They act as the bridge between conversational messages (user and assistant turns) and the specific formatting requirements of your chosen LLM. Chat templates structure the communication between the user and the agent, ensuring that every model — despite its unique special tokens — receives the correctly formatted prompt.

Get Samet Girgin’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Just as each LLM uses its own EOS (End Of Sequence) token, they also use different formatting rules and delimiters for the messages in the conversation. Special Tokens serve specific functions like marking the end of a sequence (EOS) or separating different message roles in chat models

System Messages:

• System messages (also called System Prompts) define how the model should behave. They serve as persistent instructions, guiding every subsequent interaction. When using Agents, the System Message also gives information about the available tools, provides instructions to the model on how to format the actions to take, and includes guidelines on how the thought process should be segmented.

System Messages

Chat Templates:

• Chat templates help maintain context by preserving conversation history, storing previous exchanges between the user and the assistant. This leads to more coherent multi-turn conversations. Templates can handle complex multi-turn conversations while maintaining context.
• As mentioned, chat templates are essential for structuring conversations between language models and users. They guide how message exchanges are formatted into a single prompt.
• Another point we need to understand is the difference between a Base Model vs. an Instruct Model. A Base Model is trained on raw text data to predict the next token. An Instruct Model is fine-tuned specifically to follow instructions and engage in conversations. For example, SmolLM2–135M is a base model, while SmolLM2–135M-Instruct is its instruction-tuned variant.
• To make a Base Model behave like an instruct model, we need to format our prompts in a consistent way that the model can understand. This is where chat templates come in.
• ChatML is one such template format that structures conversations with clear role indicators (system, user, assistant). If you have interacted with some AI API lately, you know that’s the standard practice. It’s important to note that a base model could be fine-tuned on different chat templates, so when we’re using an instruct model we need to make sure we’re using the correct chat template.
• Because each instruct model uses different conversation formats and special tokens, chat templates are implemented to ensure that we correctly format the prompt the way each model expects.
• In transformers, chat templates include Jinja2 code that describes how to transform the ChatML list of JSON messages, as presented in the above examples, into a textual representation of the system-level instructions, user messages, and assistant responses that the model can understand. This structure helps maintain consistency across interactions and ensures the model responds appropriately to different types of inputs.

How transformers uses chat templates: https://huggingface.co/docs/transformers/main/en/chat_templating

• The easiest way to ensure our LLM receives a conversation correctly formatted is to use the chat_template from the model’s tokenizer.

• To convert the previous conversation into a prompt, we load the tokenizer and call apply_chat_template:

• The rendered_prompt returned by this function is now ready to use as the input for the model we chose. This apply_chat_template() function will be used in the backend of our API when you interact with messages in the ChatML format.

What are tools?

• A good tool should be something that complements the power of an LLM. For instance, if we need to perform arithmetic, giving a calculator tool to our LLM will provide better results than relying on the native capabilities of the model. Furthermore, LLMs predict the completion of a prompt based on their training data, which means that their internal knowledge only includes events before their training. Therefore, if your agent needs up-to-date data you must provide it through some tool.
• Here are some commonly used tools in AI agents:

• A Tool should contain: A textual description of what the function does, a Callable (something to act), Arguments with typings and Outputs (optional) with typings.

How do tools work?:

• What we mean when we talk about providing tools to an Agent is that we teach the LLM about the existence of tools and ask the model to generate text that will invoke tools when it needs to. For example, if we provide a tool to check the weather at a location from the Internet and then ask the LLM about the weather in Paris, the LLM will recognize that question as a relevant opportunity to use the “weather” tool we taught it about.
• The LLM will generate text in the form of code, to invoke that tool. It is the responsibility of the Agent to parse the LLM’s output, recognize that a tool call is required, and invoke the tool on the LLM’s behalf. The output from the tool will then be sent back to the LLM. The output from a tool call is another type of message in the conversation.
• Tool-calling steps are typically not shown to the user: The agent retrieves the conversation, calls the tool(s), gets the outputs, adds them as a new conversation message, and sends the updated conversation to the LLM again.
• From the user’s point of view, it’s like the LLM had used the tool, but in fact, it was our application code (the Agent) that did it.

How do we give tools to an LLM?:

• We will implement a simplified calculator tool that will just multiply two integers. This could be our Python implementation:

• All of these details are important. Let’s put them together in a text string that describes our tool for the LLM to understand. Tool Name is “calculator”. Description is “Multiply two integers”. Arguments are “int, b: int”. Outputs are int
• When we pass the previous string as part of the input to the LLM, the model will recognize it as a tool and will know what it needs to pass as inputs and what to expect from the output. If we want to provide additional tools, we must be consistent and always use the same format. This process can be fragile, and we might accidentally overlook some details.
• All we need is for the tool implementation to use type hints, docstrings, and sensible function names. We will write some code to extract the relevant portions from the source code. After we are done, we’ll only need to use a Python decorator to indicate that the calculator function is a tool:

• With the implementation we’ll see next, we will be able to retrieve the following text automatically from the source code via the to_string() function provided by the decorator.

Understanding AI Agents through the Thought-Action-Observation Cycle

Agents work in a continuous cycle of: thinking (Thought) → acting (Act) , and observing (Observe).

• Thought: The LLM part of the Agent decides what the next step should be.
• Action: The agent takes an action by calling the tools with the associated arguments.
• Observation: The model reflects on the response from the tool.

Our system prompt may look like this:

The weather Agent (Using Weather API tool):

Prompt → Thought (Internal Reasoning)

Action (Tool Stage) → Observation (Feedback from the Environment)

Updated thought (Reflecting) → Final Action

Thought: Internal Reasoning and the Re-Act Approach

• Thoughts represent the Agent’s internal reasoning and planning processes to solve the task. This utilizes the agent’s Large Language Model (LLM) capacity to analyze information when presented in its prompt. It is like agent’s internal dialogue, where it considers the task at hand and strategizes its approach.
• The Agent’s thoughts are responsible for accessing current observations and decide what the next action(s) should be.
• Through this process, the agent can break down complex problems into smaller, more manageable steps, reflect on past experiences, and continuously adjust its plans based on new information.

Examples of common thoughts

The Re-Act Approach:

• A key method is the ReAct approach, which is the concatenation of “Reasoning” (Think) with “Acting” (Act).
• ReAct is a simple prompting technique that appends “Let’s think step by step” before letting the LLM decode the next tokens.
• Indeed, prompting the model to think “step by step” encourages the decoding process toward the next tokens that generate a plan, rather than a final solution, since the model is encouraged to decompose the problem into sub-tasks. This allows the model to consider sub-steps in more detail, which in general leads to fewer errors than trying to generate the final solution directly.
• We have recently seen a lot of interest in reasoning strategies. This is what’s behind models like Deepseek R1 or OpenAI’s o1, which have been fine-tuned to “think before answering”. These models have been trained to always include specific thinking sections
• This is not just a prompting technique like ReAct, but a training method where the model learns to generate these sections after analyzing thousands of examples that show what we expect it to do.

Actions: Enabling the Agent to Engage with Its Environment

• Actions are the concrete steps an AI agent takes to interact with its environment.
• Whether it’s browsing the web for information or controlling a physical device, each action is a deliberate operation executed by the agent. For example, an agent assisting with customer service might retrieve customer data, offer support articles, or transfer issues to a human representative.

Types of Agent Actions:

• One crucial part of an agent is the ability to STOP generating new tokens when an action is complete, and that is true for all formats of Agent: JSON, code, or function-calling. This prevents unintended output and ensures that the agent’s response is clear and precise.
• The LLM only handles text and uses it to describe the action it wants to take and the parameters to supply to the tool.

The Stop and Parse Approach: (The agent’s output is structured and predictable)

• Generation in a Structured Format: The agent outputs its intended action in a clear, predetermined format (JSON or code).
• Halting Further Generation: Once the action is complete, the agent stops generating additional tokens. This prevents extra or erroneous output.
• Parsing the Output: An external parser reads the formatted action, determines which Tool to call, and extracts the required parameters.

Code Agents:

• Code agents generate an executable code block — typically in a high-level language like Python

The advantages of Code Agents are:

1. Expressiveness: Code can naturally represent complex logic, including loops, conditionals, and nested functions, providing greater flexibility than JSON.
2. Modularity and Reusability: Generated code can include functions and modules that are reusable across different actions or tasks.
3. Enhanced Debuggability: With a well-defined programming syntax, code errors are often easier to detect and correct.
4. Direct Integration: Code Agents can integrate directly with external libraries and APIs, enabling more complex operations such as data processing or real-time decision — making.

Observe: Integrating Feedback to Reflect and Adapt

Observations are how an Agent perceives the consequences of its actions. They provide crucial information that fuels the Agent’s thought process and guides future actions. They are signals from the environment — whether it’s data from an API, error messages, or system logs — that guide the next cycle of thought. In the observation phase, the agent:

• Collects Feedback: Receives data or confirmation that its action was successful (or not).
• Appends Results: Integrates the new information into its existing context, effectively updating its memory.
• Adapts its Strategy: Uses this updated context to refine subsequent thoughts and actions.

Let’s look at the weather AI agent Case:

• If a weather API returns the data “partly cloudy, 15°C, 60% humidity”, this observation is appended to the agent’s memory (at the end of the prompt).
• The Agent then uses it to decide whether additional information is needed or if it’s ready to provide a final answer.
• This iterative incorporation of feedback ensures the agent remains dynamically aligned with its goals, constantly learning and adjusting based on real-world outcomes.
• These observations can take many forms, from reading webpage text to monitoring a robot arm’s position. This can be seen in Tool “logs” that provide textual feedback on the Action execution.

In essence, AI agents represent a paradigm shift in how we interact with and leverage artificial intelligence. By combining the reasoning power of LLMs with the practical capabilities of external tools, these agents are capable of tackling complex tasks and adapting to dynamic environments. From enhancing user experiences with virtual assistants and chatbots to revolutionizing gameplay with dynamic NPCs, the potential applications are vast and varied. As we continue to refine tool integration, optimize prompting techniques, and advance the underlying LLM technology, we move closer to a future where AI agents seamlessly integrate into our daily lives, automating processes, solving problems, and unlocking new possibilities.

Reference:

tokenizer_config.json · HuggingFaceTB/SmolLM2-135M-Instruct at main

GitHub - huggingface/agents-course: This repository contains the Hugging Face Agents Course.

Hugging Face - The AI community building the future.