<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alex Gonzalez on Medium]]></title>
        <description><![CDATA[Stories by Alex Gonzalez on Medium]]></description>
        <link>https://medium.com/@alecgg27895?source=rss-b28db33b9ddd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zmfUFPL6T5wh853y76AJJw.jpeg</url>
            <title>Stories by Alex Gonzalez on Medium</title>
            <link>https://medium.com/@alecgg27895?source=rss-b28db33b9ddd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:47:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alecgg27895/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Jinja2 prompting — A guide on using jinja2 templates for prompt management in GenAI applications]]></title>
            <link>https://medium.com/@alecgg27895/jinja2-prompting-a-guide-on-using-jinja2-templates-for-prompt-management-in-genai-applications-e36e5c1243cf?source=rss-b28db33b9ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/e36e5c1243cf</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[jinja2]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[prompt]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Alex Gonzalez]]></dc:creator>
            <pubDate>Mon, 20 May 2024 17:12:29 GMT</pubDate>
            <atom:updated>2024-05-20T17:12:29.168Z</atom:updated>
            <content:encoded><![CDATA[<h3>Jinja2 prompting — A guide on using jinja2 templates for prompt management in GenAI applications</h3><p><strong>Jinja2</strong> is a popular templating engine that generates dynamic outputs from static templates using <strong>placeholders</strong>, originally designed for dynamic HTML it is used in many applications that require dynamic text generation, such as AI prompting. Some features, like <strong>inheritance</strong>, <strong>looping</strong> and <strong>conditions</strong>, complement its use in <strong>Generative AI (GenAI)</strong> apps where prompts are built for <strong>LLMs</strong> to achieve specific outcomes. The content of these prompts typically varies based on user inputs, prior interactions, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5XVdUhQarNnHIJxRMlkQ4g.png" /></figure><h3>Why Use Jinja2 for Prompting?</h3><p>Using <strong>Jinja2 templates</strong> for prompting offers several benefits:</p><ol><li>Since I started working on GenAI projects, I’ve found prompts embedded within the code, even in LangChain docs. This mixes LLM instructions with application logic, complicating code understanding. Using Jinja2 for prompts helps to maintain a clean and <strong>organized code.</strong></li><li>As <strong>GenAI projects scale</strong>, developers need to manage many unique, lengthy prompts. Storing these prompts in Jinja2 templates improves <strong>organization</strong>, facilitating prompt iteration.</li><li>In <strong>prompt engineering</strong>, it’s common to reuse parts of prompts which is not optimal. Jinja2 allows to use a <strong>single styling template</strong> that can be imported into other prompts, avoiding duplication. Additionally, <strong>Jinja2’s built-in inheritance</strong> enables the use of base templates that other templates can inherit from.</li></ol><h3>How to Use Jinja2 for Prompt Management</h3><p>Start by installing Jinja2 using pip or another package manager like Poetry.</p><pre>pip install jinja2<br># or<br>poetry add jinja2</pre><p>Once installed, you can begin using Jinja2 by creating template files. Here is a basic example of how to invoke Jinja2 from Python, utilizing a .txt file as a template. For the sake of this example, let’s imagine we are instructing an <strong>LLM-based chatbot</strong> to address the user by name and adjust the complexity of its responses according to the user’s technical level:</p><ol><li><strong>Create a template file: </strong>Save the content below in a file named prompt_template.txt</li></ol><pre>- When interacting with the user call him by his name.<br>- The name of the user is {{ user_name }}.<br><br>{% if is_user_technical == &quot;True&quot; %} <br>- The user is technical, you can answer the user using technical terms.<br>{% else %}<br>- The user is non technical, please answer him using non technical terms.<br>{% endif %}</pre><p>2. <strong>Render the template in Python:</strong> Run the following code in the directory containing the template. If the template is in a different directory, adjust FileSystemLoader(directory_path).</p><pre>env = Environment(loader=FileSystemLoader(&quot;&quot;))<br>template = env.get_template(&quot;prompt_template.txt&quot;)<br>output = template.render({&quot;user_name&quot;: &quot;Alex&quot;, &quot;is_user_technical&quot;: &quot;True&quot;})<br>print(output)</pre><p>When the template is rendered, the resulting prompt will reflect the user’s name and adapt its language based on the user’s technical profile:</p><pre>- When interacting with the user call him by his name.<br>- The name of the user is Alex.<br><br>- The user is technical, you can answer the user using technical terms.<br></pre><p>You can find all the code and examples used in the post in this repository:</p><p><a href="https://github.com/alexgg278/jinja2-llm-prompting">GitHub - alexgg278/jinja2-llm-prompting</a></p><h3>All In One Function: Integrating Jinja2 with LangChain</h3><p>A downside of using Jinja2 directly for rendering templates as prompt strings is that <strong>LangChain’s built-in PromptTemplate</strong> objects are not used. However, by using the following function, we can still use Jinja2 templates while outputting the resulting prompt as a LangChain PromptTemplate. This integration takes advantage of the <strong>LCEL Runnable interface</strong>, which supports various operations like invoke and stream (<a href="https://python.langchain.com/v0.1/docs/modules/model_io/prompts/quick_start/">LangChain docs</a>) to implement complex chains that mix prompts, parsers, model calls, etc.</p><pre>from langchain_core.prompts import PromptTemplate<br><br>def get_prompt_template_from_jinja2(<br>    prompt_path: str,<br>    prompt_name: str,<br>    input_variables: list[str]=[],<br>    partial_variables: dict[str, str]={},<br>    jinja2_placeholders: dict[str, str]={},<br>    ) -&gt; PromptTemplate:<br> &quot;&quot;&quot;<br> Loads a .txt file as a Jinja2 template and converts it into a LangChain PromptTemplate.<br><br> Parameters:<br>        prompt_path: Path to the prompt.<br>        prompt_name: Filename of the prompt, including its extension.<br>        input_variables: A list of variable names expected by the LangChain PromptTemplate.<br>        partial_variables: A dictionary of partial variables carried by the LangChain PromptTemplate.<br>        jinja2_placeholders: A dictionary of placeholders to be replaced in the Jinja2 template.<br> &quot;&quot;&quot;<br><br> env = Environment(loader=FileSystemLoader(prompt_path))<br> template = env.get_template(prompt_name)<br> prompt_string = template.render(jinja2_placeholders)<br><br> prompt_template = PromptTemplate(<br>  template=prompt_string,<br>  input_variables=input_variables,<br>  partial_variables=partial_variables,<br> )<br> return prompt_template</pre><pre>get_prompt_template_from_jinja2(<br>    prompt_path=&quot;&quot;,<br>    prompt_name=&quot;prompt_template.txt&quot;,<br>    jinja2_placeholders={&quot;user_name&quot;: &quot;Alex&quot;, &quot;is_user_technical&quot;: &quot;True&quot;}<br>)</pre><p>After executing the function, we obtain a PromptTemplate object that can be used as a runnable within LCEL chains:</p><pre>PromptTemplate(input_variables=[], template=&#39;- When interacting with the user call him by his name.\n- The name of the user is Alex.\n\n \n- The user is technical, you can answer the user using technical terms.\n&#39;)</pre><h3>Additional Jinja2 Template Applications for Prompting</h3><h4>Loops</h4><p>Jinja2 templates can <strong>iterate through data structures such as lists</strong>. For example, if you have a list of programming languages that the user is familiar with, these can be dynamically included in the prompt by iterating through the list. This method eliminates the need to manually format the list into a string for the prompt:</p><pre>- When answering coding questions from the user, you have to provide him the answer in the following languages:<br><br>{% for programming_language in programming_languages_list %}<br> Programming language number {{ loop.index }}: {{ programming_language }}<br>{% endfor %}</pre><p>Execute the following code to render this Jinja2 template:</p><pre>programming_languages_list = [&quot;python&quot;, &quot;r&quot;, &quot;javascript&quot;, &quot;c#&quot;]<br><br>template = env.get_template(&quot;loop_prompt_template.txt&quot;)<br>rendered_output = template.render({&quot;programming_languages_list&quot;: programming_languages_list})<br>print(rendered_output)</pre><p>After rendering, the output is:</p><pre>- When answering coding questions from the user, you have to provide him the answer in the following languages:<br><br>Programming language number 1: python<br><br>Programming language number 2: r<br><br>Programming language number 3: javascript<br><br>Programming language number 4: c#</pre><h4>Template inheritance</h4><p>Jinja2 allows for the creation of specific templates from a base template, allowing to reuse text. This feature is particularly useful for using the same <strong>instructions across multiple prompts.</strong></p><p>In this example, we want to have a solid style for the responses of our application, hence, we will be using the same styling instructions across all the prompts, therefore, we include this instructions in a base_template.txt and then build the rest of the prompts from this base template.</p><p>base_template.txt</p><pre>- You are an AI assistant that helps to solve problems in various fields.<br>- Please, answer using formal language.<br>- Use short and direct sentences.<br><br>{% block user_type %}<br>{% endblock %}</pre><p>specific_template.txt</p><pre>{% extends &quot;base_template.txt&quot; %}<br><br>{% block user_type %}<br>{% if is_user_technical == &quot;True&quot; %} <br> - The user is technical, you can answer the user using technical terms.<br>{% else %}<br> - The user is non technical, please answer him using non technical terms.<br>{% endif %}<br>{% endblock %}</pre><p>Rendering specific_template.txt :</p><pre>template = env.get_template(&quot;specific_template.txt&quot;)<br>output = template.render({&quot;is_user_technical&quot;: &quot;True&quot;})<br>print(output)</pre><p>In the result below it is possible to observe how just by rendering the specific_template.txt the instructions from the inherited base_template.txtare also included.</p><pre>- You are an AI assistant that helps to solve problems in various fields.<br>- Please, answer using formal language.<br>- Use short and direct sentences.<br>- The user is technical, you can answer the user using technical terms.</pre><h3>Conclusion</h3><p>All in all, Jinja2 templates aid the management of prompts in GenAI projects, ensuring code remains organized. With features like loops, template inheritance and conditions, they facilitate efficient development across diverse scenarios. Here you can find a handy Jinja cheatsheet.</p><p><a href="https://devhints.io/jinja">Jinja cheatsheet</a></p><p>Thanks for reading, see you soon!</p><p>Feel free to follow me, to contact me and any <strong>feedback </strong>is welcome in the comments.</p><p><strong>About me:</strong></p><p><em>Alejandro González, Data Scientist / AI Engineer,</em></p><p><em>MSc in Data Science at Politechnique University of Madrid and Eötvös Loránd University of Budapest, with background in Telecommunications Engineering. Experience in different Data Science and Machine Learning domains like, Data Visualization, LLMs, Unsupervised Learning, etc.</em></p><p>You can find me on:<br>👨🏻‍💼<a href="https://www.linkedin.com/in/alexgg278/">Linekdin</a><br>👨🏻‍💻<a href="https://github.com/alexgg278">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e36e5c1243cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mistral-7b and LLaMA-2–7b: A guide to Fine-Tuning LLMs in Google Colab]]></title>
            <link>https://medium.com/@alecgg27895/mistral-7b-and-llama-2-7b-a-guide-to-fine-tuning-llms-in-google-colab-2ce78db37245?source=rss-b28db33b9ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ce78db37245</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[language-model]]></category>
            <dc:creator><![CDATA[Alex Gonzalez]]></dc:creator>
            <pubDate>Sat, 17 Feb 2024 17:57:27 GMT</pubDate>
            <atom:updated>2024-02-17T17:57:27.102Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>LLaMA-2–7b</strong> and <strong>Mistral-7b</strong> have been two of the most popular <strong>open source</strong> LLMs since their release. Having only <strong>7 billion</strong> parameters make them a perfect choice for individuals who seek fine-tuning LLMs with low hardware requirements. In this post, we review the main characteristics of Mistral and LLaMA2 models and finish with a Mistral-7b fine-tune guide.</p><h4>Side by side: Mistral-7b and LLaMA-2–7b</h4><p>The launch of LLaMA-2–7b provided a compact, open-source language model with robust performance. This <strong>democratized</strong> access to fine-tuninLLMg, eliminating the requirement for large, expensive GPUs.</p><p>On the other side, Mistral-7b added a new dimension to the open-source LLMs by <strong>improving performance</strong> while maintaining a <strong>compact size</strong> (7.3B parameters). This model also exceeded the performance of LLaMA2–7b and LLaMA2–13B across benchmarks (MMLU, HellaSwag, MATH, etc).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*8iz37daVALiC4trICejYyQ.png" /><figcaption>LLaMA models bs Mistral 7B comparison</figcaption></figure><p>Mistral-7b uses these two techniques that make it faster and more efficient:</p><ul><li><strong>Grouped-query attention (GQA):</strong> This technique <strong>groups similar queries</strong> for <strong>faster and better inference</strong>, mixing the quality of Multi-Head Attention with the speed of Multi-Query Attention approaches. If you want to read more about it click <a href="https://klu.ai/glossary/grouped-query-attention">here</a>.</li><li><strong>Sliding window attention (SWA):</strong> Working with longer sequences requires applying attention to all input tokens, a process that becomes less efficient as sequences lengthen. SWA addresses this by <strong>segmenting sequences into overlapping windows</strong>, reducing computational costs while preserving context.</li></ul><h4>Parameter Efficient Fine Tuning (PEFT): LoRA and Quantization</h4><p>Models like Mistral and LLaMA, have made possible to perform fine-tuning for free using services like Google Colab. Despite their small size, they still take up to 30Gb GPU memory. In order to reduce memory requirements and costs techniques like <strong>LoRA</strong> and <strong>Quantization</strong> are used.</p><ul><li><strong>Lora</strong>: It is a <strong>low-rank</strong> adaptation technique that reduces the number of parameters while fine-tuning. LoRA transforms the parameter matrix into two lower-rank matrices, whose dot product approximates the original matrix. The rank <strong>“r”</strong> is adjustable, balancing speed and quality — lower ranks yield <strong>quicker training</strong> but at a <strong>quality trade-off.</strong></li><li><strong>Quantization (QLoRA): </strong>LLMs usually use <strong>16-bit</strong> precision, taking a lot of memory for storing high-precision weights. Quantization reduces weight precision to <strong>4 or 8 bits</strong>, significantly reducing model size at a quality trade-off. NF4 quantization converts weights to 4-bit precision. Therefore, each weight can take up to 16 different values, normalizing them to a zero-centered distribution and making models more memory-efficient while slightly impacting quality.</li></ul><h4>Choosing a dataset for fine-tuning: RecipeNLG</h4><p>Imagine having an assistant where you could simply ask for any recipe and instantly receive a list of ingredients along with precise cooking instructions in a specific format. We can achieve this by fine-tuning using <strong>RecipeNLG</strong>, a dataset of <strong>2.23 million recipes</strong>. The dataset initially provides the ingredients and directions in lists as it is shown below. Download RecipeNLG <a href="https://recipenlg.cs.put.poznan.pl/">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ygpoo94RlGSE7ZYlSDYCgA.png" /><figcaption>1 raw sample from the RecipeNLG dataset</figcaption></figure><p>Prior to fine-tuning, some transformations on the dataset are needed. First, we convert the recipes into plain text by combining the different columns, resulting in complete recipes like the one below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*b5Sl6FWINM67xszHQuTPZw.png" /><figcaption>Transformation of raw dataset columns into recipe string</figcaption></figure><p>Afterwards, we structure the samples in a input/output format and store them in a <strong>JSONL</strong>, which is ideal for LLM instruction fine-tuning. This ensures that the fine-tuned model can easily generate recipes in the format we trained it with.</p><pre>{&quot;input&quot;:&quot;What are the ingredients and directions to prepare the Pancakes recipe?&quot;, &quot;output&quot;:&quot;Ingredients:\n1 c. flour\n1 tsp. soda\n1 tsp. salt\n1 Tbsp. sugar\n1 egg\n3 Tbsp. margarine, melted\n1 c. buttermilk\n\nDirections:\nMix dry ingredients.\nAdd egg, margarine and buttermilk. Stir by hand until well blended.\nPour onto non-stick pan or griddle with a little oil added.\nCook until bubbly and turn over.&quot;}</pre><h4>Fine-tune Mistral-7b</h4><p>The models we’re using are hosted on Hugging Face and the fine-tuning process for all LLMs hosted in huggingface is similar. In this post we fine-tune Mistral-7b, but any other LLM, like LLaMA-2–7b, can be fine-tuned by changing the huggingface model ID.</p><p>We can make use of <strong>Google Colab’s free T4 GPUs</strong> to fine-tune the model. We start by installing the necessary libraries in the Colab session.</p><pre>!pip install -q -U bitsandbytes<br>!pip install -q -U git+https://github.com/huggingface/transformers.git<br>!pip install -q -U git+https://github.com/huggingface/peft.git<br>!pip install -q -U git+https://github.com/huggingface/accelerate.git</pre><p>Next, we import the data. We loaded it from Google Drive, however, you can also import the data by uploading the .csv to Colab. Additionally, we use some functions to transform the data into the desired format and we create the input and output strings to fine-tune the model.</p><pre>import pandas as pd<br>import pandas as pd<br><br>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)<br><br>def convert_string_to_list(string):<br>    try:<br>        return json.loads(string)<br>    except (ValueError, TypeError):<br>        return string<br><br>def create_input(df):<br>    df[&#39;input&#39;] = &#39;What are the ingredients and direction to prepare &#39; + df[&#39;title&#39;] + &#39; recipe?&#39;<br>    return df<br><br>def create_output(df):<br>    df[&#39;output&#39;] = &#39;Ingredients:\n&#39; + \<br>                   df[&#39;ingredients&#39;].str.join(&#39;\n&#39;) + &#39;\n\n&#39; + \<br>                   &#39;Directions:\n&#39; + \<br>                   df[&#39;directions&#39;].str.join(&#39;\n&#39;)<br>    return df<br><br>file_path = &#39;/path/to/data/recipes.csv&#39;<br>recipes_df = pd.read_csv(file_path)<br><br>recipes_df.drop([&quot;Unnamed: 0&quot;, &quot;link&quot;, &quot;source&quot;, &quot;NER&quot;], axis=1, inplace=True)<br>recipes_df = recipes_df[:30000]<br><br>recipes_df[&quot;ingredients&quot;] = recipes_df[&quot;ingredients&quot;].apply(convert_string_to_list)<br>recipes_df[&quot;directions&quot;] = recipes_df[&quot;directions&quot;].apply(convert_string_to_list)<br><br>recipes_df = create_input(recipes_df)<br>recipes_df = create_output(recipes_df)</pre><p>We split the dataset in the <strong>training-set</strong> and the <strong>validation-set</strong>. We will be using the validation test to keep track of the model training evolution. Then, we store the datasets as JSONL files and load them as Dataset objects.</p><pre>from sklearn.model_selection import train_test_split<br>from datasets import load_dataset<br><br>train_recipes_df, val_recipes_df = train_test_split(recipes_df, train_size=25000, test_size=400, random_state=10)<br><br><br>train_recipes_df[[&#39;input&#39;, &#39;output&#39;]].to_json(&#39;train_io_recipes.jsonl&#39;, orient=&#39;records&#39;, lines=True)<br>val_recipes_df[[&#39;input&#39;, &#39;output&#39;]].to_json(&#39;validation_io_recipes.jsonl&#39;, orient=&#39;records&#39;, lines=True)<br><br>train_io_recipes = load_dataset(&#39;json&#39;, data_files=&#39;train_io_recipes.jsonl&#39;, split=&#39;train&#39;)<br>validation_io_recipes = load_dataset(&#39;json&#39;, data_files=&#39;validation_io_recipes.jsonl&#39;, split=&#39;train&#39;)</pre><p>Then, we <strong>tokenize</strong> the dataset by first setting a max. sample length of <strong>500</strong>, to keep consistency. We convert the JSONL samples into strings for tokenization. Moving on, we load the model from Hugging Face using its <strong>Hugging Face ID</strong>. In this tutorial, Mistral-7b is used, but other models, like LLaMA2–7b can be fine-tuned by changing the <em>base_model_id</em>.</p><pre>from transformers import AutoTokenizer<br><br>organization = &quot;mistralai/&quot;<br>model_name = &quot;Mistral-7B-v0.1&quot;<br>base_model_id = organization + model_name<br><br>tokenizer = AutoTokenizer.from_pretrained(<br>    base_model_id,<br>    padding_side=&quot;left&quot;,<br>    add_eos_token=True,<br>    add_bos_token=True,<br>)<br>tokenizer.pad_token = tokenizer.eos_token<br><br>max_length = 500 # Set an appropriate length for the dataset<br><br>def tokenize_prompt_max_length(prompt):<br>    result = tokenizer(<br>        format_func(prompt),<br>        truncation=True,<br>        max_length=max_length,<br>        padding=&quot;max_length&quot;,<br>    )<br>    result[&quot;labels&quot;] = result[&quot;input_ids&quot;].copy()<br>    return result<br><br>def format_func(example):<br>    text = f&quot;### Question: {example[&#39;input&#39;]}\n ### Answer: {example[&#39;output&#39;]}&quot;<br>    return text<br><br>tokenized_train_io_recipes = train_io_recipes.map(tokenize_prompt_max_length)<br>tokenized_validation_io_recipes = validation_io_recipes.map(tokenize_prompt_max_length)</pre><p>Now, we load the pretrained model using the transformers library. To achieve this, we initialize the model with a <strong>4-bit quantization</strong> configuration through <strong>BitsAndBytesConfig</strong>, which significantly reduces memory usage to ensure the model fits within our Google Colab session.</p><pre>import torch<br>from transformers import AutoModelForCausalLM, BitsAndBytesConfig<br><br>bnb_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_use_double_quant=True,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_compute_dtype=torch.bfloat16<br>)<br><br>model = AutoModelForCausalLM.from_pretrained(base_model_id, quantization_config=bnb_config)</pre><p>Afterwards, we import some more libraries for the fine-tuning and set-up the accelarator.</p><pre>from accelerate import FullyShardedDataParallelPlugin, Accelerator<br>from torch.distributed.fsdp.fully_sharded_data_parallel import FullOptimStateDictConfig, FullStateDictConfig<br>from peft import prepare_model_for_kbit_training<br><br>fsdp_plugin = FullyShardedDataParallelPlugin(<br>    state_dict_config=FullStateDictConfig(offload_to_cpu=True, rank0_only=False),<br>    optim_state_dict_config=FullOptimStateDictConfig(offload_to_cpu=True, rank0_only=False),<br>)<br><br>accelerator = Accelerator(fsdp_plugin=fsdp_plugin)<br><br>model.gradient_checkpointing_enable()<br>model = prepare_model_for_kbit_training(model)</pre><p>At this point, we configure <strong>LoRA</strong> with <strong><em>“r=16”</em></strong> and <strong><em>“alpha=64”</em></strong> to balance efficiency and precision in fine-tuning. The parameter “r” adjusts the low-rank matrix’s rank, higher values yield to better performance and the cost of slower fine-tuning, “alpha” scales the weights, thus, higher values give more importance to LoRA activations. This piece of codes outputs the percentage of trainable parameters in fine-tuning.</p><pre>from peft import LoraConfig, get_peft_model<br><br>config = LoraConfig(<br>    r=16,<br>    lora_alpha=64,<br>    target_modules=[<br>        &quot;q_proj&quot;,<br>        &quot;k_proj&quot;,<br>        &quot;v_proj&quot;,<br>        &quot;o_proj&quot;,<br>        &quot;gate_proj&quot;,<br>        &quot;up_proj&quot;,<br>        &quot;down_proj&quot;,<br>        &quot;lm_head&quot;,<br>    ],<br>    bias=&quot;none&quot;,<br>    lora_dropout=0.05,<br>    task_type=&quot;CAUSAL_LM&quot;,<br>)<br><br>model = get_peft_model(model, config)<br><br>def print_train_params(model):<br>    trainable_params = 0<br>    all_param = 0<br>    for _, param in model.named_parameters():<br>        all_param += param.numel()<br>        if param.requires_grad:<br>            trainable_params += param.numel()<br>    print(<br>        f&quot;trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}&quot;<br>    )<br><br>print_train_params(model)</pre><p>As a final preparation for training, we set the model to train in parallel, accelerating the process for setups with more than one GPU.</p><pre>if torch.cuda.device_count() &gt; 1: # If more than 1 GPU<br>    model.is_parallelizable = True<br>    model.model_parallel = True<br><br>model = accelerator.prepare_model(model)</pre><p>Finally, we proceed to fine-tune the model, setting <strong><em>“max_steps”</em></strong> to <strong>500</strong> and a <strong><em>“learning_rate”</em></strong> of 1e-4. We configure the Trainer to evaluate every <strong>125</strong> steps, allowing us to monitor the learning progress. By using the <strong><em>“save_strategy”</em></strong> parameter, we save the model weights at each evaluation point, every <strong>125</strong> steps. This strategy will allow us, later on, to select the weights based on evaluation performance. In this way, we could potentially deal with <strong>overfittig</strong> by choosing weights from an earlier training stage.</p><pre>import transformers<br>from datetime import datetime<br><br>project = &quot;journal-finetune&quot;<br>base_model_name = &quot;mistral&quot;<br>run_name = base_model_name + &quot;-&quot; + project<br>output_dir = &quot;./&quot; + run_name<br><br>trainer = transformers.Trainer(<br>    model=model,<br>    train_dataset=tokenized_train_io_recipes,<br>    eval_dataset=tokenized_validation_io_recipes,<br>    args=transformers.TrainingArguments(<br>        output_dir=output_dir,<br>        warmup_steps=1,<br>        per_device_train_batch_size=2,<br>        gradient_accumulation_steps=1,<br>        gradient_checkpointing=True,<br>        max_steps=500,<br>        learning_rate=1e-4,<br>        fp16=True,<br>        fp16_full_eval=True,<br>        optim=&quot;paged_adamw_8bit&quot;,<br>        logging_steps=125,<br>        logging_dir=&quot;./logs&quot;,<br>        save_strategy=&quot;steps&quot;,<br>        save_steps=125,<br>        evaluation_strategy=&quot;steps&quot;,<br>        eval_steps=125,<br>        do_eval=True,<br><br>    ),<br>    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False),<br>)<br><br>model.config.use_cache = False<br>trainer.train()</pre><p>While training, we can analyze the evaluation performance. The image below shows the evaluation metrics computed every 125 steps. It presents how training and validation loss decrease similarly from <strong>1 to 0.93</strong>, indicating steady improvement at the end of the fine-tuning process that took <strong>55 minutes</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*n0YTW_YCeIvELzfibNLIww.png" /><figcaption>Training and validation loss evolution during fine-tuning</figcaption></figure><p>Now that we will be loading the model again into the Colab session, there will not be enough GPU memory to host the base model and the fine-tuned one. Therefore, we recommend to save the weights into the PC or Google Drive. The weights can be found at: “<em>content/mistral-journal-finetune/checkpoint-500/”. </em>Then, restart the Colab Session and uplaod the weights again.</p><p>After fine-tuning, we’re ready for inference. First, we re-import the base model, then, we use the <strong>peft</strong> library to load it with the pretrained weights from training checkpoint 500 — the final and best-performing step.</p><pre>from peft import PeftModel<br><br>base_model = AutoModelForCausalLM.from_pretrained(<br>    base_model_id,<br>    quantization_config=bnb_config,<br>    device_map=&quot;auto&quot;,<br>    trust_remote_code=True,<br>)<br><br>ft_model = PeftModel.from_pretrained(base_model, &quot;/content/mistral-journal-finetune/checkpoint-750&quot;)<br><br>eval_tokenizer = AutoTokenizer.from_pretrained(base_model_id, add_bos_token=True, trust_remote_code=True)</pre><p>Finally, we can run inference on our fine-tuned model. Let’s see if fine-tuned Mistral-7b can help us cooking a delicious Pasta Bolognesa!</p><pre>eval_prompt = &quot;Giveme the ingredients and instructions for pasta bolognesa: # &quot;<br>model_input = eval_tokenizer(eval_prompt, return_tensors=&quot;pt&quot;).to(&quot;cuda&quot;)<br><br>ft_model.eval()<br>with torch.no_grad():<br>    print(eval_tokenizer.decode(ft_model.generate(**model_input, max_new_tokens=300, repetition_penalty=1.15)[0], skip_special_tokens=True))</pre><p>And here is the answer…</p><pre>Ingredients:<br>1. Brown ground beef in a large skillet; drain off fat.<br>2 lb. ground beef<br>3 cans tomato sauce<br>1 can tomatoes, cut up<br>1/4 tsp. salt<br>1/8 tsp. pepper<br>1 Tbsp. sugar<br>1/2 tsp. oregano<br>1/2 tsp. basil<br>1/2 tsp. marjoram<br>1/2 tsp. thyme<br>1/2 tsp. parsley flakes<br>1/2 tsp. garlic powder<br>1/2 tsp. onion powder<br>1/2 tsp. celery seed<br>1/2 tsp. mustard<br>1/2 tsp. Worcestershire sauce<br>1/2 tsp. Tabasco sauce<br>1 (6 oz.) pkg. spaghetti noodles<br>1/2 c. grated Parmesan cheese<br><br>Directions:<br>Brown ground beef in a large skillet; drain off fat.<br>Add all other ingredients except noodles and cheese.<br>Simmer about 1 hour. Cook noodles according to package directions.<br>Drain well.<br>Serve with meat mixture over noodles.<br>Sprinkle with cheese. Serves 6.</pre><p>Thanks for reading, see you soon!</p><p>Feel free to follow me, to contact me and any <strong>feedback </strong>is welcome in the comments.</p><p><strong>About me:</strong></p><p><em>Alejandro González, Data Scientist / AI Engineer,</em></p><p><em>MSc in Data Science at Politechnique University of Madrid and Eötvös Loránd University of Budapest, with background in Telecommunications Engineering. Experience in different Data Science and Machine Learning domains like, Data Visualization, LLMs, Unsupervised Learning, etc.</em></p><p>You can find me on:<br>👨🏻‍💼<a href="https://www.linkedin.com/in/alexgg278/">Linekdin</a><br>👨🏻‍💻<a href="https://github.com/alexgg278">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ce78db37245" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training AI to land rockets better than SpaceX!]]></title>
            <link>https://medium.com/@alecgg27895/training-ai-to-land-rockets-better-than-spacex-19cd8ec3c9bb?source=rss-b28db33b9ddd------2</link>
            <guid isPermaLink="false">https://medium.com/p/19cd8ec3c9bb</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[sarsa]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Alex Gonzalez]]></dc:creator>
            <pubDate>Thu, 30 Nov 2023 18:33:06 GMT</pubDate>
            <atom:updated>2023-11-30T18:33:06.170Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the last decade, <strong>Reinforcement Learning (RL)</strong> has become an important player within Machine Learning, but more recently, during last year, RL has gained even more importance due to its main role in training <strong>Large Language Models (LLMs)</strong>, like <strong>ChatGPT</strong>. One of the best ways to learn RL is by applying it to one of its most popular use cases, games.</p><p>In the post we get hands on with a simple RL problem, the <strong>Lunar Lander game</strong>, where the goal is to safely land a rocket on the moon. While we initially mentioned implementing an AI algorithm to solve the problem, it turns out that for less complex RL problems, a simple <strong>numpy array</strong> and a <strong>smart trick</strong> can substitute a neural network. And yes! By only using numpy arrays we will be able to land rockets better than SpaceX.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fl2Je7TZr0eLeciOoU%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fseason-5-the-simpsons-5x3-l2Je7TZr0eLeciOoU&amp;image=https%3A%2F%2Fmedia4.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExNWUyYWMzNTJjMTYzMzIyMzFmMTc4MTg1ZTVlMDU1YTkyZGZlMDNmNiZlcD12MV9pbnRlcm5hbF9naWZzX2dpZklkJmN0PWc%2Fl2Je7TZr0eLeciOoU%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="331" frameborder="0" scrolling="no"><a href="https://medium.com/media/596fb5fb21d57e449ee79c82a769f4a9/href">https://medium.com/media/596fb5fb21d57e449ee79c82a769f4a9/href</a></iframe><h4>Reinforcement learning 101</h4><p>Reinforcement learning, a branch of machine learning, it is opposed from supervised learning as it doesn’t need a predefined target variable. Instead, RL algorithms <strong>generate their own data</strong> by <strong>interacting with the environment</strong>, allowing them to master a task. A RL model just needs to know <strong>what it should learn</strong> and it will figure out how to learn it in the <strong>most optimal</strong> way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*YtsP8LTxargmqsZYOFgvXA.png" /><figcaption>Diagram by the author</figcaption></figure><p>In RL, the agent’s actions impact the environment, transitioning to a new state and feedback via rewards<strong>.</strong> The <strong>goal </strong>of the agent is to learn optimal strategies that <strong>maximize cumulative rewards</strong> over time. This iterative process, known as an <strong>episode</strong>, involves the agent taking <strong>actions</strong>, observing the <strong>outcomes</strong>, and receiving <strong>rewards</strong>. RL episodes can be <strong>infinite </strong>(training an agent for stock market trading), or <strong>finite </strong>(the lunar lander problem ending with the rocket landing or crashing).</p><p><em>If you want to learn more about RL basics and different algorithms, check out </em><a href="https://www.youtube.com/watch?v=2pWv7GOvuf0&amp;ab_channel=GoogleDeepMind"><em>David Silver’s classes on YouTube</em></a><em>.</em></p><h4>The Lunar Lander problem</h4><p>The Lunar Lander problem, available through <strong>OpenAI’s gym library</strong>, follows the standard structure of a RL problem. It features an agent that takes actions, an environment with dynamic states, and a reward system.</p><p><strong>The Environment: </strong>The whole system containing the moon’s surface, the landing spot, etc. The state is represented by an 8-element numpy array. These elements correspond to the rocket’s coordinates, linear velocities, angle, angular velocity, and two boolean values indicating the contact of the rocket’s legs with the surface.</p><p><strong>The agent: </strong>It is represented by the rocket aiming to land on the moon. However, it’s the algorithm we’re implementing that will govern the rocket’s actions.</p><p><strong>The actions: </strong>The agent is capable of taking 4 discrete actions.</p><ol><li>Firing the left engine.</li><li>Firing the right engine.</li><li>Firing the main (central) engine.</li><li>Do nothing (not firing any engine).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*2J85342hdnFUNWorQ03nsQ.png" /><figcaption>We can see the whole system (environement), where the rocket (agent) is firing the main engine (3rd action). Photo by the author.</figcaption></figure><p><strong>The reward: </strong>At each step, the reward is provided by the environment and it depends on the state and the action taken. Successfully landing the rocket within the designated area (yellow flags) is highly rewarded. However, crashing involves a significant negative reward. Additionally, activating the engines results in small negative rewards. Thus, the agent’s objective is to land safely within the specified area while minimizing engine usage. <em>More info about rewards in </em><a href="https://www.gymlibrary.dev/environments/box2d/lunar_lander/#"><em>Gym docs</em></a><em>.</em></p><h4>How will our agent learn? <strong>→</strong> SARSA</h4><p>An optimal RL agent chooses actions that <strong>maximize long-term rewards</strong> for each state. To make these decisions, RL algorithms often use the <strong>state-action value function, Q(s,a)</strong>. Calculating Q(s,a) varies across different methods, with <strong>Monte Carlo</strong> and <strong>SARSA </strong>being two main approaches.</p><p><strong><em>Monte Carlo Control</em></strong></p><p><strong>Monte Carlo</strong> lets the agent take actions while storing the rewards until the episode concludes. Then, the value of each <em>Q(s,a)</em> is updated based on the accumulated rewards. Therefore, this method is limited to finite environments. Monte Carlo methods <strong>update Q(s,a) only after each episode ends</strong>, making the learning process <strong>slower </strong>than SARSA.</p><p><strong><em>SARSA</em></strong></p><p><strong>SARSA (State-Action-Reward-State-Action)</strong> updates <strong><em>Q(s,a)</em></strong> at <strong>every step</strong> of the agent in the environment by:</p><ol><li>Selecting the action with the highest <strong><em>Q(s,a)</em></strong> for the current state <strong><em>s.</em></strong></li><li>Receiving a reward <strong><em>R </em></strong>from the environment.</li><li>Calculating the value of the next state <strong><em>s’</em></strong> as the highest <strong><em>Q(s’,a)</em></strong> out of all actions.</li><li>Using <strong><em>R</em></strong> and <strong><em>Q(s’,a)</em></strong> to update <strong><em>Q(s,a)</em></strong></li></ol><p>The update is performed using the <strong>Bellman equation</strong> and the SARSA update formula, which gradually adjusts the initial value based on the latest computation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*HadXFIwSaxMfZK6vRWgNPQ.png" /><figcaption>SARSA update formula</figcaption></figure><p>The SARSA update formula refreshes the state-action value Q(s,a), by taking a step forward and computing the new value as the sum of the received reward (R) and the discounted value of the next state. The discount (γ) represents how important is the value of the next state for the update of the current state. The Q(s, a) value is then updated using a step size parameter alpha (α), adjusted in the direction of the difference between the newly calculated value, also known as<strong> TD-target (R + γ* Q(s’, a’))</strong> and the previous Q(s,a) value.</p><p>One of the main advantages of SARSA is that the agent is learning after every step if the action was good or bad by updating Q(s,a). As a consequence, the agent is able to <strong>take better actions</strong> for states that have been previously visited even <strong>within the same training episode</strong>, resulting in <strong>faster learning</strong>.</p><pre>class SarsaAgent(BaseAgentTabularDiscreteStatesDiscreteActions):<br><br>    def __init__(self,<br>                 env,<br>                 discount=1,<br>                 alpha=0.01,<br>                 alpha_decay=None,<br>                 epsilon=0.1,<br>                 epsilon_decay=0.995,<br>                 observation_state_shape=None,<br>                 action_space=None):<br>        super().__init__(env,<br>                         discount,<br>                         alpha,<br>                         alpha_decay,<br>                         epsilon,<br>                         epsilon_decay,<br>                         observation_state_shape,<br>                         action_space)<br><br>    def compute_target(self, reward, next_state_value):<br>        # Compute TD-target with a single step backup<br>        target = reward + self.discount * next_state_value<br>        return target<br><br>    def sarsa_update(self, state, action, target):<br>        # Update state-action value by alpha in the direction of TD error<br>        self.q[state][action] = self.q[state][action] + self.alpha * (target - self.q[state][action])<br><br>    def select_action(self, state):<br>        # Select the action with the highest Q(s,a) with probability 1 - epsilon (exploitation)<br>        if np.random.rand() &gt;= self.epsilon:<br>            return np.argmax(self.q[state])<br>        # Select a random action with probability epsilon (exploration)<br>        else:<br>            return np.random.randint(len(self.q[state]))<br><br>    def train_episode(self, max_steps_episode=None):<br>        # Variable to run episode until terminal state<br>        terminated = False<br><br>        # Get initial observation (state) and action<br>        state, info = self.env.reset()<br>        <br>        # Select action given the state<br>        action = self.select_action(state)<br><br>        while not terminated:<br>            # Environment transition for current state and selected action to the next state and reward<br>            next_state, reward, terminated, truncated, info = self.env.step(self.index_to_action(action))<br><br>            # Calculate value of next step as the e-greedy value for the state (argmax(s, a))<br>            next_action = self.select_action(next_state)<br>            next_state_value = self.q[next_state][next_action]<br><br>            # Calculate TD target after a single step (sarsa)<br>            target = self.compute_target(reward, next_state_value)<br><br>            # Update current state value alpha in the direction of TD error<br>            self.sarsa_update(state, action, target)<br><br>            # next_state and next_action are current state and action<br>            state = next_state<br>            action = next_action</pre><p>The code snippet above shows a complete episode of training the RL agent using SARSA algorithm along with <strong>epsilon greedy</strong> method, which ensures taking <strong>random actions with probability epsilon</strong> to promote <strong>exploration </strong>and to avoid getting stuck in a local optima.</p><p>In our analysis, we have presented two methods using Q(a,s) to determine optimal actions for each state. However, the <strong>challenge </strong>is <strong>calculating and storing Q values for all state-action pairs</strong>. While a <strong>numpy array</strong> is enough for <strong>discrete spaces</strong>, it is <strong>not suitable</strong> for continuous spaces with <strong>infinite states</strong>. In this case, function approximators like Neural Networks have been used to estimate Q. To address this issue without using Neural Networks, we’ll use <strong>discretization</strong> as a clever solution.</p><h4>State discretization</h4><p>In the lunar lander problem, the rocket’s coordinates ranging from -1.5 to 1.5, can take any of the infinite values within this range. Therefore, storing a Q value for each combination of state-action is unfeasible. To overcome this challenge, we apply a simple trick: the continuous values are <strong>discretized into bins</strong>. This approach allows the algorithm to learn effectively while keeping the number of states manageable, ensuring they can fit into the memory of the machine.</p><p><em>The size of the bins will determine the balance between the level of precision (smaller bins → higher dimensionality) and the number of states (larger bins → lower dimensionality).</em></p><pre># Set the desired number of bins per feature<br>num_bins = 20<br><br># Compute the range size for each feature<br>ranges = self.env.observation_space.high - self.env.observation_space.low<br><br># Compute the size of each bin given the desried num of bins<br>bin_sizes = ranges / num_bins<br><br># Discretize the state array by mapping each value to its corresponding bin<br>state_discrete = np.floor(state / bin_sizes).astype(int)</pre><p>The code discretizes an input state array by setting the number of bins per feature, e.g., <em>num_bins = 20.</em> Then, the <em>ranges </em>of the features are calculated by subtracting the lowest value from the max. value. The <em>bin_size </em>is calculated by dividing the <em>ranges </em>by the <em>num_bins. </em>Finally, each value from the state vector is discretized by dividing them by the <em>bin_size </em>and rounding them down to the closer integer.</p><h4>Training the RL agent</h4><p>The agent was trained for <strong>30,000 episodes</strong>. Similarly to a newborn, in the first episodes of liefe, the agent explores different actions under different states. As expected, the rocket constantly crashes — luckily we don’t need to spend millions of $ on a new one, I bet Elon Musk is jealous.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*y3xkXZWbPAtdBOnQSv9Guw.gif" /><figcaption>Rocket crashing on initial training episodes. GIF by the Author</figcaption></figure><p>During training, the focus of the <strong>agent shifts from exploration to exploitation</strong> of optimal actions. This is addressed by using a <strong>decaying ε-greedy strategy</strong> and a <strong>decaying learning rate</strong>. These parameters transition from <strong>higher values</strong> in initial episodes, aiding <strong>faster learning and exploration</strong>, to <strong>lower values</strong> that promote <strong>finer learning and exploitation</strong> of optimal actions. As the chart below shows, the agent was able to gradually optimize the cumulative reward over the training episodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WuFpbqztUc4L0nRjlVcuLw.png" /><figcaption>Accumulated reward over training episodes. Chart by the author.</figcaption></figure><p><strong>Finally our rocket was able to land successfully in the target area!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*wlsj-L5yYvTBrqhCadvsbg.gif" /><figcaption>Rocket landing after training. GIF by the Author</figcaption></figure><p>Thanks for reading, see you soon!</p><p>Feel free to follow me, to contact me and any <strong>feedback </strong>is welcome in the comments.</p><p><strong>About me:</strong></p><p><em>Alejandro González, Data Scientist / AI Engineer,</em></p><p><em>MSc in Data Science at Politechnique University of Madrid and Eötvös Loránd University of Budapest, with background in Telecommunications Engineering. Experience in different Data Science and Machine Learning domains like, Data Visualization, LLMs, Unsupervised Learning, etc.</em></p><p>You can find me on:<br>👨🏻‍💼<a href="https://www.linkedin.com/in/alexgg278/">Linekdin</a><br>👨🏻‍💻<a href="https://github.com/alexgg278">GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19cd8ec3c9bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>