<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alessandro Avanzi on Medium]]></title>
        <description><![CDATA[Stories by Alessandro Avanzi on Medium]]></description>
        <link>https://medium.com/@ales.avanzi?source=rss-89f1d0a91a20------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bPU0u3RQ4CEXSYSZ_Cy7AQ.jpeg</url>
            <title>Stories by Alessandro Avanzi on Medium</title>
            <link>https://medium.com/@ales.avanzi?source=rss-89f1d0a91a20------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:35:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ales.avanzi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Improve LLM Fine-Tuning: Tips for Defining Training Samples from PDF Documents]]></title>
            <link>https://medium.com/@ales.avanzi/improve-your-fine-tuning-tips-for-defining-training-samples-from-pdf-documents-9a0e8b13c90b?source=rss-89f1d0a91a20------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a0e8b13c90b</guid>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Alessandro Avanzi]]></dc:creator>
            <pubDate>Sat, 21 Sep 2024 14:47:58 GMT</pubDate>
            <atom:updated>2024-10-05T12:22:20.010Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*ZRTW3WHyOMgiZtXqG3f8Tg.jpeg" /><figcaption>Generated with <a href="https://www.mage.space/">https://www.mage.space/</a></figcaption></figure><p>In a previous article, I delved into the potential of <strong>fine-tuning a small language model (LLM) using data extracted from technical documentation</strong>. This approach, while promising, isn’t the only solution for leveraging domain-specific knowledge. Another increasingly popular method is Retrieval-Augmented Generation (RAG), which combines retrieval-based models with generative capabilities to provide up-to-date, context-aware responses. However, by focusing on fine-tuning, I was able to gain a deeper understanding of the process, particularly in how to effectively define and prepare training samples. This insight has proven invaluable in optimizing LLM performance for specialized tasks, making it a crucial part of any customization strategy.</p><p>The results of the fine-tuning experiment were promising, <strong>demonstrating the potential for small language models to adapt and improve their performance on specialized tasks</strong>. However, the quality of the data used during the fine-tuning process played a critical role in achieving these results. High-quality, relevant data ensures that the model captures the nuances of the domain, while poor or irrelevant data can degrade its performance. Additionally, selecting the right fine-tuning parameters — such as learning rate, batch size, and epochs — proved essential for balancing accuracy and model efficiency.</p><p><strong>One challenge I encountered was that the model already had prior knowledge of the technical content I selected. Unfortunately, this knowledge was either outdated or incorrect, which occasionally led to inconsistencies in the fine-tuned model’s responses.</strong> This highlighted the importance of not only curating fresh, accurate data but also understanding the pre-existing biases or knowledge embedded within the model, as these can interfere with the fine-tuning process.</p><p>To address the issue of outdated or incorrect prior knowledge in the model, two potential approaches can be explored. The first approach focuses on inhibiting the model’s prior knowledge from surfacing in its responses, essentially suppressing any pre-existing information that may conflict with the newly fine-tuned data. <strong>The second approach, which this article will explore, involves enhancing the fine-tuned content, ensuring that the updated information takes precedence over the model’s prior knowledge.</strong></p><p>This strategy aims to “amplify” the influence of the fine-tuned layer, ensuring that the model’s responses are more aligned with the fresh, domain-specific knowledge gained during the fine-tuning process. <strong>By prioritizing the new content, this approach seeks to minimize interference from outdated or irrelevant information embedded in the model, resulting in more accurate and reliable outputs.</strong></p><p>Let’s first recap some of the steps. I executed the code in Google Colab, utilizing the <strong>Mistral 7B Instruct</strong> language model. The model was obtained and fine-tuned using tools provided by Hugging Face. The following code loads the model and tokenizer. The model is loaded in a 4-bit quantized format, which allows it to run in the free Colab without encountering out-of-memory issues.</p><pre>import torch<br>import transformers<br>import re, random<br>from huggingface_hub import login<br>from transformers import AutoTokenizer, AutoModelForCausalLM<br><br>model_id = &quot;mistralai/Mistral-7B-Instruct-v0.1&quot;<br>bnb_config = transformers.BitsAndBytesConfig(<br>load_in_4bit=True,<br>bnb_4bit_use_double_quant=True,<br>bnb_4bit_quant_type=&quot;nf4&quot;,<br>bnb_4bit_compute_dtype=torch.bfloat16<br>)<br><br>login(token=&quot;insert huggingface access token here&quot;)<br><br>model = transformers.AutoModelForCausalLM.from_pretrained(<br>model_id,<br>trust_remote_code=True,<br>quantization_config=bnb_config,<br>device_map=&#39;auto&#39;,<br>)<br><br>tokenizer = transformers.AutoTokenizer.from_pretrained(<br>model_id,<br>padding_side=&quot;left&quot;,<br>add_eos_token=True,<br>add_bos_token=True<br>)<br>tokenizer.pad_token = tokenizer.eos_token</pre><p><strong>In the previous article, I outlined the steps for generating question-answer pairs using the model itself as a tool for fine-tuning. </strong>By guiding the model to generate relevant queries and responses based on the technical documentation, I was able to curate a tailored dataset for fine-tuning. Since this procedure has already been covered in detail, it will not be repeated in this article. The question-answer pairs were saved in a JSON file, which can be loaded using the following code and then split into training and evaluation sets.</p><pre>import json<br>import random<br>from datasets import Dataset<br><br>with open(&quot;/content/drive/MyDrive/data.json&quot;, &quot;r&quot;) as f:<br>   dataset = json.load(f)<br><br>if isinstance(dataset, list):<br><br>    random.shuffle(dataset)<br><br>    train_dataset_list = dataset[:1000]<br>    eval_dataset_list = dataset[1000:1200]<br><br>    for item in train_dataset_list:<br>        item[&quot;input&quot;] = &quot;&quot;<br><br>    for item in eval_dataset_list:<br>        item[&quot;input&quot;] = &quot;&quot;<br><br>    with open(&quot;/content/drive/MyDrive/train_data.json&quot;, &quot;w&quot;) as f:<br>        json.dump(train_dataset_list, f)<br><br>    with open(&quot;/content/drive/MyDrive/eval_data.json&quot;, &quot;w&quot;) as f:<br>        json.dump(eval_dataset_list, f)<br><br>else:<br>    print(&quot;Data is not a list, cannot shuffle.&quot;)</pre><p>Before tokenizing the samples, they should be arranged in the format suggested by the selected model card. For the Mistral 7B Instruct model, the instructions provided to the model should be surrounded by the INST directive. Additionally, the model assumes a structure where the “user” makes a request and the “assistant” provides a response. The tokenizer includes a chat template function that helps implement this scheme.</p><p><strong>Here’s the first tip</strong>: when formatting the samples by assigning content to the “user” and “assistant” roles, I initially left the <em>###Input</em> field empty, only populating the <em>###Instruction</em> and <em>###Response</em> sections. <strong>In this fine-tuning attempt, however, I decided to use the ###Input field to define a consistent context — a recurring text that remains the same across samples. This context can then be leveraged during inference to guide the model’s answers, aligning them more closely with the information from the fine-tuned layer. </strong>I selected and input that was representative of the technical content of the PDF document: <em>The context if SpaceWire protocol identification as reported in ECSS E ST 50 51C.</em></p><pre>import json<br>from datasets import Dataset<br><br>def format_prompt_user(sample):<br>    return f&quot;&quot;&quot;<br>Below is an instruction that describes a task or a question, paired with an optional input that provides further context. Write a response that appropriately completes the request.<br><br>### Instruction:<br>{sample[&quot;instruction&quot;]}<br><br>### Input:<br>The context if SpaceWire protocol identification as reported in ECSS E ST 50 51C<br>&quot;&quot;&quot;<br><br>def format_prompt_assistant(sample):<br>    return f&quot;&quot;&quot;<br>### Response:<br>{sample[&quot;output&quot;]}<br>&quot;&quot;&quot;<br><br>def format_prompt_chat(sample):<br>    chat = [<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: format_prompt_user(sample)},<br>        {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: format_prompt_assistant(sample)}<br>    ]<br>    text = tokenizer.apply_chat_template(chat, tokenize=False)<br>    return {&quot;formatted_text&quot;: text}<br><br>def generate_and_tokenize_prompt(sample):<br>    formatted_text = sample[&quot;formatted_text&quot;]<br>    result = tokenizer(<br>        formatted_text,<br>        truncation=True,<br>        max_length=max_length,<br>        padding=&quot;max_length&quot;,<br>    )<br>    result[&quot;labels&quot;] = result[&quot;input_ids&quot;].copy()<br>    return result<br><br>max_length = 2048  # differs from datasets to datasets<br><br>with open(&quot;/content/drive/MyDrive/train_data.json&quot;, &quot;r&quot;) as f:<br>    train_dataset_list = json.load(f)<br><br>with open(&quot;/content/drive/MyDrive/eval_data.json&quot;, &quot;r&quot;) as f:<br>    eval_dataset_list = json.load(f)<br><br>train_dataset = Dataset.from_list(train_dataset_list)<br>eval_dataset = Dataset.from_list(eval_dataset_list)<br><br>formatted_train_dataset = train_dataset.map(lambda x: format_prompt_chat(x))<br>formatted_eval_dataset = eval_dataset.map(lambda x: format_prompt_chat(x))<br><br>tokenized_train_dataset = formatted_train_dataset.map(generate_and_tokenize_prompt)<br>tokenized_eval_dataset = formatted_eval_dataset.map(generate_and_tokenize_prompt)<br><br>print(f&#39;Training data size: {len(tokenized_train_dataset)}&#39;)<br>print(f&#39;Validation data size: {len(tokenized_eval_dataset)}&#39;)</pre><p>With the given sample, the following code implements the fine-tuning. With the specified batch size, accumulation steps, and maximum epochs, the fine-tuning process iterates over the entire set of samples 8 times.</p><p><strong>Here’s the second tip: in this attempt, I increased the r parameter of the LoRA layer, allowing the model more trainable capacity. </strong>While there is plenty of literature on choosing the optimal values for <strong>r</strong> and <strong>lora_alpha</strong>, I won’t cover those details here. Instead, I experimented with various parameter combinations, and this particular attempt is showing results with a higher value of <strong>r</strong>.</p><pre>from peft import LoraConfig, PeftModel, prepare_model_for_kbit_training, get_peft_model<br>from datetime import datetime<br><br>def print_trainable_parameters(model):<br>    &quot;&quot;&quot;<br>    Prints the number of trainable parameters in the model.<br>    &quot;&quot;&quot;<br>    trainable_params = 0<br>    all_param = 0<br>    for _, param in model.named_parameters():<br>        all_param += param.numel()<br>        if param.requires_grad:<br>            trainable_params += param.numel()<br>    print(<br>        f&quot;trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}&quot;<br>    )<br><br>project = &quot;ECSS-E-ST-50-51C-finetune-test-#3&quot;<br>base_model_name = &quot;Mistral-7B-Instruct&quot;<br>run_name = base_model_name + &quot;-&quot; + project<br>output_dir = &quot;/content/drive/MyDrive/&quot; + run_name<br><br># Preapre model<br>model.config.pretraining_tp = 1<br>model.gradient_checkpointing_enable()<br>model = prepare_model_for_kbit_training(model)<br>peft_config = LoraConfig(<br>        r=128,<br>        lora_alpha=32,<br>        lora_dropout=0.05,<br>        bias=&quot;none&quot;,<br>        task_type=&quot;CAUSAL_LM&quot;,<br>        target_modules=[&quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;, &quot;gate_proj&quot;, &quot;up_proj&quot;, &quot;down_proj&quot;, &quot;lm_head&quot;]<br>    )<br>model = get_peft_model(model, peft_config)<br>print_trainable_parameters(model)<br><br>trainer = transformers.Trainer(<br>    model=model,<br>    train_dataset=tokenized_train_dataset,<br>    eval_dataset=tokenized_eval_dataset,<br>    args=transformers.TrainingArguments(<br>        output_dir=output_dir,<br>        warmup_steps=5,<br>        per_device_train_batch_size=8,<br>        gradient_accumulation_steps=2,<br>        gradient_checkpointing=True,<br>        max_steps=500,<br>        learning_rate=2.5e-5, # Want about 10x smaller than the Mistral learning rate<br>        logging_steps=50,<br>        bf16=True,<br>        optim=&quot;paged_adamw_8bit&quot;,<br>        logging_dir=&quot;./logs&quot;,        # Directory for storing logs<br>        save_strategy=&quot;steps&quot;,       # Save the model checkpoint every logging step<br>        save_steps=50,                # Save checkpoints every 50 steps<br>        evaluation_strategy=&quot;steps&quot;, # Evaluate the model every logging step<br>        eval_steps=50,               # Evaluate and save checkpoints every 50 steps<br>        do_eval=True,                # Perform evaluation at the end of training<br>        #report_to=&quot;wandb&quot;,           # Comment this out if you don&#39;t want to use weights &amp; baises<br>        run_name=f&quot;{run_name}-{datetime.now().strftime(&#39;%Y-%m-%d-%H-%M&#39;)}&quot;          # Name of the W&amp;B run (optional)<br>    ),<br>    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False),<br>)<br>model.config.use_cache = False  # silence the warnings. Please re-enable for inference!<br><br>if torch.cuda.device_count() &gt; 1: # If more than 1 GPU<br>    model.is_parallelizable = True<br>    model.model_parallel = True<br><br>#trainer.train(resume_from_checkpoint = True)<br>trainer.train()</pre><p>Once the training is finished, the code below enables loading the fine-tuned model and interacting with it via a chat interface.</p><pre>import torch<br>import transformers<br>import re, random<br>from huggingface_hub import login<br>from transformers import AutoTokenizer, AutoModelForCausalLM<br>from peft import LoraConfig, PeftModel, PeftConfig, prepare_model_for_kbit_training, get_peft_model<br><br>model_id = &quot;mistralai/Mistral-7B-Instruct-v0.1&quot;<br><br>project = &quot;ECSS-E-ST-50-51C-finetune-test-#21&quot;<br>base_model_name = &quot;Mistral-7B-Instruct&quot;<br>run_name = base_model_name + &quot;-&quot; + project<br>output_dir = &quot;/content/drive/MyDrive/&quot; + run_name<br>check_point = &quot;checkpoint-500&quot;<br>adapter_model_id = output_dir + &quot;/&quot; + check_point<br><br>bnb_config = transformers.BitsAndBytesConfig(<br>load_in_4bit=True,<br>bnb_4bit_use_double_quant=True,<br>bnb_4bit_quant_type=&quot;nf4&quot;,<br>bnb_4bit_compute_dtype=torch.bfloat16<br>)<br><br>config = PeftConfig.from_pretrained(adapter_model_id)<br><br>login(token=&quot;insert huggingface access token here&quot;)<br><br>model = transformers.AutoModelForCausalLM.from_pretrained(<br>model_id,<br>trust_remote_code=True,<br>quantization_config=bnb_config,<br>device_map=&#39;auto&#39;,<br>)<br><br>ft_model = PeftModel.from_pretrained(<br>model,<br>adapter_model_id,<br>)<br><br>tokenizer = transformers.AutoTokenizer.from_pretrained(<br>model_id,<br>padding_side=&quot;left&quot;,<br>add_eos_token=True,<br>add_bos_token=True<br>)<br>tokenizer.pad_token = tokenizer.eos_token</pre><pre>def generate_response(prompt):<br>  chat = [<br>      {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>  ]<br>  text = tokenizer.apply_chat_template(chat, tokenize=False)<br>  encodeds = tokenizer(text, return_tensors=&quot;pt&quot;, add_special_tokens=False).to(&#39;cuda&#39;)<br>  generated_ids = ft_model.generate(**encodeds, pad_token_id=tokenizer.eos_token_id, max_new_tokens=1024, do_sample=True)<br>  decoded = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]<br><br>  # Remove prompt and special characters<br>  cleaned_response = decoded.split(prompt, 1)[1]  # Split by prompt and keep the second part<br>  cleaned_response = tokenizer.clean_up_tokenization(cleaned_response)  # Remove special tokens<br>  cleaned_response = re.sub(r&quot;\ \[/INST\] &quot;, &quot;&quot;, cleaned_response)  # Match and replace [/INST] followed by a space<br><br>  return cleaned_response<br><br># Chat loop<br>while True:<br>  user_input = input(&quot;You: &quot;)<br>  if user_input.lower() == &quot;quit&quot;:<br>    break<br>  response = generate_response(user_input)<br>  print(f&quot;Computer: {response}&quot;)</pre><p>As for the previous article, the fine-tuned model was tested on 20 questions that were not included in the dataset; these questions were manually drafted from the PDF content. The model’s responses were rated on a scale from 1 to 10, where 1 indicates a completely nonsensical response and 10 represents a perfectly matched and valid answer. Both the pre-trained and fine-tuned models underwent testing on the identical set of questions.</p><p><strong>It’s important to note that when supplying the model with samples for inference in this attempt, all questions were paired with the recurring text: “</strong><em>The context is SpaceWire protocol identification as reported in ECSS E ST 50 51C</em><strong>” which was utilized during the fine-tuning process.</strong></p><p>On average, the pre-trained model received a score of 4.7, indicating that its prior knowledge was generally outdated or inaccurate. The model fine-tuned in the previous article scored 6.8. In this attempt, the model fine-tuned using the same training set and incorporating the two tips outlined in this article achieved a score of 8.0, demonstrating consistent improvement. Some of the answers were still not aligned with the technical content from the PDF, highlighting the need to improve the quality of the data samples.</p><p><strong>In conclusion:</strong></p><ul><li><strong>A small language model was fine-tuned for Q&amp;A using data extracted from a technical PDF</strong> document to explore the potential of fine-tuning.</li><li>However, an issue arose because <strong>the model already had some prior knowledge of the selected technical content, which was often outdated or inaccurate.</strong></li><li><strong>By carefully crafting the training sample’s Input field and increasing the value of r</strong>, it was possible to enhance the model’s responses by partially masking the prior knowledge and allowing it to convey the information from the fine-tuned layers more effectively.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a0e8b13c90b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Effective Q&A? LLM Fine-Tuning with PDF Documentation]]></title>
            <link>https://medium.com/@ales.avanzi/effective-q-a-llm-fine-tuning-with-pdf-documentation-91b7fc458ced?source=rss-89f1d0a91a20------2</link>
            <guid isPermaLink="false">https://medium.com/p/91b7fc458ced</guid>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Alessandro Avanzi]]></dc:creator>
            <pubDate>Sat, 13 Jul 2024 18:50:41 GMT</pubDate>
            <atom:updated>2024-09-21T14:22:05.290Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*XL5hZsr5D_BKpg5F0ubfIA.jpeg" /><figcaption>Generated with <a href="https://www.mage.space/">https://www.mage.space/</a></figcaption></figure><p>In the realm of natural language processing, <strong>achieving precise question-answering capabilities from PDF documents</strong> can be approached through two primary methodologies: fine-tuning language models and using a a retrieval-based method such as RAG. While RAG integrates information retrieval techniques with generative models to synthesize answers from retrieved documents, <strong>fine-tuning involves adapting pre-trained language models to specific tasks and datasets for enhanced performance</strong>.</p><p><strong>This article delves into the potential of fine-tuning small language models, specifically for the purpose of extracting and answering questions from technical PDF documents.</strong> Given my background in aerospace engineering, I have chosen a standard European Cooperation for Space Standardization (ECSS) document in PDF format as the basis for this exploration. This particular standard delineates the use of data protocols over the Space Wire link (a type of data link used only in space systems), providing a complex dataset ideal for demonstrating the efficacy of fine-tuning techniques.</p><p>Throughout this article, we will walk through the process of <strong>preparing the ECSS PDF document for fine-tuning</strong>, discuss the intricacies of adapting a language model for this specific technical domain, and evaluate the outcomes of our fine-tuning efforts. The aim is to explore together how fine-tuning might enhance question-answering from specialized PDF documentation, without claiming to have definitive answers but rather inviting curiosity and discovery along the way.</p><p>For this exploration, I executed the code in Google Colab, utilizing the <strong>Mistral 7B Instruct</strong> language model. The model was obtained and fine-tuned using tools provided by Hugging Face. The code below installs the necessary libraries, mounts Google Drive where the PDF document is stored, and where to save the results of the fine-tuning process.</p><pre>! pip -q install bitsandbytes accelerate xformers einops peft datasets<br>! pip -q install --upgrade huggingface_hub<br>! pip -q install PyPDF2 pycryptodome==3.15.0 sentencepiece<br><br>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)<br><br>import warnings<br>warnings.filterwarnings(&#39;ignore&#39;)</pre><p>The following code loads the model and tokenizer. Be sure to insert a valid token in the specified line to download the model from Hugging Face. Note that the model is loaded in a 4-bit quantized format, which allows it to run in the free Colab without encountering out-of-memory issues.</p><pre>import torch<br>import transformers<br>import re, random<br>from huggingface_hub import login<br>from transformers import AutoTokenizer, AutoModelForCausalLM<br><br>model_id = &quot;mistralai/Mistral-7B-Instruct-v0.1&quot;<br>bnb_config = transformers.BitsAndBytesConfig(<br>load_in_4bit=True,<br>bnb_4bit_use_double_quant=True,<br>bnb_4bit_quant_type=&quot;nf4&quot;,<br>bnb_4bit_compute_dtype=torch.bfloat16<br>)<br><br>login(token=&quot;insert huggingface access token here&quot;)<br><br>model = transformers.AutoModelForCausalLM.from_pretrained(<br>model_id,<br>trust_remote_code=True,<br>quantization_config=bnb_config,<br>device_map=&#39;auto&#39;,<br>)<br><br>tokenizer = transformers.AutoTokenizer.from_pretrained(<br>model_id,<br>padding_side=&quot;left&quot;,<br>add_eos_token=True,<br>add_bos_token=True<br>)<br>tokenizer.pad_token = tokenizer.eos_token</pre><p>The code below opens the PDF document and extracts the text, removing a specified number of initial pages (such as titles and credits) and final pages (such as references and blanks). It also removes headers and footers from the text. This process may vary from document to document, but generally, some form of pre-processing is needed to eliminate unnecessary text.</p><pre>import PyPDF2<br>import re<br><br>def pdf_to_text(pdf_path, skip_start_pages=0, skip_last_pages=0, header_lines=1, footer_lines=1):<br>  with open(pdf_path, &#39;rb&#39;) as pdf_file:<br>    pdf_reader = PyPDF2.PdfReader(pdf_file)<br>    text = &quot;&quot;<br>    num_pages = len(pdf_reader.pages)<br><br>    for page_num in range(skip_start_pages - 1, num_pages-skip_last_pages):<br>      page = pdf_reader.pages[page_num]<br>      page_text = page.extract_text()<br><br>      lines = page_text.splitlines(True)[header_lines:-footer_lines] # Removing header and footer<br><br>      lines_modified = []<br>      for line in lines:<br><br>        # Optional pre-processing of lines to correct errors<br>        # None        <br><br>        lines_modified.append(line)<br><br>      lines_joined = &quot;&quot;.join(lines_modified)<br>      text += lines_joined<br>  return text<br><br>pdf_file_path = &quot;/content/drive/MyDrive/ECSS-E-ST-50-51C(5February2010).pdf&quot;<br>raw_text = pdf_to_text(pdf_file_path, skip_start_pages=5, skip_last_pages=0, header_lines=2, footer_lines=1)<br>print(len(raw_text))</pre><p>My understanding is that the text extracted from the PDF document cannot be used as-is to fine-tune a Q&amp;A model. For the model to effectively answer questions about the PDF content, the text must be formatted as question-and-answer pairs. There are different ways to generate these pairs, one of which is to do it manually. If done by an expert on the document’s content, this method is likely to provide the highest quality context.</p><p>Other methods involve using language models, such as ChatGPT. In this case, I used the same model I’m fine-tuning, Mistral 7B Instruct. The code below extracts random chunks of text from the data source and asks the model to generate one question that can be answered within the given chunk. Then, the question and input text are provided to the model again, prompting it to answer the question. The question, input text, and answer are saved in a JSON file. The example code generates about 1200 question/input/answer triplets from a PDF document of about 20 pages.</p><pre>import json<br><br>def format_question_prompt(input):<br>  return f&quot;&quot;&quot;Below is a input text extracted from a document. Generate one question that is answered with the given text. Do not include any answer in the generated text.<br><br>### Input:<br> {input}<br>&quot;&quot;&quot;<br><br>def format_answer_prompt(input,instruction):<br>  return f&quot;&quot;&quot;Below is an instruction that describes a task or a question, paired with an input that provides context. Write a response that appropriately completes the request.<br><br>### Instruction:<br>{instruction}<br><br>### Input:<br>{input}<br>&quot;&quot;&quot;<br><br>def generate_response(prompt):<br>  chat = [<br>      {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>  ]<br>  text = tokenizer.apply_chat_template(chat, tokenize=False)<br>  encodeds = tokenizer(text, return_tensors=&quot;pt&quot;, add_special_tokens=False).to(&#39;cuda&#39;)<br>  generated_ids = model.generate(**encodeds, pad_token_id=tokenizer.eos_token_id, max_new_tokens=1024, do_sample=True)<br>  decoded = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]<br><br>  # Remove prompt and special characters<br>  cleaned_response = decoded.split(prompt, 1)[1]  # Split by prompt and keep the second part<br>  cleaned_response = tokenizer.clean_up_tokenization(cleaned_response)  # Remove special tokens<br>  cleaned_response = re.sub(r&quot;\ \[/INST\] &quot;, &quot;&quot;, cleaned_response)  # Match and replace [/INST] followed by a space<br><br>  return cleaned_response<br><br>sample_char_len_ar = [1024, 512, 256]<br>num_interations = 400<br>data= []<br><br>for  sample_char_len in sample_char_len_ar:<br>    for i in range(num_interations):<br>        print(i)<br>        random_integer = random.randint(0, len(raw_text) - sample_char_len)<br>        document_sample = raw_text[random_integer:random_integer+sample_char_len]<br>        words = document_sample.split()<br>        words.pop(0)<br>        words.pop()<br>        document_sample = &quot; &quot;.join(words)<br>        document_sample = document_sample.replace(&quot;\u2010&quot;, &quot; &quot;)<br>        prompt = format_question_prompt(document_sample)<br>        instruction = generate_response(prompt)<br><br>        prompt = format_answer_prompt(document_sample, instruction)<br>        output = generate_response(prompt)<br><br>        data.append({<br>          &quot;instruction&quot;: instruction,<br>          &quot;input&quot;: &#39; &#39;.join(document_sample.split()),<br>          &quot;output&quot;: output<br>        })<br><br>with open(&quot;/content/drive/MyDrive/data.json&quot;, &quot;w&quot;) as f:  # Open in write mode to overwrite<br>    json.dump(data, f, indent=4)  # Write the formatted data to the file<br><br>print(&quot;Data successfully written to JSON file.&quot;)</pre><p>The code extracts text chunks of varying lengths: 1024, 512, and 256 characters. If the text is too long, the model tends to generate repetitive questions. If the text is too short, the questions often don’t make sense. This is why I experimented with different lengths. At first glance, about 20% of the generated question/input/answer triplets are complete gibberish. Of the remaining ones, only a fraction are technically sound triplets. This means that the produced question/answer pairs would require some refinement, which can be done manually or with the help of a model. In this experiment, no further refinement of the triplets was performed.</p><p>Now, the data need to be prepared for model fine-tuning. The code below opens the data JSON file, shuffles the triplets, and assigns 1000 samples to the training dataset and 200 samples to the evaluation dataset. Additionally, the code empties the “input” field. Initially, I believed that including text chunks in the input would aid the training process. However, my initial attempts resulted in a model that seemed untrained. I supposed that providing too much information caused the model to derive the answer from the context rather than its memory. Consequently, I modified the code to remove the text chunks from the input.</p><pre>import json<br>import random<br>from datasets import Dataset<br><br>with open(&quot;/content/drive/MyDrive/data.json&quot;, &quot;r&quot;) as f:<br>   dataset = json.load(f)<br><br>if isinstance(dataset, list):<br><br>    random.shuffle(dataset)<br><br>    train_dataset_list = dataset[:1000]<br>    eval_dataset_list = dataset[1000:1200]<br><br>    for item in train_dataset_list:<br>        item[&quot;input&quot;] = &quot;&quot;<br><br>    for item in eval_dataset_list:<br>        item[&quot;input&quot;] = &quot;&quot;<br><br>    with open(&quot;/content/drive/MyDrive/train_data.json&quot;, &quot;w&quot;) as f:<br>        json.dump(train_dataset_list, f)<br><br>    with open(&quot;/content/drive/MyDrive/eval_data.json&quot;, &quot;w&quot;) as f:<br>        json.dump(eval_dataset_list, f)<br><br>else:<br>    print(&quot;Data is not a list, cannot shuffle.&quot;)</pre><p>Before tokenizing the samples, they should be arranged in the format suggested by the selected model card. For the Mistral 7B Instruct model, the instructions provided to the model should be surrounded by the INST directive. Additionally, the model assumes a structure where the “user” makes a request and the “assistant” provides a response. The tokenizer includes a chat template function that helps implement this scheme.</p><pre>import json<br>from datasets import Dataset<br><br>def format_prompt_user(sample):<br>    return f&quot;&quot;&quot;<br>Below is an instruction that describes a task or a question, paired with an optional input that provides further context. Write a response that appropriately completes the request.<br><br>### Instruction:<br>{sample[&quot;instruction&quot;]}<br><br>### Input:<br>{sample[&quot;input&quot;]}<br>&quot;&quot;&quot;<br><br>def format_prompt_assistant(sample):<br>    return f&quot;&quot;&quot;<br>### Response:<br>{sample[&quot;output&quot;]}<br>&quot;&quot;&quot;<br><br>def format_prompt_chat(sample):<br>    chat = [<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: format_prompt_user(sample)},<br>        {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: format_prompt_assistant(sample)}<br>    ]<br>    text = tokenizer.apply_chat_template(chat, tokenize=False)<br>    return {&quot;formatted_text&quot;: text}<br><br>def generate_and_tokenize_prompt(sample):<br>    formatted_text = sample[&quot;formatted_text&quot;]<br>    result = tokenizer(<br>        formatted_text,<br>        truncation=True,<br>        max_length=max_length,<br>        padding=&quot;max_length&quot;,<br>    )<br>    result[&quot;labels&quot;] = result[&quot;input_ids&quot;].copy()<br>    return result<br><br>max_length = 2048  # differs from datasets to datasets<br><br>with open(&quot;/content/drive/MyDrive/train_data.json&quot;, &quot;r&quot;) as f:<br>    train_dataset_list = json.load(f)<br><br>with open(&quot;/content/drive/MyDrive/eval_data.json&quot;, &quot;r&quot;) as f:<br>    eval_dataset_list = json.load(f)<br><br>train_dataset = Dataset.from_list(train_dataset_list)<br>eval_dataset = Dataset.from_list(eval_dataset_list)<br><br>formatted_train_dataset = train_dataset.map(lambda x: format_prompt_chat(x))<br>formatted_eval_dataset = eval_dataset.map(lambda x: format_prompt_chat(x))<br><br>tokenized_train_dataset = formatted_train_dataset.map(generate_and_tokenize_prompt)<br>tokenized_eval_dataset = formatted_eval_dataset.map(generate_and_tokenize_prompt)<br><br>print(f&#39;Training data size: {len(tokenized_train_dataset)}&#39;)<br>print(f&#39;Validation data size: {len(tokenized_eval_dataset)}&#39;)</pre><p>Now that the data is prepared, it’s time to fine-tune the model. Note that the number of samples is small (compared to, for example, the Alpaca dataset, which has about 50,000 samples). Therefore, I’m experimenting with a lora_alpha value of 64. There is extensive literature on the optimal selection of r and lora_alpha, so that won’t be discussed here.</p><p>If you are fine-tuning on the free Colab T4 GPU, keep the following parameters to avoid memory errors. Given the maximum epochs, the fine-tuning process iterates over the entire set 4 times. This process will take approximately 24 hours. Make sure to save the checkpoints and resume from them if you get disconnected from Colab (which happens frequently for the free users).</p><pre>per_device_train_batch_size=2,<br>gradient_accumulation_steps=4,</pre><p>For this experiment, I purchased some Colab credits to speed things up by selecting the A100 GPU. The process took approximately 3 hours. The following code implements the fine-tuning. With the specified batch size, accumulation steps, and maximum epochs, the fine-tuning process iterates over the entire set of samples 8 times.</p><pre>from peft import LoraConfig, PeftModel, prepare_model_for_kbit_training, get_peft_model<br>from datetime import datetime<br><br>def print_trainable_parameters(model):<br>    &quot;&quot;&quot;<br>    Prints the number of trainable parameters in the model.<br>    &quot;&quot;&quot;<br>    trainable_params = 0<br>    all_param = 0<br>    for _, param in model.named_parameters():<br>        all_param += param.numel()<br>        if param.requires_grad:<br>            trainable_params += param.numel()<br>    print(<br>        f&quot;trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}&quot;<br>    )<br><br>project = &quot;ECSS-E-ST-50-51C-finetune-test-#3&quot;<br>base_model_name = &quot;Mistral-7B-Instruct&quot;<br>run_name = base_model_name + &quot;-&quot; + project<br>output_dir = &quot;/content/drive/MyDrive/&quot; + run_name<br><br># Preapre model<br>model.config.pretraining_tp = 1<br>model.gradient_checkpointing_enable()<br>model = prepare_model_for_kbit_training(model)<br>peft_config = LoraConfig(<br>        r=8,<br>        lora_alpha=64,<br>        lora_dropout=0.05,<br>        bias=&quot;none&quot;,<br>        task_type=&quot;CAUSAL_LM&quot;,<br>        target_modules=[&quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;, &quot;gate_proj&quot;, &quot;up_proj&quot;, &quot;down_proj&quot;, &quot;lm_head&quot;]<br>    )<br>model = get_peft_model(model, peft_config)<br>print_trainable_parameters(model)<br><br>trainer = transformers.Trainer(<br>    model=model,<br>    train_dataset=tokenized_train_dataset,<br>    eval_dataset=tokenized_eval_dataset,<br>    args=transformers.TrainingArguments(<br>        output_dir=output_dir,<br>        warmup_steps=5,<br>        per_device_train_batch_size=8,<br>        gradient_accumulation_steps=2,<br>        gradient_checkpointing=True,<br>        max_steps=500,<br>        learning_rate=2.5e-5, # Want about 10x smaller than the Mistral learning rate<br>        logging_steps=50,<br>        bf16=True,<br>        optim=&quot;paged_adamw_8bit&quot;,<br>        logging_dir=&quot;./logs&quot;,        # Directory for storing logs<br>        save_strategy=&quot;steps&quot;,       # Save the model checkpoint every logging step<br>        save_steps=50,                # Save checkpoints every 50 steps<br>        evaluation_strategy=&quot;steps&quot;, # Evaluate the model every logging step<br>        eval_steps=50,               # Evaluate and save checkpoints every 50 steps<br>        do_eval=True,                # Perform evaluation at the end of training<br>        #report_to=&quot;wandb&quot;,           # Comment this out if you don&#39;t want to use weights &amp; baises<br>        run_name=f&quot;{run_name}-{datetime.now().strftime(&#39;%Y-%m-%d-%H-%M&#39;)}&quot;          # Name of the W&amp;B run (optional)<br>    ),<br>    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False),<br>)<br>model.config.use_cache = False  # silence the warnings. Please re-enable for inference!<br><br>if torch.cuda.device_count() &gt; 1: # If more than 1 GPU<br>    model.is_parallelizable = True<br>    model.model_parallel = True<br><br>#trainer.train(resume_from_checkpoint = True)<br>trainer.train()</pre><p>The model training began to show signs of over fitting around step 300, but it was left completing the training. Once the training is finished, the code below enables loading the fine-tuned model and interacting with it via a chat interface.</p><pre>import torch<br>import transformers<br>import re, random<br>from huggingface_hub import login<br>from transformers import AutoTokenizer, AutoModelForCausalLM<br>from peft import LoraConfig, PeftModel, PeftConfig, prepare_model_for_kbit_training, get_peft_model<br><br>model_id = &quot;mistralai/Mistral-7B-Instruct-v0.1&quot;<br><br>project = &quot;ECSS-E-ST-50-51C-finetune-test-#3&quot;<br>base_model_name = &quot;Mistral-7B-Instruct&quot;<br>run_name = base_model_name + &quot;-&quot; + project<br>output_dir = &quot;/content/drive/MyDrive/&quot; + run_name<br>check_point = &quot;checkpoint-500&quot;<br>adapter_model_id = output_dir + &quot;/&quot; + check_point<br><br>bnb_config = transformers.BitsAndBytesConfig(<br>load_in_4bit=True,<br>bnb_4bit_use_double_quant=True,<br>bnb_4bit_quant_type=&quot;nf4&quot;,<br>bnb_4bit_compute_dtype=torch.bfloat16<br>)<br><br>config = PeftConfig.from_pretrained(adapter_model_id)<br><br>login(token=&quot;insert huggingface access token here&quot;)<br><br>model = transformers.AutoModelForCausalLM.from_pretrained(<br>model_id,<br>trust_remote_code=True,<br>quantization_config=bnb_config,<br>device_map=&#39;auto&#39;,<br>)<br><br>ft_model = PeftModel.from_pretrained(<br>model,<br>adapter_model_id,<br>)<br><br>tokenizer = transformers.AutoTokenizer.from_pretrained(<br>model_id,<br>padding_side=&quot;left&quot;,<br>add_eos_token=True,<br>add_bos_token=True<br>)<br>tokenizer.pad_token = tokenizer.eos_token</pre><pre>def generate_response(prompt):<br>  chat = [<br>      {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt},<br>  ]<br>  text = tokenizer.apply_chat_template(chat, tokenize=False)<br>  encodeds = tokenizer(text, return_tensors=&quot;pt&quot;, add_special_tokens=False).to(&#39;cuda&#39;)<br>  generated_ids = ft_model.generate(**encodeds, pad_token_id=tokenizer.eos_token_id, max_new_tokens=1024, do_sample=True)<br>  decoded = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]<br><br>  # Remove prompt and special characters<br>  cleaned_response = decoded.split(prompt, 1)[1]  # Split by prompt and keep the second part<br>  cleaned_response = tokenizer.clean_up_tokenization(cleaned_response)  # Remove special tokens<br>  cleaned_response = re.sub(r&quot;\ \[/INST\] &quot;, &quot;&quot;, cleaned_response)  # Match and replace [/INST] followed by a space<br><br>  return cleaned_response<br><br># Chat loop<br>while True:<br>  user_input = input(&quot;You: &quot;)<br>  if user_input.lower() == &quot;quit&quot;:<br>    break<br>  response = generate_response(user_input)<br>  print(f&quot;Computer: {response}&quot;)</pre><p>The fine-tuned model was tested on 20 questions that were not included in the dataset; these questions were manually drafted from the PDF content. The model’s responses were rated on a scale from 1 to 10, where 1 indicates a completely nonsensical response and 10 represents a perfectly matched and valid answer. Both the pre-trained and fine-tuned models underwent testing on the identical set of questions.</p><p>On average, the pre-trained model scored 4.7, suggesting it has some prior knowledge, particularly about ECSS in general and the Space Wire interface. In contrast, the fine-tuned model averaged a score of 6.8. While this improvement is evident, I believe it may not be sufficient for a production-scale model. However, there is definitely potential for enhancement by focusing on data quality and tuning parameters.</p><p><strong>In conclusion:</strong></p><ul><li>A small language model was <strong>fine-tuned for Q&amp;A using data extracted from a technical PDF document</strong>, to explore the potential of fine tuning.</li><li>The <strong>question/answer</strong> pairs were <strong>generated by the model itself</strong>.</li><li>The fine-tuning process led to <strong>improvements in the model’s responses during Q&amp;A tests, </strong>showing promising results. However, it’s evident that the q<strong>uality of the data and the tuning parameters</strong> are crucial factors that <strong>should be further investigated to verify the limit performance</strong>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91b7fc458ced" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Zero to Hero: Understanding Transformers via GPT-2 Code]]></title>
            <link>https://medium.com/@ales.avanzi/understanding-transformers-with-a-look-at-the-code-of-gpt-2-47f8a73809d7?source=rss-89f1d0a91a20------2</link>
            <guid isPermaLink="false">https://medium.com/p/47f8a73809d7</guid>
            <category><![CDATA[artifical-intellegence]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[gpt]]></category>
            <dc:creator><![CDATA[Alessandro Avanzi]]></dc:creator>
            <pubDate>Fri, 14 Jun 2024 19:35:49 GMT</pubDate>
            <atom:updated>2024-07-06T09:46:24.041Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*Ve_Xp3dU71lYqj8xxTmsdA.png" /></figure><p>I started approaching language models with little background in artificial intelligence. I wanted to understand the principles of the technology, which I believed was going to drastically change the way most people work (including myself). It was the end of 2023. I am an aerospace engineer and I work on satellite avionics and on-board computers. I first read an introductory book on artificial intelligence, I believe it was <em>Deep Learning with Python, </em>which had a section on word embedding and language models, but was addressing the problem with the classic (at the time of the book) recurrent approach. I then looked at more recent publications, the foundational <em>Attention is all you need </em>from 2017, and then <em>Language Models are Unsupervised Multitask Learners </em>from 2018. The latter was published by OpenAI with the open source language model GPT-2. As per model card: <em>GPT-2 is a transformers model pretrained on a very large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it was trained to guess the next word in sentences.</em></p><p>In my early career I was firmware programmer, writing code for micro-controllers. I believe seeing some code in action it’s way more effective to understand how general principles are applied. So with minimal background in Python (but assuming familiarity with at least one programming language was sufficient), I cloned the GPT-2 repository and started digging into the code. Here’s what I found out.</p><p>GPT-2 is provided with an example script that generates a sequence of words from the model. The script <em>generate_unconditional_sample.py, </em>uses the term <em>unconditional</em> in the filename<em>, </em>because the output is not conditioned on any input text from the user (there is no prompt), it is simply extracted from the model. In other terms, the <em>context</em> is empty (this is not 100% true). In a language model the <em>context </em>is<em> </em>a buffer filled with text, that the model uses to guess the next word. In simple terms language models are next word predictors, they predict the next word, given the context. As long as a new word is generated, it is inserted into the context, and used to predict the next word (for the moment this gives the idea). The context has a defined length, so up to a certain amount of words can fit. Those that fall outside are discarded and not used to predict what comes next.</p><pre>    models_dir = os.path.expanduser(os.path.expandvars(models_dir))<br>    enc = encoder.get_encoder(model_name, models_dir)<br>    hparams = model.default_hparams()<br>    with open(os.path.join(models_dir, model_name, &#39;hparams.json&#39;)) as f:<br>        hparams.override_from_dict(json.load(f))<br><br>    if length is None:<br>        length = hparams.n_ctx<br>    elif length &gt; hparams.n_ctx:<br>        raise ValueError(&quot;Can&#39;t get samples longer than window size: %s&quot; % hparams.n_ctx)<br><br>    with tf.Session(graph=tf.Graph()) as sess:<br>        np.random.seed(seed)<br>        tf.set_random_seed(seed)<br><br>        output = sample.sample_sequence(<br>            hparams=hparams, length=length,<br>            start_token=enc.encoder[&#39;&lt;|endoftext|&gt;&#39;],<br>            batch_size=batch_size,<br>            temperature=temperature, top_k=top_k, top_p=top_p<br>        )[:, 1:]<br><br>        saver = tf.train.Saver()<br>        ckpt = tf.train.latest_checkpoint(os.path.join(models_dir, model_name))<br>        saver.restore(sess, ckpt)<br><br>        generated = 0<br>        while nsamples == 0 or generated &lt; nsamples:<br>            out = sess.run(output)<br>            for i in range(batch_size):<br>                generated += batch_size<br>                text = enc.decode(out[i])<br>                print(&quot;=&quot; * 40 + &quot; SAMPLE &quot; + str(generated) + &quot; &quot; + &quot;=&quot; * 40)<br>                print(text)</pre><p>GPT-2 features and parameters are collected in the structure <em>hparams. n_ctx </em>is telling that GPT-2 context length is 1024. Recent language models are equipped with contexts that spans from many thousands to millions of <em>tokens.</em></p><pre>hparams = {HParams} HParams([(&#39;n_ctx&#39;, 1024), (&#39;n_embd&#39;, 768), (&#39;n_head&#39;, 12), (&#39;n_layer&#39;, 12), (&#39;n_vocab&#39;, 50257)])<br> n_ctx = {int} 1024<br> n_embd = {int} 768<br> n_head = {int} 12<br> n_layer = {int} 12<br> n_vocab = {int} 50257</pre><p>It is known that language models do not work on words, as they do not work on alphabet symbols neither. They process something in between, <em>tokens. </em>Tokens are pieces of words, including punctuation and special characters (up to entire words in some cases) which are determined from a given text based on how many times they appear. The entire collections of possible tokens creates the model <em>vocabulary. </em>So when the context length is defined, is it expressed as number of tokens. A richer vocabulary implies that more text can be squeezed into the context, given a fixed length.</p><pre>50207: &#39;Ġkernels&#39;, 50208: &#39;ĠFranÃ§ois&#39;, 50209: &#39;ĠDuff&#39;, 50210: &#39;ĠPon&#39;, 50211: &#39;ĠLeica&#39;, 50212: &#39;ĠGarmin&#39;, 50213: &#39;Ġorphans&#39;, 50214: &#39;ĠClaudia&#39;, 50215: &#39;Ġcalendars&#39;, 50216: &#39;ĠLeilan&#39;, 50217: &#39;ento&#39;, 50218: &#39;Rocket&#39;, 50219: &#39;Ġbrunch&#39;, 50220: &#39;ĠHawking&#39;, 50221: &#39;ainers&#39;, 50222: &#39;Ġsensibilities&#39;, 50223: &#39;ĠkW&#39;, 50224: &#39;ĠKand&#39;, 50225: &#39;Ġreclaimed&#39;, 50226: &#39;Ġinterestingly&#39;, 50227: &#39;×©&#39;, 50228: &#39;romy&#39;, 50229: &#39;JM&#39;, 50230: &#39;ĠEnhancement&#39;, 50231: &#39;bush&#39;, 50232: &#39;Skip&#39;, 50233: &#39;Ġrappers&#39;, 50234: &#39;Ġgazing&#39;, 50235: &#39;pedia&#39;, 50236: &#39;athlon&#39;, 50237: &#39;Revolution&#39;, 50238: &#39;Ġsnipers&#39;, 50239: &#39;Ġreverted&#39;, 50240: &#39;Ġconglomerate&#39;, 50241: &#39;Terry&#39;, 50242: &#39;794&#39;, 50243: &#39;Ġharsher&#39;, 50244: &#39;Ġdesolate&#39;, 50245: &#39;ĠHitman&#39;, 50246: &#39;Commission&#39;, 50247: &#39;Ġ(/&#39;, 50248: &#39;âĢ¦.&quot;&#39;, 50249: &#39;Compar&#39;, 50250: &#39;Ġamplification&#39;, 50251: &#39;ominated&#39;, 50252: &#39;Ġregress&#39;, 50253: &#39;ĠCollider&#39;, 50254: &#39;Ġinformants&#39;, 50255: &#39;Ġgazed&#39;, 50256: &#39;&lt;|endoftext|&gt;&#39;}</pre><p>Here’s a snap of the last 50 tokens in the GPT-2 vocabulary, which as per <em>hparams </em>definition, is composed of 50257 tokens (if you are wondering, the Ġ character is equivalent to leading space \u0120). The very last is a special character, <em>endoftext </em>(or <em>EOT</em>) which is used by the model to understand when a sequence is completed. Each token is associated (and uniquely identified) with a number from 0 to 50256, where 50256 is <em>EOT</em>. So the model is not processing text, it is processing numbers, and the starting point are the indexes associated with each token. In other words, a piece of text shall be first <em>encoded </em>in a sequence of numbers in the range from 0 to 50256, then processed by the model, and the result <em>decoded </em>back from numbers to text. This is way the model is equipped with an <em>encoder (</em>or <em>tokenizer), </em>which purpose is to convert text to numbers for being processed, and then convert the result back to text. Multi modal models are capable of processing a variety of inputs, including text, images, and audio, but the basic concept is no different, there will be an equivalent of the <em>tokenizer</em> capable to translate the input media into a sequence of numbers, and back.</p><p>So here is the code that generates the sample sequence:</p><pre>output = sample.sample_sequence(<br>            hparams=hparams, length=length,<br>            start_token=enc.encoder[&#39;&lt;|endoftext|&gt;&#39;],<br>            batch_size=batch_size,<br>            temperature=temperature, top_k=top_k, top_p=top_p<br>        )[:, 1:]</pre><p>The context is not completely empty, it is initialized with a <em>EOT</em> token, as a <em>start_token. </em>This is an initialization technique, it might sound strange that the first token is an <em>end of text</em>, but apparently it is a common way to initialize the generation of a new sample sequence in language models. Then <em>length </em>parameter is the length of the sample that it is going to be generated, be default is 1024, equal to the length of the context. <em>Batch_size</em> is the number of sequences that the model can process simultaneously. In this script, batch size is one by default, but we could have asked the model to generate n sequences at the same time, each starting from token <em>EOT</em>, by modifying batch size. It clearly has an effect on <em>inference</em> time.</p><pre>&quot;&quot;&quot;<br>:temperature=1 : Float value controlling randomness in boltzmann<br>     distribution. Lower temperature results in less random completions. As the<br>     temperature approaches zero, the model will become deterministic and<br>     repetitive. Higher temperature results in more random completions.<br>:top_k=0 : Integer value controlling diversity. 1 means only 1 word is<br>     considered for each step (token), resulting in deterministic completions,<br>     while 40 means 40 words are considered at each step. 0 (default) is a<br>     special setting meaning no restrictions. 40 generally is a good value.<br>&quot;&quot;&quot;</pre><p><em>Temperature</em>, <em>top_k</em> and <em>top_p</em> are used to control randomness of the output; by modifying these numbers we can force the model to be deterministic, generating always the same token given the context, or to provide an output with a certain degree of randomness.</p><p>The first token is removed from the output of <em>sample.sample_sequence, </em>by applying <em>[:, 1:], </em>where the first index selects all the batches (by default we have only one) and the second index removes the first token, which was initialized as <em>EOT</em>.</p><pre>def sample_sequence(*, hparams, length, start_token=None, batch_size=None, context=None, temperature=1, top_k=0, top_p=1):<br>    if start_token is None:<br>        assert context is not None, &#39;Specify exactly one of start_token and context!&#39;<br>    else:<br>        assert context is None, &#39;Specify exactly one of start_token and context!&#39;<br>        context = tf.fill([batch_size, 1], start_token)<br><br>    def step(hparams, tokens, past=None):<br>        lm_output = model.model(hparams=hparams, X=tokens, past=past, reuse=tf.AUTO_REUSE)<br><br>        logits = lm_output[&#39;logits&#39;][:, :, :hparams.n_vocab]<br>        presents = lm_output[&#39;present&#39;]<br>        presents.set_shape(model.past_shape(hparams=hparams, batch_size=batch_size))<br>        return {<br>            &#39;logits&#39;: logits,<br>            &#39;presents&#39;: presents,<br>        }<br><br>    with tf.name_scope(&#39;sample_sequence&#39;):<br>        def body(past, prev, output):<br>            next_outputs = step(hparams, prev, past=past)<br>            logits = next_outputs[&#39;logits&#39;][:, -1, :]  / tf.to_float(temperature)<br>            logits = top_k_logits(logits, k=top_k)<br>            logits = top_p_logits(logits, p=top_p)<br>            samples = tf.multinomial(logits, num_samples=1, output_dtype=tf.int32)<br>            return [<br>                next_outputs[&#39;presents&#39;] if past is None else tf.concat([past, next_outputs[&#39;presents&#39;]], axis=-2),<br>                samples,<br>                tf.concat([output, samples], axis=1)<br>            ]<br><br>        past, prev, output = body(None, context, context)<br><br>        def cond(*args):<br>            return True<br><br>        _, _, tokens = tf.while_loop(<br>            cond=cond, body=body,<br>            maximum_iterations=length - 1,<br>            loop_vars=[<br>                past,<br>                prev,<br>                output<br>            ],<br>            shape_invariants=[<br>                tf.TensorShape(model.past_shape(hparams=hparams, batch_size=batch_size)),<br>                tf.TensorShape([batch_size, None]),<br>                tf.TensorShape([batch_size, None]),<br>            ],<br>            back_prop=False,<br>        )</pre><p>GPT-2 is based on TensorFlow, one of the open-source software libraries for machine learning. TensorFlow understanding is outside the scope here, but the overall architecture of the model can be understood without a detailed knowledge of the library.</p><p>In the first part of the <em>sample_sequence </em>call, the context is filled with the start token, the <em>EOT (</em>consider that <em>batch_size </em>is 1 by default<em>). </em>With <em>tf.while_loop </em>in the last part of the code, TensorFlow is instructed to iterate over the <em>body </em>function for a number of iterations which are required to generate a new sequence of tokens. The new sequence shall be of 1024 tokens (which is also the context length), but since the first token <em>EOT</em> is already given, Tensorflow will iterate the <em>body</em> 1023 times.</p><p>At the first iteration <em>past </em>is none, <em>prev </em>is the current context filled with only <em>EOT</em> token, and output will be the new context with one additional token added to the sequence. Note that in case the user is passing the model a <em>prompt </em>defined composed of multiple tokens, <em>prev </em>is holding the entire prompt. After having processed the prompt (which in this example is filled with <em>EOT</em> only) <em>prev </em>is holding the last generated token only (in a 1x1 tensor) while <em>output </em>is holding the entire generated sequence.</p><p>At each step, the body function is calling the model, with the following inputs: <em>hparams </em>is identifying the model, the batch size, the context length. <em>X </em>is the token generated at the previous step (or the entire <em>prompt</em>, the first time the model is run with a user given prompt), therefore the most recent token generated by the model, <em>past </em>is an intermediate product that store the computation done on all the past tokens before <em>prev (</em>we’ll see in a bit how <em>past </em>is composed<em>). </em>When adding a new token to the context, the entire context is not re-processed from scratch, but only the last generated token is processed within the model, to add a new segment of information useful the generated the next token.</p><pre>logits = next_outputs[&#39;logits&#39;][:, -1, :]  / tf.to_float(temperature)<br>logits = top_k_logits(logits, k=top_k)<br>logits = top_p_logits(logits, p=top_p)<br>samples = tf.multinomial(logits, num_samples=1, output_dtype=tf.int32)</pre><p>In order for the model to generate a new token, it generates a vector of length 1x50257, each item is representing the likelihood that the token at that position (from 0 to 50256) will be the next token. By manipulating this distribution, it is possible to extract the next sample with a certain degree of variability.</p><pre>def model(hparams, X, past=None, scope=&#39;model&#39;, reuse=False):<br>    with tf.variable_scope(scope, reuse=reuse):<br>        results = {}<br>        batch, sequence = shape_list(X)<br><br>        wpe = tf.get_variable(&#39;wpe&#39;, [hparams.n_ctx, hparams.n_embd],<br>                             initializer=tf.random_normal_initializer(stddev=0.01))<br>        wte = tf.get_variable(&#39;wte&#39;, [hparams.n_vocab, hparams.n_embd],<br>                             initializer=tf.random_normal_initializer(stddev=0.02))<br>        past_length = 0 if past is None else tf.shape(past)[-2]<br>        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))<br><br>        # Transformer<br>        presents = []<br>        pasts = tf.unstack(past, axis=1) if past is not None else [None] * hparams.n_layer<br>        assert len(pasts) == hparams.n_layer<br>        for layer, past in enumerate(pasts):<br>            h, present = block(h, &#39;h%d&#39; % layer, past=past, hparams=hparams)<br>            presents.append(present)<br>        results[&#39;present&#39;] = tf.stack(presents, axis=1)<br>        h = norm(h, &#39;ln_f&#39;)<br><br>        # Language model loss.  Do tokens &lt;n predict token n?<br>        h_flat = tf.reshape(h, [batch*sequence, hparams.n_embd])<br>        logits = tf.matmul(h_flat, wte, transpose_b=True)<br>        logits = tf.reshape(logits, [batch, sequence, hparams.n_vocab])<br>        results[&#39;logits&#39;] = logits<br>        return results</pre><p>This is the model implementation, where <em>hparams </em>is identifying the model features, the batch size, the context length. <em>X </em>is the token generated at the previous step, therefore the most recent token generated by the model, <em>past </em>is an intermediate product that store the computation done on all the past tokens before <em>X.</em></p><p>The first layer of the Transformer implements two architectural concepts: t<em>oken embedding</em> and <em>positional encoding</em>.</p><pre>h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))</pre><p>The previous token (which at first iteration is <em>EOT</em>) can be viewed as a vector 1x50257 of all zeros except one item equal to 1, the item corresponding to the token. The problem with this vector is that it is high dimensional, and sparse (only one element is different from zero). There is a more clever way to feed the token in the Transformer. Token embedding are still vectors, but are smaller, dense and composed of floating points elements. GPT-2 is using token embedding of length 1x768. The matrix <em>wte (wte </em>presumably stands for <em>weights token embedding) </em>is translating the input token into the corresponding embedding.<em> </em>Note that:</p><p><em>Shape </em>of <em>wte </em>is (50357, 768) and</p><p><em>tf.gather(wte, X)</em> extracts the <em>X</em> raw of the <em>wte</em> matrix, therefore it extracts the embedding corresponding to token <em>X</em>.</p><p>The shape of <em>h</em> is (1, 1, 768). Even though <em>X</em> has a shape of (1, 1), it’s interpreted as a single index with an extra dimension added at the end. <em>h</em> tensor holds the retrieved embedding vector. Since <em>tf.gather</em> adds an extra dimension for the index, the final shape of <em>h</em> becomes (<em>batch_size</em>, 1, <em>n_embd</em>), which in our case is (1, 1, 768). This might seem counter intuitive at first, but it’s a common way to handle single-element indices in TensorFlow operations.</p><p>The parameters of <em>wte</em> are learned during training, as the <em>get_variable </em>method informs TensorFlow that these are trainable parameters.</p><p>On word embedding: the easiest way to associated a dense vector to a token is to chose it randomly. While technically choosing at random can be considered a form of embedding, it’s far from being the most effective method. Choosing at random would create an embedding space with no internal structure. Modern word embedding techniques go beyond randomness to create a structured embedding space. This space captures semantic relationships between words, allowing models to understand the context and meaning within a sentence. Some key aspects of structured embedding spaces:</p><p>Similar words have similar vectors: Words with similar meanings will be positioned closer together in the embedding space. This allows models to identify synonyms and understand the overall sentiment of a sentence.</p><p>Relationships are reflected geometrically: Operations like vector addition or subtraction can represent meaningful relationships between words. For example, “king” — “man” + “woman” might result in a vector close to “queen”.</p><p>Reduced dimensionality: While word meanings are complex, effective word embedding techniques can represent words using a smaller number of dimensions. This allows for efficient processing and reduces computational complexity.</p><pre>h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))</pre><p>The second part of this line of code implements positional encoding. The concept is that a token shall be encoded in such a way that the encoding is carrying some information not only about the token itself, but also about its position in the sequence.</p><p>In this call:</p><p><em>X</em> is the token generated at the previous step</p><p><em>past_length</em> is the length of the sequences processed so far. The first time the model is processing <em>EOT</em> token, there is not past, therefore past_length is 0.</p><p>Positional encoding for each position in the sequence is given by tensor <em>wpe </em>(presumably stands for <em>weights positional encoding</em>), which has dimensions:</p><p><em>Shape</em> of <em>wpe</em> is (1024, 7689)</p><p>where the first index is the dimension of the context, the maximum amount of token that the Transformer considers during the prediction of the next token.</p><p><em>tf.gather(wpe, positions_for(X, past_length))</em> extract the row of the <em>wpe</em> matrix, corresponding to the position of <em>X</em> in the sequence, being the position a number between 0 and 1023.</p><p>As an example, consider we are generating a sentence, and the current token (<em>X</em>) is “the”. Therefore:</p><p>Current Token (<em>X</em>): “the”</p><p>Past Length: 2 (assuming we already processed two tokens before “the”)</p><p>Positions_for(X, past_length):</p><p>past_length = 2</p><p>This function return position 2. This number represent the relative positions of token in the sequence. “the” is the third token, hence position 2. We get the word embedding for “the” (a vector representing its meaning). This positional sequence is transformed into another vector representing the relative positions.</p><p>To summarize then:</p><p><em>h</em> is encoding the token generated at the previous step: its encoding is extracted from the embedding matrix <em>wte</em> (<em>weights token embeddings</em>) and summed the positional encoding <em>wpe</em> (<em>weights positional encoding</em>). Both <em>wpe</em> and <em>wte</em> are determined during training. This way we are obtaining a representation of the previous token which:</p><ul><li>Is defined in a structured embedding space, and it is represented by a dense floating point vector.</li><li>It is carrying information about the position of the token in the sequence generated so far.</li></ul><pre># Transformer<br>presents = []<br>pasts = tf.unstack(past, axis=1) if past is not None else [None] * hparams.n_layer<br>assert len(pasts) == hparams.n_layer<br>for layer, past in enumerate(pasts):<br>       h, present = block(h, &#39;h%d&#39; % layer, past=past, hparams=hparams)<br>       presents.append(present)<br>results[&#39;present&#39;] = tf.stack(presents, axis=1)<br>h = norm(h, &#39;ln_f&#39;)</pre><p>This code implements the encoder only Transformer. The first time the code is executed, the context is initialized with the <em>EOT</em> token, and <em>past</em> is <em>null</em>. Therefore <em>past</em> is initialized as an empty structure multiplied by the Transformer number of layers. GPT-2 is implementing 12 layers (as reported in the structure <em>haprams</em>). Each layer is processing the information of the previous one, as the code iterates over the layers in a cycle. Each layer is providing as output the next token prediction (which is encoded in <em>h </em>as an embedding vector), which is fed to the next layer up to total layer count. All layers have identical structure, but each is having its own <em>past </em>which is used to compute the next token (we’ll see in a bit what are <em>past </em>and <em>present</em>). The last layer (the twelfth) is providing the token prediction (in form of an embedding) which is first normalized to <em>mean 0</em> and <em>standard deviation 1</em> before being passed to the next code section.</p><pre>def block(x, scope, *, past, hparams):<br>    with tf.variable_scope(scope):<br>        nx = x.shape[-1].value<br>        a, present = attn(norm(x, &#39;ln_1&#39;), &#39;attn&#39;, nx, past=past, hparams=hparams)<br>        x = x + a<br>        m = mlp(norm(x, &#39;ln_2&#39;), &#39;mlp&#39;, nx*4, hparams=hparams)<br>        x = x + m<br>        return x, present</pre><p>The <em>block</em> is representing a single layer of the Transformer, and it is implementing two basic building blocks of the encoder-only architecture:</p><ul><li>Masked Self Attention layer (the <em>attn </em>call)</li><li>Feed-forward fully connected layer (the <em>mlp </em>call)</li></ul><p>Let’s have a look at how the attention mechanism is implemented in GPT-2.</p><pre>def attn(x, scope, n_state, *, past, hparams):<br>    assert x.shape.ndims == 3  # Should be [batch, sequence, features]<br>    assert n_state % hparams.n_head == 0<br>    if past is not None:<br>        assert past.shape.ndims == 5  # Should be [batch, 2, heads, sequence, features], where 2 is [k, v]<br><br>    def split_heads(x):<br>        # From [batch, sequence, features] to [batch, heads, sequence, features]<br>        return tf.transpose(split_states(x, hparams.n_head), [0, 2, 1, 3])<br><br>    def merge_heads(x):<br>        # Reverse of split_heads<br>        return merge_states(tf.transpose(x, [0, 2, 1, 3]))<br><br>    def mask_attn_weights(w):<br>        # w has shape [batch, heads, dst_sequence, src_sequence], where information flows from src to dst.<br>        _, _, nd, ns = shape_list(w)<br>        b = attention_mask(nd, ns, dtype=w.dtype)<br>        b = tf.reshape(b, [1, 1, nd, ns])<br>        w = w*b - tf.cast(1e10, w.dtype)*(1-b)<br>        return w<br><br>    def multihead_attn(q, k, v):<br>        # q, k, v have shape [batch, heads, sequence, features]<br>        w = tf.matmul(q, k, transpose_b=True)<br>        w = w * tf.rsqrt(tf.cast(v.shape[-1].value, w.dtype))<br><br>        w = mask_attn_weights(w)<br>        w = softmax(w)<br>        a = tf.matmul(w, v)<br>        return a<br><br>    with tf.variable_scope(scope):<br>        c = conv1d(x, &#39;c_attn&#39;, n_state*3)<br>        q, k, v = map(split_heads, tf.split(c, 3, axis=2))<br>        present = tf.stack([k, v], axis=1)<br>        if past is not None:<br>            pk, pv = tf.unstack(past, axis=1)<br>            k = tf.concat([pk, k], axis=-2)<br>            v = tf.concat([pv, v], axis=-2)<br>        a = multihead_attn(q, k, v)<br>        a = merge_heads(a)<br>        a = conv1d(a, &#39;c_proj&#39;, n_state)<br>        return a, present</pre><p>This code is at the core of the attention mechanism implemented by GPT-2. The inputs to the <em>attn </em>function are:</p><ul><li><em>x</em> is representing the previous token encoded with word embedding and positional encoding; before begin passed to the attention function it is normalized, therefore it is transformed into a tensor of the same dimension but with <em>mean 0 </em>and <em>standard deviation 1</em> (recalling that the dimension of x (1, 1, 768) where (batch, sequence, features)).</li><li><em>‘attn’</em> is the scope, it is used by TensorFlow as some sort of context, to track and retrieve the status of variables already computed during training.</li><li><em>n_state</em> is the length of the embedding space, which is 768 for GPT-2.</li><li><em>past</em> keeps track of all the computation already done on the previous tokens in the context; the transformer does not process all the tokens in the context to predict the next. Only the previous token estimation is processed to guess the next. While doing so however, the processing of the previous tokens in the context is considered by means of <em>past. </em>One notable exception to this is if we provide a user prompt to the model. In this case the next token inferred right after the prompt relies on the processing of the entire prompt to build up a <em>past</em>, but after that only the last token is used. For simplicity, let’s however continue to assume that the first time the model is called, only one <em>EOT</em> token is in the context<em>.</em></li><li><em>hparams</em> is a summary of the Transformer parameters.</li></ul><p>The first call is <em>conv1d, </em>which is provided with three parameters:</p><ul><li><em>x</em> is the embedding of the previous token.</li><li><em>‘c_attn’</em> is the scope (again this is for TensorFlow to retrieve the parameters computed during training).</li><li><em>n_state*3</em> is the number of elements in a token embedding representation (768), multiplies by 3, therefore we have 2304.</li></ul><pre>def conv1d(x, scope, nf, *, w_init_stdev=0.02):<br>    with tf.variable_scope(scope):<br>        *start, nx = shape_list(x)<br>        w = tf.get_variable(&#39;w&#39;, [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev))<br>        b = tf.get_variable(&#39;b&#39;, [nf], initializer=tf.constant_initializer(0))<br>        c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf])<br>        return c</pre><p>Recalling that:</p><ul><li><em>nx</em> is 768</li><li><em>nf</em> is 2304</li></ul><p>the code is first retrieving two matrices, <em>w</em> of dimension (1, 768, 2304) and <em>b</em> of dimensions (2304). The parameters of this two matrices, for each layer separately, are trainable. Then <em>x</em> and <em>w</em> are reshaped as:</p><ul><li><em>x</em> from (1, 1, 768) to (1, 768)</li><li><em>w</em> from (1, 768, 2304) to (768, 2304)</li></ul><p>then they are <em>multiplied</em> and <em>summed</em> to <em>b, or in </em>synthesis</p><p><em>c = x * w + b</em></p><p>Considering the dimensions, the results is a (1, 2304). Note that <em>w</em> and <em>b</em> do not change for each token in the context, they are always the same, but they change from layer to layer. The result is reshaped as (1, 1, 2304), where we get back the batch dimension (in the example we have seen so far is always 1 by default). Note that the function is called <em>conv1d</em> because it is implemented as 1D convolutional layer with 2304 filter size over a temporal sequence of 1 element only (the previous token represented by a 768 dimensional embedding). Since it is not really convoluting over multiple samples of the input sequence (not even if case of an entire prompt processing), the function name appears to be a residual of some past code with a different purposes.</p><p>What the code is doing here, is that it is transforming the original token embedding value, into three new token representations, all lined up in the same vector. They are called</p><ul><li><em>Query</em></li><li><em>Key</em></li><li><em>Value</em></li></ul><p>Since they are new representation of the same token, they are all 768 length, they are lined up in a single tensor, which is therefore of dimension (1, 1, 2304). So what are these new token representations? At the end of the attention function, we want to obtain a new embedding value of the previous token, which adds attention to the embedding and positional encoding: it shall carry information on how much the token is linked or similar to each of the previous tokens in the context. For example, let’s assumed that the context is filled with the following words, and assume to simplify that each word is one token. Assume additionally that we are working right now on the token <strong>it</strong>, and we have to guess what comes next</p><p><strong><em>the cake is on the table and it</em></strong><em> tastes good</em></p><p>So it is understood that the token <em>it </em>is first encoded with word embedding and positional encoding, but now we want a new representation of <em>it</em>, before using it to guess what comes next, and this new representation shall account also for how much the token is linked or similar to the other tokens in the context; specifically, we would assumed that <em>it </em>is strongly linked to <em>cake, </em>therefore the token value of <em>cake </em>shall have a quantitative larger influence on the attention-aware new value of the token <em>it. </em>The <em>Query, Key </em>and<em> Value </em>representation are used for this, to compute a new representation of the previous token, which is carrying information on how it is linked to the context content, and then use it to guess the next one.</p><pre>    with tf.variable_scope(scope):<br>        c = conv1d(x, &#39;c_attn&#39;, n_state*3)<br>        q, k, v = map(split_heads, tf.split(c, 3, axis=2))<br>        present = tf.stack([k, v], axis=1)<br>        if past is not None:<br>            pk, pv = tf.unstack(past, axis=1)<br>            k = tf.concat([pk, k], axis=-2)<br>            v = tf.concat([pv, v], axis=-2)<br>        a = multihead_attn(q, k, v)<br>        a = merge_heads(a)<br>        a = conv1d(a, &#39;c_proj&#39;, n_state)<br>        return a, present</pre><p>Going back to the <em>attn </em>call, right after <em>conv1d </em>a call to <em>map </em>is extracting from <em>c</em> the following:</p><ul><li><em>q </em>which is for Query</li><li><em>k </em>which<em> </em>is for Key</li><li><em>v </em>which is<em> f</em>or Value</li></ul><p>For the moment let us ignore the <em>split_heads</em> function, it is shown later but it does not change to general architecture of the attention algorithm. Let us assume that all the three tensors share the same dimension, which is (1, 1, 768), where we have (batch, sequence, feature).</p><p>The tensor <em>k</em> and <em>v</em> are stacked and stored with <em>tf.stack</em>, and the same tensors are un-stacked and restored from pasts: these are the <em>Key </em>and <em>Value, </em>computed for the previous tokens in the context, which are reused to compute the attention-aware embedding for the current token. Here it is clear what the tensor <em>past</em> was carrying from the previous steps. It is the so called <em>k v cache, </em>the <em>keys </em>and v<em>alues </em>computed for the previous tokens in the context. It is likely that <em>p </em>in <em>pk</em> and <em>pv </em>stands for <em>past</em>.</p><p>Now after stacking and un-staking, if we assume for example that in the context that are 20 tokens, including the current one, then the dimensions of the <em>q</em>, <em>k</em>, and <em>v</em> will be</p><ul><li><em>q -&gt;</em> (1, 1, 768)</li><li><em>k</em> -&gt; (1, 20, 768)</li><li><em>v</em> -&gt; (1, 20, 768)</li></ul><p><em>q</em> is computed only for the current token. Let’s have a look at the <em>multihead_attn </em>call.</p><pre>    def multihead_attn(q, k, v):<br>        # q, k, v have shape [batch, heads, sequence, features]<br>        w = tf.matmul(q, k, transpose_b=True)<br>        w = w * tf.rsqrt(tf.cast(v.shape[-1].value, w.dtype))<br><br>        w = mask_attn_weights(w)<br>        w = softmax(w)<br>        a = tf.matmul(w, v)<br>        return a</pre><p>The first thing that the <em>multihead_attn</em> function does is multiply the <em>query</em> for the <em>keys</em>, where key tensor is transposed. Let us stick to the example of having 20 token in the context, including the current one. If we look at the dimension, we have</p><p>(1, 768) x (20, 768)t = (1, 20)</p><p>So basically the first <em>matmul</em> call does the dot product between the q<em>uery</em> of the current token and <em>key</em> of each token in the context, including itself. The result of the dot product is one scalar value for each <em>query/key</em> pair, in our example 20 scalar values. The resulting tensor <em>w</em> (which stand for w<em>eights</em>) is first normalized for the sqrt of the length of the embedding space.</p><p>The <em>mask_attn_weights </em>is masking (setting to zero) the weights of the future tokens. To understand it let’s assume to have two cases:</p><ul><li>the context is filled with <em>EOT </em>token only (as the example from which it started), there are no future tokens, therefore this call does not do anything on <em>w. </em>Once the next token is predicted after <em>EOT, </em>it is filled in the context and used to guess next one. While processing the new token, still no future tokens are present, therefore this call does not change <em>w</em>.</li><li>the context is filled with a user generated prompt, composed of many token (let’s assume we have 20 token). In this case, to predict the first after the user prompt, the model shall compute <em>keys</em> and <em>values</em> for every token. In this case, assuming the model is processing the prompt token number 10, <em>mask_attn_weights </em>is masking out the weight associated to future tokens, from 11 to 20.</li></ul><p>The <em>softmax</em> function it is used to modify the elements of weights <em>w, </em>so that the order of the element is preserved but they all add up to one. In other word, after <em>softmax</em> all the weight elements adds up to 1.</p><p>Finally, the weights <em>w</em> are multiplied by all the token <em>values</em> in the context. We obtain therefore a new set a <em>values </em>which are weighted by the <em>w </em>we just computed. They are also all summed up to create a new embedding representation for the current token. Looking at the dimensions:</p><p>(1, 20) x (20, 768) = (1, 768)</p><p>On <em>split_heads</em>: the weights and the new token embedding are not computed on the entire embedding length (768) at once; the computation is split into heads, for GPT-2 the number of heads is 12. The embedding length is split and reduced in order to have multiple heads of computation. Specifically the dimension of the <em>query</em> will be</p><p>(1, 12, 1, 64) where</p><p>(batch, head, sequence, feature)</p><p>Note that <em>feature</em> takes only a segment of the original embedding space -&gt; 768 / 12 = 64. These split allows the Transformer to learns more complex relation between tokens in the context, since all the heads are independent, even thou they work on a subset of the embedding space. This implies that each head sees only a portion of the embedding space, and it is allowed to move in a reduced subset of possible directions. I suppose the consequence of this is that each head specializes into identifying specific relations between tokens in the context, given the reduced embedding space that is assigned to it.</p><pre>with tf.variable_scope(scope):<br>        c = conv1d(x, &#39;c_attn&#39;, n_state*3)<br>        q, k, v = map(split_heads, tf.split(c, 3, axis=2))<br>        present = tf.stack([k, v], axis=1)<br>        if past is not None:<br>            pk, pv = tf.unstack(past, axis=1)<br>            k = tf.concat([pk, k], axis=-2)<br>            v = tf.concat([pv, v], axis=-2)<br>        a = multihead_attn(q, k, v)<br>        a = merge_heads(a)<br>        a = conv1d(a, &#39;c_proj&#39;, n_state)<br>        return a, present</pre><p>Going back to the <em>attn </em>call, we recall that the new embedding value that was computed for the current token, is a weighted representation of all the previous tokens (including itself). This way the Transformer encoded in the current token representation, all the contribution of the previous tokens, without having to inject them directly into the computation every time. This a an alternative to the recurrent neural network architecture, where the past token contribution to the prediction is stored in a sort of <em>state</em>, that is used together with the current token.</p><p>Recalling that the embedding was divided in heads, therefore the dimension of <em>a</em> is</p><p>(1, 12, 1, 64) where</p><p>(batch, head, sequence, feature)</p><p>an instruction (<em>merge_heads</em>) merges all the heads, so that we go back to the usual token embedding representation</p><p>(1, 1, 768)</p><p>Before completing the Attention function, the value of <em>a</em> is injected in a in a 1D convolutional layer, which acts on a sequence of 1 sample only, with a number of filters equal to the embedding space. This is in fact a fully connected feed foreword neural network which take as input as single sample of 768 elements and return a single sample of 768 elements, as linear combination. The parameters of this projections are learned during training. Note that this projection is generally used to reduce or adapt the dimensions of the embedding, if the input and output dimensions are different. In this case they are the same, so I guess in principle this step is optional.</p><pre>def block(x, scope, *, past, hparams):<br>    with tf.variable_scope(scope):<br>        nx = x.shape[-1].value<br>        a, present = attn(norm(x, &#39;ln_1&#39;), &#39;attn&#39;, nx, past=past, hparams=hparams)<br>        x = x + a<br>        m = mlp(norm(x, &#39;ln_2&#39;), &#39;mlp&#39;, nx*4, hparams=hparams)<br>        x = x + m<br>        return x, present</pre><p>Going back to the <em>block</em> function, which is a single layer of a 12 layer transformer: the attention function is completed, and we obtained an embedding representation of the current token, which is attention-aware: it is a formed as a weighted combination of the embedding of all the previous token. So now we create a new value for the current token, which is:</p><p><em>x = x + a</em></p><p>This way <em>x </em>is not substituted with <em>a</em>, but summing all the contribution together. In the new <em>x</em> there will be the original <em>token embedding + positional encoding + attention-aware encoding</em>. I understand this was the objective, and now it can be used to predict the next token, because all the information necessary to predict what comes next is packed in it.</p><p>In order to predict the next token, the new representation of the current token if fed to a <em>mlp</em> layer. <em>mlp</em> stands for multi-layer perceptron, it is a type of three layer feed foreword network, composed of input linear layer, a non linear function and an output projection layer.</p><pre>def mlp(x, scope, n_state, *, hparams):<br>    with tf.variable_scope(scope):<br>        nx = x.shape[-1].value<br>        h = gelu(conv1d(x, &#39;c_fc&#39;, n_state))<br>        h2 = conv1d(h, &#39;c_proj&#39;, nx)<br>        return h2</pre><p>First we have a <em>conv1d</em> linear layer operating on one sample, with 768 x 4 number of filters, it is in fact operating the following (again I focus on dimensions of the tensors because it helps understand what is going on)</p><p>(1, 768) x (768, 768 x 4) + (1, 768 x 4)</p><p>The result is fed to a non linear <em>gelu</em> function, then to the final projection layer which does (it goes back to the embedding dimension)</p><p>(1, 768 x 4) x (768 x 4, 768) + (1, 768)</p><p>Note that all weights are learned during the training of the Transformer. The very last step to compute the value of the new token, is to add the result of the <em>mlp</em> layer to the current token; I guess this implies that the new token is predicted as a delta with respect to the current token, and this makes sense if we recall that in token embedding, similar tokens are “near” in the space of the embeddings.</p><pre>x = x + m</pre><p>Now one <em>block</em> layer is completed, so it is time to go back to the Transformer structure.</p><pre>def model(hparams, X, past=None, scope=&#39;model&#39;, reuse=False):<br>    with tf.variable_scope(scope, reuse=reuse):<br>        results = {}<br>        batch, sequence = shape_list(X)<br><br>        wpe = tf.get_variable(&#39;wpe&#39;, [hparams.n_ctx, hparams.n_embd],<br>                             initializer=tf.random_normal_initializer(stddev=0.01))<br>        wte = tf.get_variable(&#39;wte&#39;, [hparams.n_vocab, hparams.n_embd],<br>                             initializer=tf.random_normal_initializer(stddev=0.02))<br>        past_length = 0 if past is None else tf.shape(past)[-2]<br>        h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length))<br><br>        # Transformer<br>        presents = []<br>        pasts = tf.unstack(past, axis=1) if past is not None else [None] * hparams.n_layer<br>        assert len(pasts) == hparams.n_layer<br>        for layer, past in enumerate(pasts):<br>            h, present = block(h, &#39;h%d&#39; % layer, past=past, hparams=hparams)<br>            presents.append(present)<br>        results[&#39;present&#39;] = tf.stack(presents, axis=1)<br>        h = norm(h, &#39;ln_f&#39;)<br><br>        # Language model loss.  Do tokens &lt;n predict token n?<br>        h_flat = tf.reshape(h, [batch*sequence, hparams.n_embd])<br>        logits = tf.matmul(h_flat, wte, transpose_b=True)<br>        logits = tf.reshape(logits, [batch, sequence, hparams.n_vocab])<br>        results[&#39;logits&#39;] = logits<br>        return results</pre><p>The same layer that we have seen so far is repeated 12 times in GPT-2, where the next token prediction is fed as input to the next layer. I guess it is improper to say that out of the first layer there will be the next token prediction; I suppose what we have is another representation of the current token, which is closer to the next token. Intuitively, if we assume that tokens are complete words, we might think that out of the first layer there will be a new word, which is not the current one, it is still not the next one, but it is a word which is somehow linking the two together. Layer by layer, this new word representation is again processed for attention to the sequence of previous tokens (recalling that each layer has its own set of weight for <em>q</em>, <em>k</em>, and <em>v</em> computation, and its own weights and biases for the <em>mlp</em> layer). Each layer will then capture different correlations between words, moving into a specific direction in the embedding space. Only the final layer is producing the estimation for the next token.</p><p>At the final stage, the next token estimation which is in the embedding space, is translated in the vocabulary space by multiplication of the transposed of the <em>wte, </em>the matrix that is connecting the vocabulary and the embedding space. The result will be a vector, called <em>logits</em>, of dimension</p><p>(1, 50257)</p><p>where each item carries an index with the likelihood of that specific token in the vocabulary. The <em>logits</em> probability distribution if first manipulated to inject some level of controlled randomness in the prediction outcome, and is then sampled via <em>tf.mutlinomianl</em>, which takes a random sample according to given probability distribution. Then everything starts over again for the next token.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47f8a73809d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>