<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gabriel Furnieles on Medium]]></title>
        <description><![CDATA[Stories by Gabriel Furnieles on Medium]]></description>
        <link>https://medium.com/@gabrielfurnieles?source=rss-e77c10fd9715------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bSzawM73yHvD9qc74D-Yig.jpeg</url>
            <title>Stories by Gabriel Furnieles on Medium</title>
            <link>https://medium.com/@gabrielfurnieles?source=rss-e77c10fd9715------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:52:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gabrielfurnieles/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Modern RAG Agent in 2026: Qwen3 Embeddings and Vector Database in Qdrant.]]></title>
            <link>https://pub.towardsai.net/building-a-modern-rag-pipeline-in-2026-qwen3-embeddings-and-vector-database-in-qdrant-ebeca2bbe338?source=rss-e77c10fd9715------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebeca2bbe338</guid>
            <category><![CDATA[qdrant]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[embedding]]></category>
            <dc:creator><![CDATA[Gabriel Furnieles]]></dc:creator>
            <pubDate>Tue, 17 Feb 2026 16:01:02 GMT</pubDate>
            <atom:updated>2026-02-19T08:05:42.050Z</atom:updated>
            <content:encoded><![CDATA[<h4>The ArXiv RAG Project, a scalable RAG Agent for ArXiv CS papers.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*47Xm6VzgBnOOct3SRg0qDQ.png" /></figure><p>Imagine chatting with an agent that has <strong>full access</strong> to all the scientific articles ever published on <a href="https://arxiv.org/">arxiv.com</a>, and that can answer any question you ask with <strong>scientific rigor</strong> and <strong>validated references</strong>.</p><blockquote>This post belongs to The ArXiv RAG Project Series, in which I’m building a scalable RAG Agent for ArXiv Computer Science papers. You can check the project’s source code on <a href="https://github.com/GabrielFurnieles/arxiv-rag">GitHub</a>.</blockquote><p>This very first post contains the technical implementation details and tools I’ve used to build an <strong>ETL pipeline</strong> that lays the foundations for the RAG Agent. The pipeline extracts the ArXiv metadata for over 500.000 papers from the <a href="https://www.kaggle.com/datasets/Cornell-University/arxiv">ArXiv Dataset</a>, computes its embeddings using the latest <strong>Qwen3-embedding-8b</strong> model (one of the strongest models available for RAG today), and finally loads everything into a <strong>Qdrant Vector DB</strong> that will serve as the query engine for the RAG Agent.</p><h3>Contents</h3><p><a href="#5de0"><strong>Introduction</strong></a><strong><br> </strong><a href="#ac65"><strong>What is RAG (Retrieval Augmented Generation)?</strong></a><strong><br> </strong><a href="#6cf3"><strong>Text Embeddings and Document Embeddings</strong></a><strong><br> </strong><a href="#242b"><strong>Modern RAG techniques in 2026</strong></a></p><p><a href="#51c5"><strong>Overall Project Structure</strong></a></p><p><a href="#1167"><strong>Building a CLI App Package</strong></a></p><p><a href="#d9b9"><strong>Extracting and Cleaning the Data</strong></a></p><p><a href="#556d"><strong>Computing the Embeddings</strong></a><strong><br> </strong><a href="#b97a"><strong>Preliminaries</strong></a><strong><br> </strong><a href="#06a8"><strong>The OpenAI Batch API</strong></a><strong><br> </strong><a href="#4dd2"><strong>Managing Batch Requests with a Local SQLite Database</strong></a><strong><br> </strong><a href="#40f7"><strong>Downloading the Embeddings</strong></a></p><p><a href="#4c4b"><strong>Storing Embeddings in a Vector DB</strong></a><strong><br> </strong><a href="#725f"><strong>Docker Setup</strong></a><strong><br> </strong><a href="#6bf8"><strong>Creating Collections</strong></a><strong><br> </strong><a href="#559f"><strong>Ingesting 500k Vectors At Scale</strong></a></p><p><a href="#dd0a"><strong>Wrap Up</strong></a><strong><br> </strong><a href="#fcd8"><strong>Next Steps</strong></a></p><h3>Introduction</h3><p>The science field, and especially Computer Science, is evolving faster than ever. Every day new advancements are published and previous paradigms that were considered SOTA become quickly obsolete. AI newsletters are a good way to keep up to date with all this buzz, but they lack something vital to learning: <strong>Interactivity</strong>. ArXiv RAG not only aims to be a practical personal project, but also to provide a tool I can use on a daily basis to explore new topics and study concepts from the scientific literature, using trustworthy sources and updated information from the cutting edge of research.</p><blockquote><strong>Note</strong>. If you are already familiar with RAG, text embeddings, and their concepts, skip to the next section where I start talking about the juicy implementation details.</blockquote><h4>What is RAG (Retrieval Augmented Generation)?</h4><p>Retrieval Augmented Generation is a technique used in Large Language Models (LLM) that allows them to consult and extract useful information from documents or any other sources (<strong>Knowledge base</strong>) so that they can respond in an accurate and informed way.</p><p>In a nutshell, RAG matches a Query <em>q </em>(input text) with every possible document or source from the Knowledge Base <em>D</em> and finds the top K most relevant ones. Finally, it includes the documents’ contents in the LLM context window alongside the original query, so the LLM has all the necessary information to elaborate a response. <a href="https://arxiv.org/abs/2005.11401">Research</a> shows that using RAG improves <strong>performance by up to 30% or more on open-domain questions</strong>. However, where RAG systems excel is in closed domains, where private and specific data like the user’s history, the product description and current price or the company’s legal documents are needed to elaborate a proper response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HAX_l6-a1B3_NC6u4EiivQ.png" /><figcaption><strong>RAG Diagram</strong>. Image by the author.</figcaption></figure><h4>Text Embeddings and Document Embeddings</h4><p>To be able to match a query with a document, RAG needs to convert both into a numerical representation (computers work with numbers, not text). This is where the family of <strong>Embedding Models</strong> comes into the scene. Embedding Models are AI models trained to generate a vector representation from a chunk of text, aiming at capturing as much of the semantic meaning as possible from it.</p><p>Vectors are a great representation for texts because we can compute distance metrics between them (Euclidean distance, Manhattan, Cosine similarity, …), getting a measure of how similar two vectors are. Embedding Models try to capture the text’s semantics by assigning closer vector representations to similar texts and vice versa.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F931fDWOOLAtEj0GZafBdg.png" /><figcaption><strong>Text Embeddings</strong>. In the diagram, only single-word texts are represented, but in practice, larger sentences and paragraphs are used. Image by the author.</figcaption></figure><blockquote><strong>Note.</strong> It is important to note that very large texts or documents can’t be converted into a single vector because of the limits of the Embedding Models (they can only convert up to a certain text length). In those cases <a href="https://qdrant.tech/course/essentials/day-1/chunking-strategies/">text chunking techniques</a> are applied to divide the large text/document into smaller meaningful pieces.</blockquote><h4>Modern RAG techniques in 2026</h4><p>Since the term RAG was first coined by <a href="https://arxiv.org/abs/2005.11401">Lewis et al., 2020</a>, a large number of advancements and techniques have emerged from the scientific literature.</p><p><em>Note to myself, this is an excellent use case for ArXiv RAG. Repeat this experiment when the agent is finished and display the response below.</em></p><p>According to Gemini 3:</p><blockquote><em>As of February 8, 2026, Retrieval-Augmented Generation (RAG) has moved far beyond simple &quot;vector search + prompt.&quot; In the current landscape, the focus has shifted from </em><strong><em>static pipelines</em></strong><em> to </em><strong><em>agentic reasoning</em></strong><em> and </em><strong><em>structured knowledge</em></strong><em>.<br> <br></em><strong>1. The Three Modern Architectures</strong><em><br>The &quot;standard&quot; RAG pipeline has diverged into three specialized paths depending on the complexity of the data:</em></blockquote><blockquote><strong><em>- Agentic RAG (The &quot;Researcher&quot;)</em></strong><em>: Instead of a fixed workflow, an LLM agent uses reasoning to plan its retrieval. It can decide to search multiple times, self-correct if the first results are poor, and use tools to browse the web or query specific databases.</em></blockquote><blockquote><strong><em>- GraphRAG (The &quot;Connected&quot; Approach):</em></strong><em> Combines vector search with Knowledge Graphs. It extracts entities and relationships from text, allowing the AI to answer &quot;multi-hop&quot; questions (e.g., &quot;How is the CEO&#39;s recent strategy influenced by the 2024 acquisition?&quot;) that standard vector search usually misses.</em></blockquote><blockquote><strong><em>- Modular RAG (The &quot;Toolkit&quot;):</em></strong><em> A highly flexible architecture where modules (Reranker, Router, Search, Refiner) can be swapped or rearranged dynamically based on the query type.</em></blockquote><blockquote><strong>2. Advanced Retrieval &amp; Processing Techniques</strong><em><br>If you are building or optimizing a system today, these are the &quot;must-have&quot; techniques:<br><br></em><strong><em>Hybrid Search &amp; Reranking</em></strong><em><br>- </em><strong><em>Hybrid Search:</em></strong><em> Merging Semantic (Vector) search with Lexical (BM25/Keyword) search is now the baseline. This ensures you catch both &quot;vibes&quot; and exact technical terms/IDs.<br>- </em><strong><em>Cross-Encoder Reranking:</em></strong><em> After retrieving the top 50-100 chunks, a smaller, high-precision model (like a BGE-Reranker) re-scores them to ensure the top 5 sent to the LLM are the absolute best.<br><br></em><strong><em>Context Optimization</em></strong><em><br>- </em><strong><em>Parent-Child Chunking:</em></strong><em> Storing small &quot;child&quot; chunks for high-precision retrieval but passing the larger &quot;parent&quot; context to the LLM to provide better background.<br>- </em><strong><em>RAPTOR (Recursive Abstractive Processing):</em></strong><em> Recursively clustering and summarizing chunks at different levels, allowing the model to answer both granular and high-level thematic questions.<br>- </em><strong><em>Contextual Retrieval:</em></strong><em> Pre-pending a short summary of the whole document to every individual chunk so that the &quot;meaning&quot; isn&#39;t lost when chunks are retrieved in isolation.<br><br></em><strong>3. Emerging &quot;State-of-the-Art&quot; (Early 2026)</strong></blockquote><blockquote><em>- </em><strong><em>State-Aware Retrieval:</em></strong><em> Systems that maintain a &quot;retrieval plan&quot; based on the user&#39;s conversation history and intent, rather than treating every query as a fresh start.</em></blockquote><blockquote><em>- </em><strong><em>Multimodal RAG:</em></strong><em> Directly retrieving and reasoning over images, PDFs with complex tables, and video clips using unified embedding spaces (like CLIP or more modern 2025/26 vision-language models).</em></blockquote><blockquote><em>- </em><strong><em>Long-Context RAG:</em></strong><em> With 2026 models supporting millions of tokens in context, RAG is no longer just for &quot;finding&quot; information but for filtering it. Modern RAG acts as a &quot;curator&quot; that stuffs the most relevant 100k tokens into the window to minimize cost and latency while maximizing accuracy.<br><br></em><strong>4. The 2026 Toolkit</strong><em><br>The ecosystem has matured into several heavy-hitting frameworks:<br><br>- </em><strong><em>Orchestration:</em></strong><em> LangChain (still dominant), LlamaIndex (data-centric), and Dify (visual/enterprise).<br>- </em><strong><em>Evaluation:</em></strong><em> Ragas and Arize Phoenix are the standard for measuring &quot;faithfulness&quot; and &quot;relevance&quot; to prevent hallucinations.<br>- </em><strong><em>Memory: </em></strong><em>Mem0, which provides a persistent, evolving memory layer for RAG agents to remember user preferences across sessions.</em></blockquote><p>In this project, I aim to combine some of these techniques, exploring Agentic RAG workflows with Hybrid Search and Reranking strategies.</p><h3>Overall Project Structure</h3><p>First things first, I want to clarify how I’m organizing this project as a multi-package repository, meaning that <a href="https://github.com/GabrielFurnieles/arxiv-rag/">the GitHub repo</a> is going to contain two or more code packages. For the moment, I’m only thinking of two packages: one for the pipeline and another for the agent, but in the future I might add new packages like a RESTful API or a frontend layer.</p><pre>.<br>├── LICENSE<br>├── README.md<br>├── pipeline           # package 1<br>│   ├── README.md<br>│   ├── pyproject.toml<br>│   ├── src/<br>│   └── uv.lock<br>└── agent              # package 2 (yet to be published in the repo)<br>    ├── README.md<br>    ├── pyproject.toml<br>    ├── src/<br>    └── uv.lock</pre><p><strong><em>Why multiple packages?<br></em></strong><em> </em>I want to make a clear separation of concerns: The <strong>pipeline</strong> package will handle all the logic and resources necessary to build up the Knowledge Base, while the <strong>agent</strong> package will contain all the agent workflow, including the RAG mechanisms and LLM API calls. Each package has its own set of dependencies and execution profiles, with varying resource needs: pipeline will run most of the time (if not all) offline, scheduled to be executed every certain time to update the Knowledge Base, and will likely demand more memory to process the data; whereas agent will be running online every time the user is connected, waiting for an input and consuming less memory (though it will require from the resources of other services like the Vector DB and the LLM API). These arguments make up a good statement to build separate packages.</p><p><strong><em>Why use a single repo?<br></em></strong>Well, there is no need. And technically, if this were a final product or if I were working with other people, I wouldn’t do it. But I want to keep all the code centralized in a single place where I can share it and show it. One repo is easier to maintain than two.</p><blockquote><strong>Note.</strong> This post focuses only on the building of the pipeline package. Future posts will deal with the agent.</blockquote><h3>Building a CLI App Package</h3><pre>&quot;GitHub Repo files reference:&quot;<br><br># /pipeline/src/pipeline/cli.py</pre><p>Even though the pipeline package will run in offline mode, i.e. a scheduled cron job will spawn a new process and execute the scripts from the package. I want to give it a clean and intuitive interface to test and call the pipeline steps.</p><p>For this project, I’m using the <a href="https://typer.tiangolo.com/"><strong>Typer</strong></a> library, a very convenient and easy-to-use Python library for building Command-Line Interface applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6iubJLOBV_rU6hR1gXMYkA.png" /><figcaption><strong>CLI Pipeline App</strong>. Typer´s help guide. Image by the author.</figcaption></figure><h3>Extracting and Cleaning the Data</h3><pre>&quot;GitHub Repo files reference:&quot;<br><br># /pipeline/src/pipeline/etl/extract.py<br># /pipeline/src/pipeline/etl/transform.py</pre><p>The <a href="https://www.kaggle.com/datasets/Cornell-University/arxiv">Kaggle’s ArXiv Dataset</a> contains metadata information about all the published papers in ArXiv since 25–04–1986. It is updated on a weekly basis and as of today’s date, it contains <strong>2.938.427 articles </strong>from different science domains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TjVxBebX-OHwmSPpJU0fGQ.png" /><figcaption><strong>First Dataset Rows from the Kaglle´s ArXiv Dataset</strong>. Image by the author.</figcaption></figure><p>To narrow down the scope of the project, I’ve decided to only filter down some categories related to Computer Science, which left me with a bit over <strong>560.000 articles</strong>.</p><pre>CATEGORIES = [<br>    &quot;cs.AI&quot;,  # &quot;Artificial Intelligence&quot;,<br>    &quot;cs.CG&quot;,  # &quot;Computational Geometry&quot;,<br>    &quot;cs.CL&quot;,  # &quot;Computational Language&quot;,<br>    &quot;cs.CV&quot;,  # &quot;Computer Vision and Pattern Recognition&quot;,<br>    &quot;cs.IR&quot;,  # &quot;Information Retrieval&quot;,<br>    &quot;cs.LG&quot;,  # &quot;Machine Learning&quot;,<br>    &quot;cs.MA&quot;,  # &quot;Multiagent Systems&quot;,<br>    &quot;cs.NE&quot;,  # &quot;Neural and Evolutionary Computing&quot;,<br>    &quot;cs.RO&quot;,  # &quot;Robotics&quot;<br>]</pre><p>The Extraction and Cleaning pipeline works in two stages, creating intermediate files to store the temporal results:</p><ol><li><strong>Download metadata </strong>→ data/kaggle/arxiv-metadata-oai-snapshot.json</li><li><strong>Filter categories and clean columns </strong>→ data/interim/clean-arxiv-metadata-oai.parquet</li></ol><blockquote><strong>Note.</strong> To download the Kaggle dataset, you must use the Kaggle API for Python and identify yourself with a secret key that you have to generate in your Kaggle account.</blockquote><p>Because the original metadata file is really big (over 5GB), I’m using polars lazy scan to stream the file and process it using <a href="https://docs.pola.rs/user-guide/lazy/"><strong>lazy evaluation</strong></a>. This is a crucial step to ensure the pipeline can run in lighter hardware settings and delegates the optimization process to polars.</p><pre># /pipeline/src/pipeline/etl/transform.py<br>lf = pl.scan_ndjson(source=self.file, schema=JSON_SCHEMA)</pre><p>Note that to make the process more efficient I even specify the file’s schema to save polars the overhead.</p><p>Finally, I apply some basic preprocessing operations to the texts, like removing unnecessary new line characters \n and extra spaces, and formatting dates and authors so that the data is more suited for queries.</p><pre># /pipeline/src/pipeline/etl/transform.py<br>.with_columns(<br>    pl.col(&quot;title&quot;)<br>    .str.replace_all(&quot;\n&quot;, &quot; &quot;)<br>    .str.replace_all(r&quot;\s+&quot;, &quot; &quot;)<br>    .str.strip_chars(),<br>    pl.col(&quot;abstract&quot;)<br>    .str.replace_all(&quot;\n&quot;, &quot; &quot;)<br>    .str.replace_all(r&quot;\s+&quot;, &quot; &quot;)<br>    .str.strip_chars(),<br>    pl.col(&quot;authors_parsed&quot;).list.eval(<br>        pl.element().list.reverse().list.join(&quot; &quot;).str.strip_chars()<br>    ),<br>    pl.col(&quot;versions&quot;)<br>    .list.last()<br>    .struct.field(&quot;created&quot;)<br>    .str.to_datetime(format=&quot;%a, %d %b %Y %H:%M:%S GMT&quot;)<br>    .alias(&quot;last_submitted&quot;),<br>)</pre><h3>Computing the Embeddings</h3><pre>&quot;GitHub Repo files reference:&quot;<br><br># /pipeline/src/pipeline/embeddings.py<br># /pipeline/src/pipeline/schemas.py<br># /pipeline/src/pipeline/utils.py<br># /pipeline/src/pipeline/db/*</pre><p>With the dataset already downloaded and cleaned, it is time to compute the embeddings. To do so, I use the title and abstract columns from the dataset, concatenating them into a single text chunk:</p><pre>{TITLE}\n<br>\n<br>{ABSTRACT}</pre><h4>Preliminaries</h4><p>Before throwing myself into writing the code to compute the embeddings, I had to clear two important questions:</p><p><strong><em>Which Embedding Model to use?<br></em></strong>At the beginning of the project, I started exploring OpenAI <a href="https://developers.openai.com/api/docs/models/text-embedding-3-small"><strong>text-embedding-3-small</strong></a>. However, it was later that I headed to the <a href="https://huggingface.co/spaces/mteb/leaderboard"><strong>MTEB Leaderboard</strong></a><strong> </strong>to see the latest model evaluations, and it turned out that the <a href="https://qwen.ai/blog?id=qwen3-embedding"><strong>Qwen3-Embedding-8b</strong></a> is a much stronger model for RAG, supporting advanced RAG techniques such as Matryoshka Representative Learning (more on that in the next post), and with a reasonable price, so I finally opted to go with it.</p><p><strong><em>What model provider should I choose?<br></em></strong>Qwen3-Embedding is an open source model, meaning that I can download it and run it in any environment I want. However, I don’t have a powerful GPU at home, and although I could use a Cloud provider to rent one, the overhead of setting up the environment and optimizing the serving process to get the embeddings in a reasonable amount of time was too much trouble. In the end, I opted to use <a href="https://nebius.com/services/token-factory"><strong>Nebius.com</strong></a><strong> </strong>because of its reasonable price ($0.01 / 1M Tokens In) and compatibility with the OpenAI Python API (which I had already started to use with my previous tests)</p><h4>The OpenAI Batch API</h4><p>Calling the API to compute the embedding for one text is very straightforward. Doing it again for a couple of hundreds or thousands of texts is fine. But computing <strong>+500.000 embeddings</strong> isn’t that easy.</p><p>You have to deal with the <strong>requests and token limits</strong>, which in most cases are insufficient for this amount of texts. And then you have to also set up a well engineered mechanism to <strong>parse and store all of the responses on the flow</strong>, implementing asynchronous steps nicely, handling errors and retires, and ensuring your memory is not going to overflow on the process. The second is not much a big issue but a technical challenge that can be addressed. However, because of the API limits, getting +500k embeddings require an indecent amount of time, and it is not a solution that is going to scale well (imagine doing so for the entire ArXiv dataset). The alternative, and proper way to calculate embeddings for large bodies of text, is to use the <a href="https://developers.openai.com/api/docs/guides/batch/"><strong>Batch API</strong></a>, which processes all the requests offline and has very large rate limits that aren’t restricted by time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BHY2nyhhG7BnX_KkUP4kxw.png" /><figcaption><strong>Batch API</strong>. Requests are grouped in JSONL files up to a certain file size limit and sent to the API. Responses can take up to 24h. Image by the author.</figcaption></figure><p>Each line of the Batch file contains a JSON object with all the information for the request, like the API endpoint to reach, the model selected and the input texts.</p><pre>{&quot;custom_id&quot;: &quot;batch-0001-0001-request-00001&quot;, &quot;method&quot;: &quot;POST&quot;, &quot;url&quot;: &quot;/v1/embeddings&quot;, &quot;body&quot;: {&quot;model&quot;: &quot;Qwen/Qwen3-Embedding-8B&quot;, &quot;input&quot;: [&quot;Pareto and Boltzmann-Gibbs behaviors in a deterministic multi-agent system\n\nA deterministic system of interacting agents is considered as a model for economic dynamics. The dynamics of the system is described by a coupled map lattice with near neighbor interactions. The evolution of each agent results from the competition between two factors: the agent&#39;s own tendency to grow and the environmental influence that moderates this growth. Depending on the values of the parameters that control these factors, the system can display Pareto or Boltzmann-Gibbs statistical behaviors in its asymptotic dynamical regime. The regions where these behaviors appear are calculated on the space of parameters of the system. Other statistical properties, such as the mean wealth, the standard deviation, and the Gini coefficient characterizing the degree of equity in the wealth distribution are also calculated on the space of parameters of the system.&quot;, &quot;On the Effects of Idiotypic Interactions for Recommendation Communities in Artificial Immune Systems\n\nIt has previously been shown that a recommender based on immune system idiotypic principles can out perform one based on correlation alone. This paper reports the results of work in progress, where we undertake some investigations into the nature of this beneficial effect. The initial findings are that the immune system recommender tends to produce different neighbourhoods, and that the superior performance of this recommender is due partly to the different neighbourhoods, and partly to the way that the idiotypic effect is used to weight each neighbours recommendations.&quot;,...]}}</pre><p>Note that the Batch files have also some constraints. According to <a href="https://docs.tokenfactory.nebius.com/ai-models-inference/batch-inference">Nebius Batch API docs</a>, each file can be up to <strong>10GB</strong> size or <strong>5.000.000 requests</strong>. The texts to encode cannot exceed the maximum model’s context length, which for Qwen3-Embedding-8b is <strong>41k token context</strong>. And a a maximum tokens per request (TPR) is usually enforced (note that multiple texts can be sent in a single request). Although I haven’t been able to find an specific TPR limit in Nebius, I know from other providers like OpenAI that they allow a maximum of 300.000 TPR accross all texts. So far the limits you need to consider when using the Batch API are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kf9OJnSOWBBaHeg7bPORTg.png" /><figcaption><strong>Batch API Limits</strong>. Image by the author.</figcaption></figure><blockquote><strong>Note.</strong> One interesting thing I’ve found during my tests with the OpenAI API is that they estimate the number of tokens using a 0.25 tokens per utf-8 byte formula. This can led to API limit errors if you measure the tokens using other more accurate libraries like tiktoken, so make sure you use a slightly smaller limit.</blockquote><p>Using the Batch API requires to store and manage files, IDs, statuses and other important information. In the next subsection I explain how I leverage a Local SQLite Database to deal with all this data and maintain a record of every batch so that later I can consult its status and download the reponses.</p><h4>Managing Batch Requests with a Local SQLite Database</h4><p>An important aspect when working with the Batch API is to keep track of the batch IDs and statuses. While all the processing and individual requests are handled on the provider’s side, you still have to check the status and download the response files using the provided IDs. Instead of relying in plain files, I’ve decided to set up a simlpe <strong>SQLite Database</strong> to store this information, so that I can regularly query and update each batch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ELKZxdKilnbnKP-93Fxyw.png" /><figcaption><strong>Relational Diagram for the SQLite Database</strong>. Image by the author.</figcaption></figure><p>This new feature comes at an additional cost, as now, in addition to creating the database models, I also have to manage connections and implement CRUD operations. For this purpose I chose <a href="https://www.sqlalchemy.org/"><strong>SQLAlchemy</strong></a> as it is the classic industry ORM standard. To have greater control over the data handled by the application, I have also created different schemas with built-in validation using Pydantic. You can check all of this in /pipeline/src/pipeline/db/* and /pipeline/src/pipeline/schemas.py.</p><p>The <strong>Job</strong> entity represents a pipeline call that processes multiple texts from a given file. When submitting a new Job, the <strong>BatchManager</strong> creates the different Batches based on the API limits. Then, the files are uploaded to the providers’ servers via the API, and finally a batch requests pointing to those files is sent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I70kRm427HCP0MpG6lh3yw.png" /><figcaption><strong>compute-embeddings</strong>. Creating and sending Batch API requests for a test with 50.000 texts, console output. Image by the author.</figcaption></figure><p>After submitting the Batch requests, I’ve created a command check-embeddings-status that displays a summary table that refreshes every minute with all the batch statuses and relevant information. This command also updates the statuses from the SQLite database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GfASdn0ajYZXvbv_upwxUA.png" /><figcaption><strong>check-embeddings-status</strong>. Summary table with the status for each Batch Request from Job 9. Because the TPR is very high and the texts are quite short (title + abstract) many texts can fit within a single request. In total, this batch has 59 requests. Image by the author.</figcaption></figure><h4>Downloading the Embeddings</h4><p>Once all the batches from a given Job are completed, it is time to download and parse the results.</p><p>This is also a very crucial step, since for each text we’ve submitted we are getting a 4.096 float32 vector. Thus, the response files are x8 or even x10 times the size of their respective requests, and <strong>memory resources can be exhausted very easily</strong>.</p><p>To address the potential memory issues I’ve implemented streaming pipelines and file batch processing.</p><pre># /pipeline/src/pipeline/embeddings.py<br># Line 597. Streaming the response file to a local file.<br><br># Stream OAI file to local<br>with client.files.with_streaming_response.content(fout) as content:<br>    with open(fnew, &quot;wb&quot;) as f:<br>        for chunk in content.iter_bytes():<br>            f.write(chunk)</pre><p><strong><em>Sorting Request within each Batch: Sorted Maps VS File Sorting<br></em></strong>A quiet annoying downside of working with the Batch API is that the <a href="https://developers.openai.com/api/docs/guides/batch#:~:text=Note%20that%20the%20output%20line%20order%20may%20not%20match%20the%20input%20line%20order.%20Instead%20of%20relying%20on%20order%20to%20process%20your%20results%2C%20use%20the%20custom_id%20field%20which%20will%20be%20present%20in%20each%20line%20of%20your%20output%20file%20and%20allow%20you%20to%20map%20requests%20in%20your%20input%20to%20results%20in%20your%20output">Batch Response files aren’t sorted in the same order as the original requests</a>, i.e. <strong>the output line order may not match the input line order</strong>. According to the docs: ”<em>Instead of relying on order to process your results, use the </em><em>custom_id which will be present in each line of your output file and allow you to map requests in your input to results in your output.</em>”.<br>However, because I new I was going to parse the output file line by line to extract the embeddings, I decided to use a <strong>sorted map pattern</strong>: Instead of sorting the entire file by copying and pasting each line to a new <em>“sorted_file.jsonl”,</em> I create a list of ordered tuples with the values (custom_id, line_position) that later the parsing function will use to parse the results in the correct order. This method not only saves the overhead to create and write to new files (one per Batch) but also is a lot more efficient, since the custom_id appears at the beginning of every line in the batch response, I don’t need to parse an entire 100+ MB line (each one packed with multiple 4019-dimensional arrays) but only read a tiny small fraction of it, more specifically 500 bytes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t9sxVKDOQxd3G8VA4dy58Q.png" /><figcaption><strong>Sorted Map Diagram</strong>. Note that the line positions (offset) are specified in bytes. Image by the author</figcaption></figure><p><strong><em>From JSONL to RAW to NPY<br></em></strong>The API response files come in JSONL format. However, although this file type is great for HTTP responses, it isn’t the most efficient one to store the vector embeddings. Ideally, I decided to use a single <strong>.npy</strong> file, which for +500k vectors is fine. But there was a problem with it: .npy files force you to define the dimensions of the data beforehand when creating the file, so I couldn&#39;t add more vectors on the fly. For this reason the whole download process looks like:</p><ol><li><strong>Download.</strong> Stream download the JSONL files to local.</li><li><strong>Sort.</strong> Sort requests within each batch.</li><li><strong>Parse.</strong> Parse the JSONL files line by line extracting the embeddings in the correct order and appending them into a single .raw file (during this step I also count the total number of vectors since I have to iterate over them one by one).</li><li><strong>To NPY.</strong> Stream the .raw file to a .npy file in 100MB chunks.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_fXyM0B8RxzDnZvj3w85xA.png" /><figcaption><strong>get-embeddings</strong>. Console output. Image by the author.</figcaption></figure><p>With the data cleaned and embedded, the next step is building the query engine for the RAG agent. At this stage, performance and efficiency become critical: ingestion pipelines must be optimized to handle +500k datapoints without bottlenecks, and infrastructure choices directly impact scalability. These are the Qdrant setup and methods I’ve built.</p><h3>Storing Embeddings in a Vector DB</h3><pre>&quot;GitHub Repo files reference:&quot;<br><br># /pipeline/compose.yaml<br># /pipeline/src/pipeline/etl/load.py</pre><p><strong><em>What is a Vector DB?<br></em></strong>A Vector Database is a special type of database built to <strong>store and search vectors</strong>. Unlike relational databases, which store the data as records (rows), vector databases store data as <strong>points</strong>, where the coordinates are defined by the vector and a payload with the metadata (fields) is linked to it. Vector databases are optimized to compare vectors efficiently, more specifically compute the distances or similarities between them (Euclidean, Manhattan, Cosine, …) This is useful for tasks like semantic search, recommendation systems, or finding similar images. <br>Because running these operations among hundreds of thousands or even millions and billions of vectors is usually prohibitively expensive, vector databases use specialized indexing algorithms such as <a href="https://www.elastic.co/blog/understanding-ann">Approximate Nearest Neighbors (ANN)</a>, which trade a small amount of accuracy for much faster search. <br>It is important to note that although vector databases are powerful for similarity search, they are not meant to replace relational databases. They are not optimized for complex joins, heavy transactions, or traditional reporting queries.</p><p><strong><em>Why Qdrant?<br></em></strong>Qdrant is one of the top Open Source vector databases for AI and semantic search. Written in Rust, it optimizes every byte of memory and performs operations blazingly fast. Qdrant not only offers a managed Cloud service, but because it is OS you can host it and run it on any machine you want without any additional cost.</p><blockquote><strong>Note.</strong> This post hasn’t been sponsored by Qdrant.</blockquote><h4>Docker Setup</h4><p>Setting up a Docker container for Qdrant is very straightforward. The only two things to keep in mind are:</p><ol><li><strong>Configuring the volumes</strong> to store the data (otherwhise the data won’t be persistent and will be lost once the container is restarted or stoped)</li><li><strong>Enabling API Key Authentication</strong> by adding the QDRANT__SERVICE__API_KEY envvar.</li></ol><pre>services:<br>  qdrant:<br>    image: qdrant/qdrant:latest<br>    restart: always<br>    container_name: qdrant<br>    ports:<br>      - 6333:6333<br>      - 6334:6334<br>    volumes:<br>      - ./qdrant_data:/qdrant/storage<br>    environment:<br>     - QDRANT__SERVICE__API_KEY=${QDRANT_API_KEY}<br>    healthcheck:<br>      test: [&quot;CMD&quot;, &quot;curl&quot;, &quot;-f&quot;, &quot;http://localhost:6333/healthz&quot;]<br>      interval: 30s<br>      timeout: 10s<br>      retries: 3<br>    networks:<br>      - qdrant-network<br><br>networks:<br>  qdrant-network:<br>    driver: bridge<br><br>volumes:<br>  qdrant_data:</pre><blockquote><strong>Note.</strong> Qdrant exposes two ports 6333 and 6334 since it supports two different communication protocols: 6333 is for HTTP/HTTPS and 6334 for gRPC, a high-performance binary protocol developed by Google for high-speed data ingestion. Qdrant client libraries normally use the gRPC port while the 6333 port serves the Web UI and other lighter checks like manual data checks or health checks.</blockquote><p>Qdrant also has the option to enable <strong>TLS certificates</strong> so that all requests are redirected through HTTPS. This is very important for production environments in which otherwise the API Key will be exposed. However, note that if you use a reverse proxy like Traefik or Nginx to route all the traffic to the server, you don’t need to worry about this, since you can handle the TLS security check at the proxy level.</p><p>Finally, Qdrant exposes a Web UI at {URL}:6333/dashboard for the users to visualize and manage the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*etbLSu9W26AOfJMM2Wgngg.png" /><figcaption><strong>Qdrant Web UI</strong>. In local development <em>{URL} = </em><a href="http://localhost/"><em>http://localhost</em></a>. Image by the author.</figcaption></figure><h4>Creating Collections</h4><p>Qdrant stores points (vectors with a payload) in <a href="https://qdrant.tech/documentation/concepts/collections/">Collections</a>.</p><blockquote>“A collection is a named set of points among which you can search. The vector of each point within the same collection must have the same dimensionality and be compared by a single metric.”</blockquote><p>You can create collections directly from the Web UI. However, this isn’t ideal since later you wouldn’t be able to quicly recreate the same setup. Instead I’ve used Qdrant’s Python client library:</p><pre># /pipeline/src/pipeline/etl/load.py<br>from qdrant_client import QdrantClient, models<br>...<br><br>class QdrantLoader:<br>    def __init__(self):<br>        self.client = QdrantClient(<br>            url=config(&quot;QDRANT_URL&quot;), api_key=config(&quot;QDRANT_API_KEY&quot;)<br>        )<br>        self._embeddings = None<br>        self._metadata = None<br><br>    def create_collection(self, name: str, emb_dim: int, recreate: bool = False):<br>        if recreate:<br>            self.client.delete_collection(collection_name=name)<br>            logger.info(f&quot;🧹 Removed &#39;{name}&#39; collection&quot;)<br><br>        self.client.create_collection(<br>            collection_name=name,<br>            vectors_config=models.VectorParams(<br>                size=emb_dim, distance=models.Distance.COSINE, on_disk=True<br>            ),<br>            quantization_config=models.BinaryQuantization(<br>                binary=models.BinaryQuantizationConfig(always_ram=True)<br>            ),<br>            optimizers_config=models.OptimizersConfigDiff(<br>                max_segment_size=60_000  # ~1/8 of the total number of vectors<br>            ),<br>            hnsw_config=models.HnswConfigDiff(<br>                m=0,  # Disable indexing for for upload. NOTE. Make sure to enable indices after upload<br>                on_disk=False,  # Keep indices always on RAM<br>            ),<br>        )<br>   ...</pre><p>Note that in order to create the collection, I need to fix the <strong>vector size </strong>(in this case 4096 for Qwen3 embeddings) and the <strong>distance metric</strong> (Cosine Similarity) Because I’ll be performing a bulk upload of a large number of points, I temporarily <strong>disable indexing</strong>. This significantly speeds up the ingestion process and makes the upload much more efficient.</p><h4>Ingesting 500k Vectors At Scale</h4><p>The Qdrant Python library has various methods for <a href="https://qdrant.tech/course/essentials/day-4/large-scale-ingestion/">large-scale data ingestion</a>. Because I want to make this project as scalable as possible up to millions of points, I use .upload_collection.</p><pre># /pipeline/src/pipeline/etl/load.py    <br>class QdrantLoader:<br>    <br>    ...<br>    <br>    def load_vectors(<br>        self,<br>        job_id: int,<br>        collection: str,<br>        chunk_size: int = 10_000,<br>        recreate: bool = False,<br>    ):<br>        if not self._check_job_status(job_id):<br>            return<br><br>        if recreate:<br>            current_collection = self.client.get_collection(collection)<br>            emb_dim = current_collection.config.params.vectors.size<br>            self.create_collection(collection, emb_dim, recreate=True)<br><br>        total_points = self._init_resources(job_id)<br>        logger.info(<br>            f&quot;Loading {total_points} vectors into &#39;{collection}&#39; Qdrant collection...&quot;<br>        )<br><br>        for offset in range(0, total_points, chunk_size):<br>            end = min(offset + chunk_size, total_points)<br><br>            vectors, metadata = self._download_chunk(job_id, offset, end)<br><br>            self.client.upload_collection(<br>                collection_name=collection,<br>                vectors=vectors,<br>                payload=metadata.collect().to_dicts(),<br>                ids=tqdm.tqdm(range(offset, end)),<br>                batch_size=256,<br>                parallel=os.cpu_count(),<br>            )<br><br>        # Update indices<br>        self.client.update_collection(<br>            collection,<br>            hnsw_config=models.HnswConfigDiff(m=16, on_disk=False),<br>        )</pre><p>One important step to bear in mind is that self._download_chunk() returns a tuple of iterators instead of loading into memory the whole chunk of vectors + metadata. This approach ensures that the RAM usage stays low.</p><p>Finally, I enable the indices after the upload is completed. Note that they might take some time to fully propagate across all points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EYVeCrfwuQ2ST6UVcGbfUg.png" /><figcaption><strong>Qdrant arxiv-rag Collection from the Web UI</strong>. The graph displays the top 5 nearest points according to their Cosine Similarity. Image by the author.</figcaption></figure><h3>Wrap Up</h3><p>Implementing the foundational layer of the ArXiv RAG project has been a challenging and funny journey. Not only I have learned the intricacies and functioning of the Batch API, but I have also implemented memory-efficient pipelines to create and read files while keeping track of records in an in-memory SQLite database.</p><p>The key takeaways from this phase have been:</p><ul><li><strong>The Power of Batching:</strong> Mastering the <strong>Batch API</strong> has been a game-changer for cost and rate-limit management.</li><li><strong>Lazy DataFrames and File reading efficiency</strong>: Harnessing Polars Lazy DataFrames and using patterns like <em>map-sorted</em> or batch file reading greatly reduced the memory consumption and made the project scalable and resilient.</li><li><strong>Keeping track with SQLite:</strong> Although it brought more complexity to the project, implementing the SQLite storage helped make it more robust and production-ready.</li><li><strong>Qdrant experience:</strong> Working with Qdrant’s client has been a smooth and enjoyable experience. It just simply works, and it is really fast. I will definitely recommend it for future projects.</li></ul><h4>Next Steps</h4><p>A RAG system is only as good as its retrieval accuracy. In the next posts in the series, I’ll be implementing and sharing the techniques and design choices that transform a basic pipeline into a truly capable RAG agent. From evaluating the embedding quality to more refined retrieval and reasoning patterns, I’ll be building a production-grade RAG Agent in 2026 that scales and works. Stay tuned and enjoy life!</p><p>✅ <strong>Phase I:</strong> Knowledge Base</p><p>⬜ <strong>Phase II: The Retrieval Engine</strong>. Evaluating embedding quality, implementing Hybrid Search, and the first RAG Agent prototype.</p><p>⬜ <strong>Phase III: Agentic RAG</strong>. Moving from simple retrieval to autonomous reasoning.</p><p>⬜ <strong>Phase IV: Deployment and Scaling.</strong></p><p><em>Hi, I’m Gabriel Furnieles, a Mathematical Engineer specializing in Artificial Intelligence, Data Pipelines, and Software Engineering. If you’ve made it this far, thank you very much for reading the article.</em> <em>I hope you enjoyed it and found it helpful. If so, please consider following me </em><a href="https://medium.com/u/e77c10fd9715"><em>Gabriel Furnieles</em></a>,<em> and subscribing to my newsletter so stories will be sent directly to you! 👇</em></p><p><a href="https://medium.com/@gabrielfurnieles">Gabriel Furnieles - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebeca2bbe338" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/building-a-modern-rag-pipeline-in-2026-qwen3-embeddings-and-vector-database-in-qdrant-ebeca2bbe338">Building a Modern RAG Agent in 2026: Qwen3 Embeddings and Vector Database in Qdrant.</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self-Hosting My First Web App On a 5€ VPS]]></title>
            <link>https://levelup.gitconnected.com/self-hosting-my-first-web-app-on-a-5-vps-879f0991358d?source=rss-e77c10fd9715------2</link>
            <guid isPermaLink="false">https://medium.com/p/879f0991358d</guid>
            <category><![CDATA[hosting]]></category>
            <category><![CDATA[vps]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Gabriel Furnieles]]></dc:creator>
            <pubDate>Tue, 21 Jan 2025 21:07:39 GMT</pubDate>
            <atom:updated>2025-01-21T21:07:39.967Z</atom:updated>
            <content:encoded><![CDATA[<h4>Lessons I Learned From Deploying My First React App on a Private Server</h4><blockquote><strong>Author’s Note</strong>. This article is a bit different from what I’m used to sharing here on Medium. My focus of study has usually been on topics related to AI, ML, and Python; and although I still have a lot to learn (and will continue to do so), I’ve always had a thorn in my side to learning how to deploy and share my projects with the world. In the last few months, I’ve focused my attention on web development to fulfill this purpose. This article is the result of one of the tiny projects I’m planning to build and I hope it might be of inspiration to you🚀<br>Disclaimer: I am not a professional web developer and the advice I share is the result of my self-learning. All constructive comments are welcome.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*knqpi-nr5FGcdfIYMs78_g.png" /><figcaption>Background by <a href="https://unsplash.com/es/@solomin?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Dima Solomin</a> from <a href="https://unsplash.com/es/fotos/un-primer-plano-de-un-objeto-rojo-con-un-fondo-borroso-wEK9gAJPL8s?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a>. Image edited by the author.</figcaption></figure><h3>rgb2code</h3><p><a href="https://rgb2code.com/"><strong>rgb2code</strong></a> is a color converter tool for designers and programmers to convert multiple colors simultaneously from different color codes (RGB, HEX, etc.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hk5xxyPq3dwgTUzhh7KliQ.png" /><figcaption>Screenshot taken from <a href="https://rgb2code.com/">rgb2code.com</a>. Image by the author.</figcaption></figure><p>It is probably not the most valuable app on the internet, nor the one with many features and functionality, but it solves one of my pains: <em>converting and formatting colors manually one by one</em>. When creating visualizations or designing reports, I’ve noticed that I lost several minutes of my time converting the color codes and formatting them in the right way for my code. Using custom colors that were in different codes made my code uglier and produced me goosebumps.</p><p>rgb2code not only converts multiple colors at the same time but also supports different input formats, and prints a small visor with the color to help you select the ones you like to use.</p><p>This tiny app taught me a lot about <strong>React</strong>, <strong>Tailwind CSS</strong>, and deploying a web app in a <strong>cheap VPS</strong> without recurring to expensive platforms like Vercel.</p><h3>Deployment Configuration</h3><h4>VPS</h4><p>A Virtual Private Server (VPS) is a virtual machine sold as a service by an internet hosting service. In other words, when you purchase a VPS, you’re buying a machine in the cloud with all its specifications (CPU, RAM, Disk Memory). In reality, you aren’t probably buying an entire machine but rather a <a href="https://www.google.com/search?q=virtualized&amp;rlz=1C1ONGR_enAU1144AU1144&amp;oq=virtualized&amp;gs_lcrp=EgZjaHJvbWUyBggAEEUYOdIBCDE0MDhqMGo5qAIAsAIB&amp;sourceid=chrome&amp;ie=UTF-8#:~:text=a%20process%20that%20allows%20a%20computer%20to%20share%20its%20hardware%20resources%20with%20multiple%20digitally%20separated%20environments">virtualized</a> part of it (thus the name “virtual”).</p><p>Renting a VPS has many <strong>advantages.</strong> To start with, you get a computer on the cloud at a very cheap price compared to built-in cloud solutions. Moreover, you have full autonomy and customization of the software and applications you install on it.</p><p>On the other hand, owning a VPS makes you <strong>responsible</strong> for all the things running on it, i.e., securing access to the server, managing application runtime, running backups, etc. Some people might find this annoying, but I simply love it, and I’m learning a lot from the experience.</p><p>With many companies offering VPS at cheap prices: Digital Ocean, Hostinger, Hetzner, etc. I finally decided to opt for <a href="https://www.hetzner.com/cloud/"><strong>Hetzner</strong></a> as it has a well-known reputation and is currently one of the cheapest options. Right now I rent a CX22 instance with automated backups that cost me 5.38€/month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xDqzoPJRjpIW5PVN5rsOKA.png" /><figcaption>Hetzner VPS table of prices. Screenshot taken from <a href="https://www.hetzner.com/cloud/">hetzner.com</a></figcaption></figure><p>It may seem like very few resources to run a web server (just 2 vCPU and 4GB RAM) but if Pieter Levels can run all his web apps on a single 16 vCPU server while making 225k/m, then all my apps will do well too without spending 20$/m per app on hosting providers.</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//x.com/levelsio/status/1827305940418187669&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/22ca0b6ab4de467faae3a67c57773859/href">https://medium.com/media/22ca0b6ab4de467faae3a67c57773859/href</a></iframe><h4>Security Measures</h4><p>The first thing you must do when purchasing a VPS is run all the security checks to protect it from unwanted people.</p><p>These security checks go from hardening the SSH connection to configuring a firewall for incoming requests. I did all of this following tutorials and internet guides while documenting the process in case I need it in the future. Here is my checklist for securing the VPS:</p><p><a href="https://github.com/GabrielFurnieles/vps-security-checklist">GitHub - GabrielFurnieles/vps-security-checklist</a></p><h4>Deployment Stack: Docker + Traeffik</h4><blockquote><strong><em>Note</em>.</strong> All the instructions and steps I followed to configure and set up my VPS and the deployment stack come from <a href="https://youtu.be/F-9KWQByeU0?si=vG8LBnNNNtT7xhhx"><strong>this video by Dreams Of Code</strong></a>. I definitely recommend watching it!</blockquote><p>Whether you are running local tests on your machine or deploying any software to the cloud, <a href="https://www.docker.com/"><strong>Docker</strong></a><strong> </strong>is your best friend. I love how easy it is to set up any application with all its different services: Database, Redis cache, etc.</p><p>In my VPS I use Docker with <a href="https://docs.docker.com/compose/"><strong>Docker compose</strong></a> (a tool for defining and running multi-container applications) so all I need to do to deploy an application is:</p><ol><li>Build a docker image from my React app and push it to a container registry (I use GitHub)</li><li>Create a compose.yaml file to run a docker container using that image, upload it to my VPS and run it using docker compose up -d</li></ol><pre># Dockerfile<br># This is the file I used to build rgb2code docker image<br>FROM node:20-alpine AS builder<br>WORKDIR /app<br>COPY . .<br>RUN npm install<br>RUN npm run build<br><br>FROM nginx:1.25.4-alpine-slim AS prod<br>COPY --from=builder /app/dist /usr/share/nginx/html<br>COPY nginx.conf  /etc/nginx/conf.d<br>EXPOSE 3000<br>CMD [&quot;nginx&quot;, &quot;-g&quot;, &quot;daemon off;&quot;]</pre><pre># compose.yaml<br># This file describes all the services that rgb2code needs to run<br># Each service runs in a different docker container and is connected using the<br># same network to communicate with each other.<br>services:<br>  reverse-proxy:<br>    image: traefik:v3.1<br>    container_name: rgb2code-reverse-proxy<br>    restart: always<br>    command:<br>      - &quot;--providers.docker&quot;<br>      - &quot;--providers.docker.exposedbydefault=false&quot;<br>      - &quot;--entryPoints.web.address=:80&quot;<br>      - &quot;--entryPoints.websecure.address=:443&quot;<br>      - &quot;--certificatesresolvers.myresolver.acme.tlschallenge=true&quot;<br>      - &quot;--certificatesresolvers.myresolver.acme.email=test@gmail.com&quot;<br>      - &quot;--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json&quot;<br>      - &quot;--entrypoints.web.http.redirections.entrypoint.to=websecure&quot;<br>      - &quot;--entrypoints.web.http.redirections.entrypoint.scheme=https&quot;<br>    ports:<br>      - &quot;80:80&quot;       # HTTP<br>      - &quot;443:443&quot;     # HTTPS<br>    volumes:<br>      - letsencrypt:/letsencrypt<br>      - /var/run/docker.sock:/var/run/docker.sock         # Required for Traefik to auto-discover containers<br>    environment:<br>      - TZ=UTC<br>    networks:<br>      - traefik<br><br>  frontend-rgb2code:<br>    image: ghcr.io/gabrielfurnieles/rgb2code/frontend:latest<br>    labels:<br>      - &quot;traefik.enable=true&quot;<br>      - &quot;traefik.http.routers.rgb2code.rule=Host(`rgb2code.com`)&quot;<br>      - &quot;traefik.http.routers.rgb2code.entrypoints=websecure&quot;<br>      - &quot;traefik.http.routers.rgb2code.tls=true&quot;<br>      - &quot;traefik.http.routers.rgb2code.tls.certresolver=myresolver&quot;<br>    networks:<br>      - traefik<br>    deploy:<br>      mode: replicated<br>      replicas: 3<br>      resources:<br>        limits:<br>          cpus: &#39;0.50&#39;<br>          memory: 512M<br>    restart: always<br><br><br>volumes:<br>  letsencrypt:<br><br>networks:<br>  traefik:<br>    external: true</pre><p>As you can see, the compose.yaml file has two services:</p><ul><li>reverse-proxy: Introduces and configures the second main component of the deployment stack I use: <a href="https://traefik.io/traefik/"><strong>Traefik</strong></a> (more on that next)</li><li>frontend-rgb2code: Uses the image from my private GitHub repository to build the React app. The labels I added are for Traefik to recognize the containers and redirect the users. Finally, I specify Docker to run 3 replicas of the app (so Traefik can perform load balancing) and assign limited resources to each of them, just in case the containers don’t overload the server.</li></ul><p>As I advanced, Traefik is a service for reverse proxy; this means that all the users that try to access my server via a web domain (e.g. rgb2code.com) will arrive first at the Traefik service and then they’ll be redirected to the corresponding service within the server (in this case the frontend-rgb2code service)</p><p>Using Traefik has many advantages. For instance, I can have multiple domains pointing to the same server but each of them serving a different application. Moreover, Traefik integrates free SSL certificates and performs load balancing automatically.</p><h3>CI/CD with GitHub Actions</h3><p>As I explained in the previous section, my deployment workflow consists of two stages:</p><ol><li>Build a Docker image and pushing to a container registry</li><li>Run the containers on the server using docker compose up -d</li></ol><p>Every time I want to make an update to my app I need to run both steps, which quickly became annoying (especially when doing tests at the beginning)</p><p>Using GitHub actions I managed to automate the first step of building and pushing the image.However, I still need to work on automating the second of the steps:</p><pre>name: Deploy<br><br>on:<br>  push:<br>    branches:<br>      - main<br><br>permissions:<br>  contents: read<br>  packages: write<br><br>jobs:<br>  check-commit:<br>    runs-on: ubuntu-latest<br>    if: contains(github.event.head_commit.message, &#39;deploy&#39;)<br>    outputs:<br>      should_run: ${{ steps.check.outputs.should_run }}<br>    steps:<br>      - id: check<br>        run: echo &quot;should_run=true&quot; &gt;&gt; $GITHUB_OUTPUT<br><br>  commit-hash:<br>    needs: check-commit<br>    runs-on: ubuntu-latest<br>    outputs:<br>      commit_hash: ${{ steps.get_commit.outputs.commit_hash }}<br>    steps:<br>      - uses: actions/checkout@v4<br>      - name: Get commit hash<br>        id: get_commit<br>        run: echo &quot;commit_hash=$(git rev-parse HEAD)&quot; &gt;&gt; $GITHUB_OUTPUT<br><br>  build-and-test:<br>    needs: check-commit<br>    runs-on: ubuntu-latest<br>    steps:<br>      - uses: actions/checkout@v4<br>      - name: Setup Node.js<br>        uses: actions/setup-node@v4<br>        with:<br>          node-version: &#39;20.x&#39;<br>      - name: Install dependencies<br>        working-directory: ./frontend<br>        run: npm ci<br>      - name: Build<br>        working-directory: ./frontend<br>        run: npm run build<br><br>  build-and-push-image:<br>    needs:<br>      - build-and-test<br>      - commit-hash<br>    runs-on: ubuntu-latest<br>    steps:<br>      - name: Checkout repository<br>        uses: actions/checkout@v4<br><br>      - name: Set short git commit SHA<br>        id: vars<br>        run: |<br>          calculatedSha=$(git rev-parse --short ${{ github.sha }})<br>          echo &quot;COMMIT_SHORT_SHA=$calculatedSha&quot; &gt;&gt; $GITHUB_ENV<br><br>      - name: Set Docker image name<br>        run: |<br>          echo &quot;DOCKER_IMAGE_NAME=${GITHUB_REPOSITORY,,}&quot; &gt;&gt; $GITHUB_ENV<br>      <br>      - name: Log in to the Container registry<br>        uses: docker/login-action@f054a8b539a109f9f41c372932f1ae047eff08c9<br>        with:<br>          registry: ghcr.io<br>          username: ${{ github.actor }}<br>          password: ${{ secrets.GITHUB_TOKEN }}<br><br>      - name: Build and push Docker image<br>        uses: docker/build-push-action@ad44023a93711e3deb337508980b4b5e9bcdc5dc<br>        with:<br>          context: ./frontend<br>          push: true<br>          tags: |<br>            ghcr.io/${{ env.DOCKER_IMAGE_NAME }}/frontend:latest<br>            ghcr.io/${{ env.DOCKER_IMAGE_NAME }}/frontend:${{ needs.commit-hash.outputs.commit_hash }}</pre><h3>Future Projects</h3><p>What I like the most about this approach when deploying my projects is that I can use the same VPS to deploy new apps and solutions. I can share the resources and test my ideas for the same 5$; and if one of the projects starts attracting so much attention, I can vertically scale the VPS or isolate it on a dedicated VPS.</p><p>Most web apps do not need advanced technologies or expensive resources, they are just plain static files with one or two functionalities. This has the great advantage for us to move fast and ship as soon as possible without earning a degree in Kubernetes!</p><p>I still have a lot to learn and perfect my workflow, but sharing ideas on the internet has never been as easy as it is today. Good luck with your projects!🚀</p><h3>Pssst</h3><p>I’m also starting to share my journey and learnings in X (formerly Twitter) I’m planning on shipping more projects soon so stay tunned!🔥</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//x.com/gfurnieles_/status/1879841247894094286&amp;image=" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b4c9343bf61603f47fb53997d150253/href">https://medium.com/media/4b4c9343bf61603f47fb53997d150253/href</a></iframe><h3>References</h3><ul><li><a href="https://youtu.be/F-9KWQByeU0?si=whU6AzNYl3Nz_RVr"><strong>Setting up a production ready VPS is a lot easier than I thought.</strong> Dreams Of Code. Youtube 6 Sept 2024.</a></li></ul><p><em>Hi, I’m Gabriel Furnieles, a Mathematical Engineer specializing in Artificial Intelligence, Data Pipelines, and MLOps. I hope you enjoyed the article and found it helpful, if so, please consider following me </em><a href="https://medium.com/u/e77c10fd9715"><em>Gabriel Furnieles</em></a>,<em> and subscribing to my newsletter so stories will be sent directly to you 👇</em></p><p><a href="https://medium.com/@gabrielfurnieles">Gabriel Furnieles - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=879f0991358d" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/self-hosting-my-first-web-app-on-a-5-vps-879f0991358d">Self-Hosting My First Web App On a 5€ VPS</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[No, Poetry Is Not Going To Solve All Your Dependencies… But uv Does!]]></title>
            <link>https://levelup.gitconnected.com/no-poetry-is-not-going-to-solve-all-your-dependencies-but-uv-does-5b6a7c504e93?source=rss-e77c10fd9715------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b6a7c504e93</guid>
            <category><![CDATA[code]]></category>
            <category><![CDATA[poetry]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[package]]></category>
            <category><![CDATA[uv]]></category>
            <dc:creator><![CDATA[Gabriel Furnieles]]></dc:creator>
            <pubDate>Tue, 03 Dec 2024 18:43:33 GMT</pubDate>
            <atom:updated>2024-12-03T18:43:33.472Z</atom:updated>
            <content:encoded><![CDATA[<h4>Solving Transitive Explicit Dependencies in Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J6wkRzidRryQ8k1YMWocwA.jpeg" /><figcaption>Image generated with <a href="https://creator.nightcafe.studio/">NightCafe</a>.</figcaption></figure><p>A few days ago I was working on a series of <strong>private packages</strong> with dependencies between them forming a chain: package 1 was a dependency of package 2 and this was a dependency of package 3, so package 3 also depended on package 1 through package 2 (see Figure 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SR41SR9ZE00XXEunu8I8-w.png" /><figcaption>Figure 1. Dependency diagram. Image by the author</figcaption></figure><p>In my head, when I installed package 2 on package 3, so would the corresponding version of package 1 (this is the expected behavior with dependencies). However, I was surprised when I discovered <a href="https://python-poetry.org/">Poetry</a> could not resolve this simple dependency tree.</p><pre>Updating dependencies<br>Resolving dependencies... (0.4s)<br><br>Because no versions of p2 match &gt;0.1.0,&lt;0.2.0<br> and p2 (0.1.0) depends on p1 (&gt;=0.1.0,&lt;0.2.0), p2 (&gt;=0.1.0,&lt;0.2.0) requires p1 (&gt;=0.1.0,&lt;0.2.0).<br>So, because no versions of p1 match &gt;=0.1.0,&lt;0.2.0<br> and p3 depends on p2 (^0.1.0), version solving failed.</pre><h3>The Problem</h3><p>Poetry can resolve any chain of dependencies originating from the same index¹, such as <a href="https://pypi.org/">PyPI</a>. However, when additional indices are configured — such as a private PyPI server hosted by an organization or even a GitHub repository— Poetry encounters issues resolving transitive dependencies (i.e. dependencies indirectly required by other dependencies)</p><p>¹A Python index is a local storage for Python libraries and tools to be published, shared, and retrieved.</p><h4>How Poetry Manages additional Indices (sources)</h4><p>To include an additional index, Poetry lets you define sources (check <a href="https://python-poetry.org/docs/repositories/#package-sources:~:text=repository%20foo%2Dpub-,Package%20Sources,-By%20default%2C%20Poetry">documentation</a>📄)</p><pre># Defining a new source in Poetry named foo using a URL<br>poetry source add foo https://foo.bar/simple/</pre><p>Each source is assigned a <strong>priority</strong> that allows Poetry to resolve dependencies. In the latest version 1.8 of Poetry there are three types of priorities:</p><ul><li><em>primary</em></li><li><em>supplemental</em></li><li><em>explicit</em></li></ul><p>When installing a new dependency, Poetry will first look into all the <em>primary</em> sources and if none of these yield a compatible version, it then checks the <em>supplemental</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YpMDMIA84VsViRYpRSgZBw.png" /><figcaption>Figure 2. Poetry search diagram. Image by the author.</figcaption></figure><p>There is also a major downside with these priorities: <strong>search in primary and supplemental sources is performed</strong> <strong>independently of whether the package is found in one or more sources. </strong>This means conflicts will arise if the same package exists in multiple sources with the same priority².</p><p>The third and most recommended option is to set <em>explicit</em> priority. With explicit, Poetry will only look at the source (and just one source) when a package configuration explicitly indicates it. This is always done by default when you add a package with the --source flag: poetry add &lt;PACKAGE&gt; --source &lt;SOURCE&gt;</p><pre># pyproject.toml<br>...<br><br>[tool.poetry.dependencies]<br>package1 = { version = &quot;^0.1.0&quot;, source = &quot;private-pypi&quot; }</pre><p>Explicit is the most secure and optimal option — since Poetry will only search in a specific source instead of checking several the search is faster and there is no risk of dependency conflicts—. But when dealing with transitive dependencies it results in error.</p><ul><li><a href="https://github.com/python-poetry/poetry/issues/7981">Transitive private dependencies from `explicit` sources are not pulled · Issue #7981 · python-poetry/poetry</a></li><li><a href="https://github.com/orgs/python-poetry/discussions/7988">Poetry can not resolve dependencies when `priority = &quot;explicit&quot;` is set for a private package registry · python-poetry · Discussion #7988</a></li></ul><p>According to the Poetry team, this is what explicit priority is expected to do. However, it doesn’t seem correct, and using <em>supplemental</em> priority is not recommended at all (there is still the risk of <strong>dependency confusion attacks</strong>³).</p><p>²Suppose you want to install a package named numpy from another index with the same priority as PyPI. Running poetry add numpy will then raise an error because Poetry won’t know which of the two packages to install (PyPI or the other repository)</p><p>³An attacker can introduce malware by publishing a package in PyPI with the same name as the private package you want to install (which Poetry will download first because PyPI is by default a primary source*).<br>*If configured the other way around (PyPI as supplemental and the private repo as primary) for most packages Poetry will have to search the private repo first which is very suboptimal.</p><h4>Testing on my own Pypi Server</h4><p>To test this behavior, I decided to set up my own local PyPI server using Docker (<a href="https://testdriven.io/blog/private-pypi/">check this helpful tutorial</a>). If you want to try it, you can clone my GitHub repo and follow the README instructions to set up everything.</p><p><a href="https://github.com/GabrielFurnieles/private-pypi-server">GitHub - GabrielFurnieles/private-pypi-server: Private PyPI server</a></p><h3>Poetry’s workaround</h3><p>According to the documentation and as explained in <a href="https://stackoverflow.com/questions/76542312/poetry-can-we-use-explicit-priority-for-chained-dependencies/78666390#78666390">this thread</a> from StackOverflow, the only solution to this problem in Poetry is to manually add the transitive dependencies with a wildcard * version — the wildcard tells Poetry to use any compatible version, which by default will try to be the latest one—.</p><pre># Package 3<br>poetry add package1@* --source=private-pypi --group explicit</pre><p>However, with more complex dependency trees this becomes very annoying and is just a workaround.</p><h3>uv To The Rescue</h3><p><a href="https://docs.astral.sh/uv/">uv</a> is a relatively new Python package manager developed in Rust by Astral.sh.</p><p>Among its advantages is an amazing speed in resolving and installing dependencies from Python indices, installing and managing Python itself, running standalone Python scripts (even if Python is not installed on the system), and creating and managing Python environments.</p><p>But what is most important, <strong>uv can solve chained explicit dependencies</strong>!🥳</p><h4>Brief uv guide for managing indices</h4><p>The uv CLI is very intuitive and similar to the one of Poetry. To create a uv project run:</p><pre>uv init</pre><p>uv is still in an early phase and its team is working on many features. At the moment of this post, there isn’t a command to add new indices (sources) so you need to manually add them in the pyproject.toml file:</p><pre>[[tool.uv.index]]<br>name = &quot;private-pypi&quot;<br>url = &quot;http://localhost:8080&quot;  # My local PyPI server is running on port 8080<br>explicit = true</pre><p>To set up the index credentials use <a href="https://docs.astral.sh/uv/configuration/indexes/">environment variables</a>:</p><pre># In case you want to preserve these credentials between different console <br># sessions, add these lines at the end of the file ~/.bashrc<br>export UV_INDEX_PRIVATE_PYPI_USERNAME=pypi-username<br>export UV_INDEX_PRIVATE_PYPI_PASSWORD=pypi-password</pre><p>Then, when adding a package from the new source it is also necessary to explicit again the index URL:</p><pre>uv add package1 --index private-pypi=http://localhost:8080</pre><p>Check that when installing package 2 in package 3 so does package 1 without raising any error.</p><p>To test the package you can create a Python environment:</p><pre>uv venv  # Creates environment<br>uv sync  # Synchronises dependencies (equiv. to poetry install)<br>...      # Run any script</pre><p>Finally, in order to build and publish a package you need to run two separate commands and again use the URL and credentials from the private index:</p><pre>uv build<br>uv publish --publish-url http://localhost:8080 --username pypi-user --password pypi-password</pre><blockquote><strong>Keep in mind </strong>that uv is updating very frequently and this brief guide might change in the future. I am using uv 0.5.2.</blockquote><h3>Conclusion</h3><p>Python package managers are essential for handling project dependencies and environments. When setting an additional index it is very important to consider which priority to assign it (usually explicit) While <strong>Poetry</strong> is a solid choice for most cases, it can become cumbersome when dealing with complex setups involving private and other indices, which is an important limitation to consider.</p><p>In contrast, <strong>uv</strong> addresses these challenges and offers several compelling features. Although, it might not be a safe option for production environments yet — as it is still under development and many things will change shortly —, definitely is a project to keep an eye close and it is a matter of time before it becomes the <em>de facto</em> option for building and sharing Python code.</p><p><em>Hi, I’m Gabriel Furnieles, a Mathematical Engineer specializing in Artificial Intelligence, Data Pipelines, and MLOps. I hope you enjoyed the article and found it helpful, if so, please consider following me </em><a href="https://medium.com/u/e77c10fd9715"><em>Gabriel Furnieles</em></a>,<em> and subscribing to my newsletter so stories will be sent directly to you 👇</em></p><p><a href="https://gabrielfurnieles.medium.com/">Gabriel Furnieles - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b6a7c504e93" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/no-poetry-is-not-going-to-solve-all-your-dependencies-but-uv-does-5b6a7c504e93">No, Poetry Is Not Going To Solve All Your Dependencies… But uv Does!</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Validation with Pandera in Python]]></title>
            <link>https://medium.com/data-science/data-validation-with-pandera-in-python-f07b0f845040?source=rss-e77c10fd9715------2</link>
            <guid isPermaLink="false">https://medium.com/p/f07b0f845040</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-validation]]></category>
            <category><![CDATA[pipeline]]></category>
            <dc:creator><![CDATA[Gabriel Furnieles]]></dc:creator>
            <pubDate>Mon, 18 Nov 2024 17:53:59 GMT</pubDate>
            <atom:updated>2024-11-18T17:53:59.340Z</atom:updated>
            <content:encoded><![CDATA[<h4>Validating your Dataframes for Production ML Pipelines</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fnEG1guS0sYHTxc4guHNag.jpeg" /><figcaption>Image generated with <a href="https://creator.nightcafe.studio/">NightCafe</a>.</figcaption></figure><p>Data validation is a crucial step for production applications. You need to ensure the data you are ingesting is compatible with your pipeline and that unexpected values aren’t present. Moreover, validating the data is a security measure that prevents any corrupted or inaccurate information from being further processed, raising a flag on the first steps.</p><p>Python already counts with a great OS project for this task called <a href="https://docs.pydantic.dev/latest/">Pydantic</a>. However, when dealing with large dataframe-like objects such as in Machine Learning, <a href="https://pandera.readthedocs.io/en/stable/index.html">Pandera</a> is a much faster and scalable way of validating data (check <a href="https://www.union.ai/blog-post/pandera-0-17-adds-support-for-pydantic-v2">this article</a> with public notebooks).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/0*JHg4ZVNkQ-WYyQRH.png" /><figcaption>Performance comparison between pandera and row-wise validation with Pydantic for different-sized pandas.DataFrame objects. Image from <a href="https://www.union.ai/blog-post/pandera-0-17-adds-support-for-pydantic-v2#:~:text=%22records%22))-,Benchmarking%20Pandera%E2%80%99s%20row%2Dwise%20validation%20with%20Pydantic,-Because%20Pandera%20validates">source</a>.</figcaption></figure><p>In addition, Pandera offers support for a great variety of dataframe libraries like pandas, polars, dask, modin, and pyspark.pandas. For more information on these refer to <a href="https://pandera.readthedocs.io/en/stable/index.html">Pandera’s docs📄</a>.</p><blockquote><strong>Disclaimer.</strong> Pandera is an open-source project licensed under the MIT License. I have no affiliation with the Pandera team or Union.ai. This post has no commercial interest.</blockquote><h3>Validating data with Pandera</h3><p>Pandera has two ways of defining validators: <strong>Schemas</strong> and <strong>Models</strong>. I will focus on the second one because of its similarity with Pydantic models and the cleanness of the code.</p><p>To define a Pandera model create a child class that inherits from DataframeModel and start declaring the columns and dtypes that the dataframe must have:</p><pre>import pandera as pa<br><br><br>class UserModel(pa.DataFrameModel):<br>    id: int<br>    username: str<br>    email: str<br>    is_active: bool<br>    membership: str<br>    creation_date: pd.DatetimeTZDtype</pre><pre># Use<br>df = pd.DataFrame(...)<br>UserModel.validate(df) # &lt;- If invalidad raises SchemaError</pre><p>Note that to define the user’s creation timestamp I used Pandas native date type instead of others like datetime.datetime. Pandera only supports built-in Python, NumPy, and Pandas data types. You can also create <a href="https://pandera.readthedocs.io/en/stable/dtypes.html#logical-data-types">custom data types</a>, but this is an advanced topic and rarely necessary in most cases.</p><h4>Validating column properties</h4><p>With Pandera, you can also validate other column properties in addition to the type of data:</p><pre>class UserModel(pa.DataFrameModel):<br>    id: int  = pa.Field(unique=True, ge=0)<br>    username: str = pa.Field(str_matches=r&quot;^[a-zA-Z0-9_]+$&quot;)<br>    email: str = pa.Field(str_matches=r&quot;^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$&quot;)<br>    is_active: bool<br>    membership: str = pa.Field(isin=[&quot;premium&quot;, &quot;free&quot;])<br>    creation_date: pd.DatetimeTZDtype = pa.Field(dtype_kwargs={&quot;unit&quot;: &quot;ns&quot;, &quot;tz&quot;: &quot;UTC&quot;})</pre><p>Here I am using pandera’s Field just like pydantics’.</p><ul><li>First, I am specifying that the id column must not contain duplicated values and these have to be greater or equal to 0.</li><li>In username and email I’m checking using regex expressions if strings are valid. User names must only contain alphanumeric characters and underscore, while emails can also contain dashes and dots but always follow the pattern “smth@smth.smth”.</li><li>membership can only take a value from the list. A better approach is using a StrEnum to define the valid values instead of hardcoding them.</li><li>Finally, creation_date must be in nanosecond units and UTC timezone. This line can be cleaner using Annotated from typing library creation_date: Annotated[pd.DatetimeTZDtype, &quot;ns&quot;, &quot;UTC&quot;]</li></ul><p>Check out <a href="https://pandera.readthedocs.io/en/stable/reference/generated/pandera.api.dataframe.model_components.Field.html#pandera.api.dataframe.model_components.Field">the docs</a> to read all Field options😋</p><h4>Custom Validations</h4><p>Sometimes it is necessary to add your own custom validations. Pandera allows you to inject <a href="https://pandera.readthedocs.io/en/stable/dataframe_models.html#dataframe-checks:~:text=as%20class%20methods.-,Column/Index%20checks,-%C2%B6">column/index checks</a> (custom checks of single columns) and <a href="https://pandera.readthedocs.io/en/stable/dataframe_models.html#dataframe-checks:~:text=1%3A%20%3CCheck%20check_means%3E-,DataFrame%20Checks,-%C2%B6">dataframe checks</a> (checks between several columns).</p><pre>import pandera as pa<br>from pandera.typing import Series<br><br><br>class UserModel(pa.DataFrameModel):<br>    id: int = pa.Field(unique=True, ge=0)<br>    username: str = pa.Field(str_matches=r&quot;^[a-zA-Z0-9_]+$&quot;)<br>    email: str = pa.Field(<br>        str_matches=r&quot;^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$&quot;<br>    )<br>    is_active: bool<br>    membership: str = pa.Field(isin=[&quot;premium&quot;, &quot;free&quot;])<br>    creation_date: Annotated[pd.DatetimeTZDtype, &quot;ns&quot;, &quot;UTC&quot;]<br><br>    # column/index checks<br>    @pa.check(&quot;username&quot;, name=&quot;username_length&quot;)<br>    def username_length(cls, x: Series[str]) -&gt; Series[bool]:<br>        &quot;&quot;&quot;<br>        Check username length is between 1 and 20 characters<br>        &quot;&quot;&quot;<br>        return x.str.len().between(1, 20)<br><br>    @pa.check(&quot;creation_date&quot;, name=&quot;min_creation_date&quot;)<br>    def min_creation_date(cls, x: Series[pd.DatetimeTZDtype]) -&gt; Series[bool]:<br>        &quot;&quot;&quot;<br>        Check creation date is after 2000-01-01<br>        &quot;&quot;&quot;<br>        return x &gt;= dt.datetime(2000, 1, 1, tzinfo=dt.timezone.utc)<br>    <br>    # dataframe check<br>    @pa.dataframe_check<br>    def membership_is_valid(<br>        cls, df: pd.DataFrame, name=&quot;membership_is_valid&quot;<br>    ) -&gt; Series[bool]:<br>        &quot;&quot;&quot;<br>        Check account age for free memebers is &lt;= 30 days<br>        &quot;&quot;&quot;<br>        current_time = dt.datetime.now(dt.timezone.utc)<br>        thirty_days = dt.timedelta(days=30)<br><br>        return (df[&quot;membership&quot;] == &quot;premium&quot;) | (<br>            (df[&quot;membership&quot;] == &quot;free&quot;)<br>            &amp; ((current_time - df[&quot;creation_date&quot;]) &lt;= thirty_days)<br>        )</pre><p>Keep in mind that you are working with entire column objects (Series) so that operations in checks should be vectorized for better performance.</p><h4>Other Configurations</h4><p><strong>Aliases<br></strong>When column names can’t be declared as Python variables due to the language syntax, Pandera allows setting an alias for the column validator to match the dataframe.</p><pre>class MyModel(pa.DataFrameModel):<br>    alias_column: int = pa.Field(..., alias=&quot;Alias Column&quot;)<br>    ...</pre><p><strong>Strict and Coerce<br></strong>When the strict option is set to true, it forces the validated dataframe to only contain the columns defined in the Pandera DataFrameModel. On the other hand, when the coerce option is activated, Pandera will try to cast the column data to match the model’s dtype.</p><pre>class MyModel(pa.DataFrameModel):<br>    ...<br>    <br>    class Config:<br>        strict = True # defaul: False<br>        coerce = True # default: False</pre><p>The coerce option can be set at the Field level too using pa.Field(..., coerce=True)</p><p><strong>Lazy validation<br></strong>By default, pandera raises an error whenever a validation check isn’t passed. This can be annoying because it only displays the first validation error encountered, and prevents the rest of the data from being checked.</p><p>In some cases, it is better to let the whole dataframe validate and collect all errors in one run, rather than fixing them one by one and waiting for the validation to run again. The first is what lazy validation does.</p><pre>df = pd.DataFrame(...)<br>Mymodel.validate(df, lazy_validation=True)</pre><h3>An ML Production Pipeline with Data Validation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tcTDRG7zmsbKNFbxpBQHTg.jpeg" /><figcaption>Image generated with <a href="https://creator.nightcafe.studio/">NightCafe</a>.</figcaption></figure><p>Because the majority of ML Pipelines are trained in Python with tabular data encoded into dataframe structures, <strong>Pandera</strong> is a great and powerful tool to validate their Inputs and Outputs.</p><pre># pipeline.py<br><br>class MLPipeline:<br>    &quot;&quot;&quot;General ML Pipeline&quot;&quot;&quot;<br>    def __init__(self, model_id: str):<br>        self.model_id = model_id<br>    <br>    def load_model(self) -&gt; None:<br>        ...<br>    <br>    def transform_data(self, df: pd.DataFrame) -&gt; pd.DataFrame:<br>        ...   # &lt;- Potential invalid data error<br>        return df_transform<br><br>    def predict(self, df: pd.DataFrame) -&gt; pd.DataFrame:<br>        self.load_model()<br>        df_transform = self.transform(df)<br>        df[&#39;score&#39;] = self.model.predict(df_transform)  # &lt;- Potential invalid data error<br>        return df</pre><p>We want to avoid the model raising an error due to invalid data. That would mean that we’ve done all the work of loading the model into memory and processing the raw data for nothing, wasting resources and preventing the rest of the data points from being evaluated.</p><p>Similarly, if the model’s output has an incorrect structure our postprocessing pipeline (uploading results to DB, returning results by RESTful API, etc.) will fail.</p><p>After defining the validation models using Pandera, we can leverage its <a href="https://pandera.readthedocs.io/en/stable/decorators.html#:~:text=Auto%20color%20theme-,Decorators%20for%20Pipeline%20Integration,-%C2%B6">decorators for pipeline integration</a> to perform I/O validation.</p><pre># models.py<br>import pandera as pa<br><br>class InputModel(pa.DataFrameModel):<br>    ...<br><br>class PredictorModel(pa.DataFrameModel):<br>    ...<br><br># OutputModel inherits all InputModel validation fields<br># and also includes the score<br>class OutputModel(InputModel):<br>    score: float = pa.Field(ge=0, le=1) # assuming model returns probab.</pre><pre># pipeline.py<br>import pandera as pa<br>from .models import InputModel, PredictorModel, OutputModel<br><br><br>class MLPipeline:<br>    &quot;&quot;&quot;General ML Pipeline&quot;&quot;&quot;<br>    def __init__(self, model_id: str):<br>        self.model_id = model_id<br>    <br>    def load_model(self) -&gt; None:<br>        ...<br>    <br>    @pa.check_io(df=InputModel.to_schema(), out=PredictorModel.to_schema(), lazy=True)<br>    def transform_data(self, df: pd.DataFrame) -&gt; pd.DataFrame:<br>        ...<br>        return df_transform<br><br>    @pa.check_output(OutputModel.to_schema(), lazy=True)<br>    def predict(self, df: pd.DataFrame) -&gt; pd.DataFrame:<br>        self.load_model()<br>        df_transform = self.transform(df)  <br>        df[&#39;score&#39;] = self.model.predict(df_transform)<br>        return df</pre><p>Because we are generating an intermediate dataframe object df_transform in the ML Pipeline, it is a good practice to validate it too to prevent errors. The <em>predict</em> method input is not validated as it is already done by <em>transform_data</em>.</p><h4>Handling invalid rows</h4><p>We don’t want our pipeline to break just because some data points have incorrect data. In case of a validation error, the strategy should be to set aside the problematic data points and continue running the pipeline with the rest of the data. The pipeline cannot stop!🔥</p><p>Pandera models have the option to automatically remove all invalid rows:</p><pre>class MyModel(pa.DataFrameModel):<br>    ...<br><br>    class Config:<br>        drop_invalid_rows = True</pre><p>However, dropping all invalid rows without logging them can be dangerous. You need to know why those data points were invalid so that later you can communicate to the client or to the data engineer what was the cause of the error.</p><p>That is why instead of using pandera decorators I rather create my own validation helper functions:</p><pre>from typing import Tuple<br>import logging<br><br>logging.basicConfig(level=logging.INFO)<br>logger = logging.getLogger(__name__)<br><br><br>def log_pandera_errors(exc: pa.errors.SchemaErrors) -&gt; None:<br>    &quot;&quot;&quot;<br>    Logs all errors from a SchemaErrors exception.<br>    &quot;&quot;&quot;<br>    for err_type, categories in exc.message.items():<br>        for _, errors in categories.items():<br>            for err in errors:<br>                logger.error(f&quot;{err_type} ERROR: {err[&#39;column&#39;]}. {err[&#39;error&#39;]}&quot;)<br><br><br>def handle_invalid(<br>    df: pd.DataFrame, exc: pa.errors.SchemaErrors<br>) -&gt; Tuple[pd.DataFrame, pd.DataFrame]:<br>    &quot;&quot;&quot;<br>    Handles invalid data in a DataFrame based on a SchemaErrors exception.<br>    &quot;&quot;&quot;<br>    log_pandera_errors(exc)<br><br>    df_failure = exc.failure_cases<br><br>    # Check for errors that cannot be resolved<br>    # i.e. they aren&#39;t associated with a specific row index<br>    nan_indices = df_failure[&quot;index&quot;].isna()<br>    if nan_indices.any():<br>        error_msg = &quot;\n&quot;.join(<br>            f&quot;    - Column: {row[&#39;column&#39;]}, check: {row[&#39;check&#39;]}, &quot;<br>            f&quot;failure_case: {row[&#39;failure_case&#39;]}&quot;<br>            for row in df_failure[nan_indices].to_dict(&quot;records&quot;)<br>        )<br>        raise ValueError(<br>            f&quot;Schema validation failed with no possibility of continuing:\n{error_msg}\n&quot;<br>            &quot;The pipeline cannot continue 😢. Resolve before rerunning&quot;<br>        )<br><br>    invalid_idcs = df.index.isin(df_failure[&quot;index&quot;].unique())<br>    df_invalid = format_invalid_df(df.loc[invalid_idcs, :], exc)<br>    df_valid = df.iloc[~invalid_idcs]<br><br>    return df_valid, df_invalid<br><br><br>def validate(<br>    df: pd.DataFrame, model: pa.DataFrameModel<br>) -&gt; Tuple[pd.DataFrame, pd.DataFrame]:<br>    &quot;&quot;&quot;<br>    Validates a DataFrame against a DataFrameModel and handles errors.<br>    &quot;&quot;&quot;<br>    try:<br>        return model.validate(df, lazy=True), pd.DataFrame()<br>    except pa.errors.SchemaErrors as ex:<br>        return handle_invalid(df, ex)</pre><p>Output forcing some errors and removing column id:</p><pre># Error output<br>ERROR:__main__:SCHEMA ERROR: UserModel. column &#39;id&#39; not in dataframe. Columns in dataframe: [&#39;username&#39;, &#39;email&#39;, &#39;membership&#39;, &#39;is_active&#39;, &#39;creation_date&#39;]<br>ERROR:__main__:DATA ERROR: username. Column &#39;username&#39; failed element-wise validator number 0: str_matches(&#39;^[a-zA-Z0-9_]+$&#39;) failure cases: b%09<br>ERROR:__main__:DATA ERROR: email. Column &#39;email&#39; failed element-wise validator number 0: str_matches(&#39;^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$&#39;) failure cases: ef.com<br>ERROR:__main__:DATA ERROR: UserModel. DataFrameSchema &#39;UserModel&#39; failed element-wise validator number 0: &lt;Check membership_is_valid&gt; failure cases: c, ef.com, free, True, 2000-12-31 00:00:00+00:00<br><br>ValueError: Schema validation failed with no possibility of continuing:<br>    - Column: UserModel, check: column_in_dataframe, failure_case: id<br>The pipeline cannot continue 😢. Resolve before rerunning</pre><p>In case of an unresolvable error that involves an entire column, the pipeline cannot continue.</p><h4>Testing</h4><p>Last but not least, Pandera models and schemas also incorporate a method for generating sample data according to their definition. You will need to install <a href="https://hypothesis.readthedocs.io/en/latest/">hypothesis</a> library to use it.</p><p>However, after testing it with some examples I do not recommend it. As soon as you start adding a few constraints, it takes too long to generate the synthetic data and most of the time it isn’t varied (the generated data do not cover the entire restriction space and repeats itself) The best alternative I found is to add data generators for each model you want to test — after all, there aren’t so many data frames to validate in a pipeline either — .</p><pre>class UserModel(pa.DataFrameModel):<br>    ...<br><br>    def sample(size: int = 10) -&gt; pd.DataFrame:<br>        &quot;&quot;&quot;Added method to generate valid test data manually&quot;&quot;&quot;<br>        current_time = dt.datetime.now(dt.timezone.utc)<br>        return pd.DataFrame(<br>            {<br>                &quot;id&quot;: range(size),<br>                &quot;username&quot;: [f&quot;user_{i}&quot; for i in range(size)],<br>                &quot;email&quot;: [f&quot;user_{i}@example.com&quot; for i in range(size)],<br>                &quot;is_active&quot;: [True] * size,<br>                &quot;membership&quot;: [&quot;premium&quot;] * size,  # All premium to pass checks<br>                &quot;creation_date&quot;: [current_time] * size,<br>            }<br>        )</pre><h3>Conclusion</h3><p>Data validation is vital for every data processing pipeline and especially in Machine Learning. Pandera simplifies a lot of this work by providing a flexible, and efficient model-based approach to validating data in dataframes.</p><p>With Pandera, you can define model classes that enforce column types, ranges, and even complex conditional constraints. This makes it easy to catch data quality issues early in the pipeline, ensuring that the data conforms to expected standards before it reaches the next steps.</p><p>By integrating Pandera into an ML pipeline, you can create robust data checks that help prevent errors and improve the reliability of model outputs.</p><p>Final pandera.DataFrameModel used in the tests:</p><pre>import pandas as pd<br>import pandera as pa<br>from pandera.typing import Series<br>from typing import Annotated<br>import datetime as dt<br><br><br>class UserModel(pa.DataFrameModel):<br>    id: int = pa.Field(unique=True, ge=0, coerce=False)<br>    username: str = pa.Field(str_matches=r&quot;^[a-zA-Z0-9_]+$&quot;)<br>    email: str = pa.Field(<br>        str_matches=r&quot;^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$&quot;<br>    )<br>    is_active: bool<br>    membership: str = pa.Field(isin=[&quot;premium&quot;, &quot;free&quot;])<br>    creation_date: Annotated[pd.DatetimeTZDtype, &quot;ns&quot;, &quot;UTC&quot;]<br><br>    @pa.check(&quot;username&quot;, name=&quot;username_length&quot;)<br>    def username_length(cls, x: Series[str]) -&gt; Series[bool]:<br>        &quot;&quot;&quot;<br>        Check username length is between 1 and 20 characters<br>        &quot;&quot;&quot;<br>        return x.str.len().between(1, 20)<br><br>    @pa.check(&quot;creation_date&quot;, name=&quot;min_creation_date&quot;)<br>    def min_creation_date(cls, x: Series[pd.DatetimeTZDtype]) -&gt; Series[bool]:<br>        &quot;&quot;&quot;<br>        Check creation date is after 2000-01-01<br>        &quot;&quot;&quot;<br>        return x &gt;= dt.datetime(2000, 1, 1, tzinfo=dt.timezone.utc)<br><br>    @pa.dataframe_check<br>    def membership_is_valid(<br>        cls, df: pd.DataFrame, name=&quot;membership_is_valid&quot;<br>    ) -&gt; Series[bool]:<br>        &quot;&quot;&quot;<br>        Check account age for free memebers is &lt;= 30 days<br>        &quot;&quot;&quot;<br>        current_time = dt.datetime.now(dt.timezone.utc)<br>        thirty_days = dt.timedelta(days=30)<br><br>        return (df[&quot;membership&quot;] == &quot;premium&quot;) | (<br>            (df[&quot;membership&quot;] == &quot;free&quot;)<br>            &amp; ((current_time - df[&quot;creation_date&quot;]) &lt;= thirty_days)<br>        )<br><br>    class Config:<br>        strict = True<br>        coerce = True<br><br>    def sample(size: int = 10) -&gt; pd.DataFrame:<br>        &quot;&quot;&quot;Added method to generate valid test data manually&quot;&quot;&quot;<br>        current_time = dt.datetime.now(dt.timezone.utc)<br>        return pd.DataFrame(<br>            {<br>                &quot;id&quot;: range(size),<br>                &quot;username&quot;: [f&quot;user_{i}&quot; for i in range(size)],<br>                &quot;email&quot;: [f&quot;user_{i}@example.com&quot; for i in range(size)],<br>                &quot;is_active&quot;: [True] * size,<br>                &quot;membership&quot;: [&quot;premium&quot;]<br>                * size,  # All premium to avoid date restrictions<br>                &quot;creation_date&quot;: [current_time] * size,<br>            }<br>        )</pre><p><em>Hi, I’m Gabriel Furnieles, a Mathematical Engineer specializing in Artificial Intelligence, Data Pipelines, and MLOps. I hope you enjoyed the article and found it helpful, if so, please consider following me </em><a href="https://medium.com/u/e77c10fd9715"><em>Gabriel Furnieles</em></a>,<em> and subscribing to my newsletter so stories will be sent directly to you 👇</em></p><p><a href="https://medium.com/@gabrielfurnieles">Gabriel Furnieles - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f07b0f845040" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/data-validation-with-pandera-in-python-f07b0f845040">Data Validation with Pandera in Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crack The Algorithm: An AI driven Strategy to Boost Visibility and Leads in Real Estate Portals]]></title>
            <link>https://medium.com/menhirtech/crack-the-algorithm-an-ai-driven-strategy-to-boost-visibility-and-leads-in-real-estate-portals-b8cf1ff45b5d?source=rss-e77c10fd9715------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8cf1ff45b5d</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[real-estate]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[case-study]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[Gabriel Furnieles]]></dc:creator>
            <pubDate>Wed, 28 Aug 2024 12:08:59 GMT</pubDate>
            <atom:updated>2024-08-28T12:08:59.602Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Menhir Technologies Case Study for Real Estate Companies.</h4><p>Some companies promote their services and products on marketplaces where they compete for attention against other professionals. These online markets host many advertisers and the competition can get really fearce. If you are not in the top positions of the feed, you barely get any attention and visibility.</p><p>For real estate companies, this challenge is particularly daunting, as the digital marketplace teems with listings and agencies competing for the attention of prospective buyers. Securing a prime position in search results can significantly enhance lead generation and client acquisition, which is why it is critical to understand and leverage the intricate algorithms that govern these rankings.</p><p>At Menhir we’ve been working on strategies based on AI algorithms and data analysis to improve the visibility of our clients in one of the main real estate web portals in Spain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cIk1TekBurW37Ucx_hx0mg.png" /><figcaption>Image generated by AI. Edited by the author.</figcaption></figure><h3>How?</h3><p><strong>Long story short:</strong> We trained a Machine Learning model to predict and mimic the actual marketplace feed and cracked it using our xAI tools developed at Menhir to understand and explain the model’s behaviour.</p><p>To crack the code of the positioning algorithm, we divided our approach into three key phases common to any machine learning project:</p><ol><li><strong>Collecting the data.</strong></li><li><strong>Training a model to crack the code.</strong></li><li><strong>Ensuring explainability.</strong></li></ol><p>Each phase played a crucial role in developing a robust strategy to enhance our clients’ online presence in the most important real state portals in Spain.</p><h3>Collecting the data</h3><p>As the first phase of any project involving Artificial Intelligence, obtaining a good quality dataset is the most important task.</p><blockquote>Models are only as good as the quality of the data they are trained on. Or as they say: Garbage In, Garbage Out.</blockquote><p>Most marketplaces are public and open to internet users. However, they do not usually provide their own data openly on a large scale through APIs or other services (or they have too elevated costs). This is not the first time we’ve faced a similar challenge at Menhir. Using automated <strong>web data mining techniques</strong>, we collect information from public websites quickly and efficiently without the need for manual intervention.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSLuHl6xJjhrSOgcg_JX_A.png" /><figcaption>We use scheduled cronjobs to periodically collect the latest updated information from various web sources and store it in our database for further processing and analysis.</figcaption></figure><p>By automating the data collection pipeline, we can consistently gather up-to-date data with minimal manual intervention, allowing the team to focus on analysis and strategy development. This robust infrastructure is crucial for delivering high-quality, actionable insights to our clients, setting the stage for the subsequent phases of model training and explainability.</p><h3>Training a model to Crack the Code</h3><p>With a solid dataset in hand, we train a model to decipher the marketplace’s ranking algorithm. We use advanced Machine Learning algorithms to process the data and uncover the hidden rules governing search rankings. The goal is to create a predictive model that simulates the algorithm’s behavior, being able to infer the positioning of an asset given their publication variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oo-KFXFEXC7iXJ6fYDFjUA.png" /><figcaption>The model’s objective is to predict the probability that the asset will be in one or another position in the list.</figcaption></figure><p>Having an accurate model allows us to test different values of the predictor variables and strategies, enabling us to identify the most influential factors and find the optimal thresholds that have the greatest impact on product visibility and positioning. This is further addressed in the model’s explainablity which is the third phase.</p><h3><strong>Ensuring explainability</strong></h3><p>Understanding how our model makes its predictions is just as important as the predictions themselves. At Menhir we employ explainable AI techniques developed internally (xAI Menhir) to break down the model’s decision-making process, ensuring that we can clearly explain why certain listings performed better than others. This transparency is crucial for two reasons:</p><ol><li>It helps us refine our strategies more effectively.</li><li>It allows us to communicate our findings and recommendations to our clients in a straightforward, actionable way.</li></ol><blockquote>The explainability of artificial intelligence models is a promising and intriguing area of AI research. It aims to make complex algorithms more transparent and understandable, enabling users to trust and effectively utilize AI systems in critical decision-making processes.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BldkaWDyJUATmNJUz9ZsIQ.png" /><figcaption>Sample image from Menhir’s <a href="https://www.menhir.ai/industry/real-estate">website</a>.</figcaption></figure><p>By opening and studying the model’s predictions, we can discover which variables most affect the positioning. This way we can “<em>hack</em>” the algorithm and generate optimized ads that quickly rise to the highest positions in the ranking. In other words, we use the model’s predictions to generate <strong>actionable insights that drive profits.</strong></p><h3>Repeat!</h3><p>Marketplaces are constantly updating their products, and along with them, their internal positioning algorithms. Monitoring and tracking the ranking behavior allows us to always stay ahead of the market and anticipate changes, while repeating the analysis ensures that the strategies remain effective. By continuously gathering data and retraining models, we can adapt to new trends and maintain optimal visibility at any time.</p><p>Furthermore, keeping track of marketplaces’ rankings offers numerous other benefits. It enables competitor monitoring and facilitates the identification of seasonal trends within time series data, which allow marketing campaigns to be further optimized.</p><h3>Conclusion</h3><p>The use of AI-based strategies for real estate portals has shown great potential that can be extrapolated to many other areas such as e-commerce, hospitality or digital marketing.</p><p>Ranking algorithms can be very complex and sofisticated, but with the right set of features and trained models it is certainly possible to approximate them and discover their hidden rules.</p><p>Menhir’s solution not only focuses on model development and prediction, but covers the entire lifecycle of an AI application. From data collection via our own automated pipelines, through analysis and model training, to the deployment in production of a modern, interactive UI that allows the user to observe the results and explainability of the model, as well as take action to improve the visibility of their ads.</p><p><a href="https://www.menhir.ai/">Menhir AI</a> is a software company based in Madrid, Spain, specialized in the development of Machine Learning models for finance and real estate. Explore career opportunities and join our innovative team. Check out <a href="https://www.menhir.ai/why-menhir#:~:text=J,d"><strong>open roles</strong></a>!</p><blockquote>Ready to improve your business using the power of AI? <a href="https://www.menhir.ai/contact"><strong>Contact us</strong></a> today for impactful results.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*YBYV9CMLk9kDcoRmM08t-Q.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8cf1ff45b5d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/menhirtech/crack-the-algorithm-an-ai-driven-strategy-to-boost-visibility-and-leads-in-real-estate-portals-b8cf1ff45b5d">Crack The Algorithm: An AI driven Strategy to Boost Visibility and Leads in Real Estate Portals</a> was originally published in <a href="https://medium.com/menhirtech">Menhir Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The evolution of Language and the emergence of Natural Language Processing.]]></title>
            <link>https://gabrielfurnieles.medium.com/the-evolution-of-language-and-the-emergence-of-natural-language-processing-ef13384923f5?source=rss-e77c10fd9715------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef13384923f5</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[language]]></category>
            <dc:creator><![CDATA[Gabriel Furnieles]]></dc:creator>
            <pubDate>Mon, 16 Oct 2023 05:01:33 GMT</pubDate>
            <atom:updated>2023-10-16T05:01:33.186Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Evolution of Language and the Emergence of Natural Language Processing: Challenges and Applications of NLP</h3><h4>Transformers In Depth — Chapter 1: Introduction to Language Models Part I</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FUJ63m-C5J6vzeBOvGDzjA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@simonppt?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">SIMON LEE</a> on <a href="https://unsplash.com/photos/FSbgF7PpUio?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p><em>In commencing this first chapter of the series, I wanted to first provide a brief historical overview of Human Language and its subsequent evolution in computing with Natural Language Processing (NLP). Embracing the past is a cornerstone for understanding why and how things have evolved, and sets a context that allows us to study the smartest and most wonderful minds that have ever existed.</em></p><p><em>However, for those whose interests are firmly fixed on the specifics of NLP and are inclined to forgo the historical narrative, you can skip to Part II of this chapter </em><a href="#"><strong><em>Introduction to NLP. Text preprocessing, Tokenizers, Probabilistic Models, and Word embeddings</em></strong></a><em>, where I delve into the intricacies of the concepts and models that make up the broad landscape of this deep field.</em></p><p><em>If anyone has any comments, ideas, suggestions, or questions, please do not hesitate to leave a comment and it will be a pleasure to answer and take them into account to improve the reading and contents of the article. Thanks for reading!</em> <em>— The author </em>✨</p><p><strong>Contents</strong></p><ol><li><strong>A Brief History of Language</strong><br>Why +6900 languages?<br>Language in our Brains</li><li><strong>Natural Language Processing (NLP)<br></strong>Why NLP?<br><strong> </strong>A Historical Journey through NLP<strong><br></strong>Challenges and Applications of NLP</li><li><strong>Summary</strong></li></ol><h3>A Brief History of Language</h3><blockquote>A language is not just words. It’s a culture, a tradition, a unification of a community, a whole history that creates what a community is. It’s all embodied in a language. — Noam Chomsky</blockquote><p>Language allows us to communicate our emotions, ideas or desires, organize in society, imagine and create, and ultimately evolve as a species. Nowadays there exist more than <strong>6900 different languages </strong>[<a href="https://www.linguisticsociety.org/content/how-many-languages-are-there-world">source</a>], each impenetrable from the others (i.e. it is impossible to understand Spanish by just knowing English). But when and how did language arise? Why there exist so many languages? And how did they evolve over time?</p><p>According to scientists [<a href="https://www.linguisticsociety.org/resource/faq-how-did-language-begin">source</a>], human language most likely emerged between 50,000 and 100,000 years ago in the apogee of the <strong>Homo Erectus species</strong>. The Homo Erectus is considered the differentiating link between primates and humans; they had greater cranial capacity than their predecessors and faster cognitive development, and they were <strong>the first to start moving on two legs</strong> (bipedalism).</p><p>According to <a href="https://www.youtube.com/watch?v=nd5cklw6d6Q">Michael Corballis</a>, because of bipedalism, the Homo Erectus hands were freed and their face further exposed, allowing for richer and more complex body expressions — before Homo Erectus it is believed that hominids communicated by repetitive, automatic and emotional gestures, just like other animal species — . Then, when Homo Erectus started to create <strong>the first tools</strong>, their hands were occupied again and communication became based on the face and eventually the mouth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*USOU4h5ujvfuRhfwKDIE6A.png" /><figcaption>Facial reconstruction of a Homo erectus woman, from John Gurche, in a display at the Smithsonian National Museum of Natural History, via <a href="https://artsandculture.google.com/asset/reconstruction-of-homo-erectus-female-john-gurche/jgFIBGdNmA_REg">Google Arts and Culture</a>.</figcaption></figure><p>From that point, language started evolving, becoming more complex and adapting to new necessities (cultural, emotional, technicalities, …) But what is even more important is the emergence of new forms of communication:</p><p>↪ First with cave art and religious icons in the Prehistory (64.000 years ago)<br>↪ Then by pictograms in ancient Mesopotamia (9000 B.C.)<br>↪ Followed by the first alphabets: Sumerian cuneiforms and Egyptian hieroglyphs (3400 B.C.)<br>↪ Evolved in many different ways around the world during the following millennia (Latin, Vedic Sanskrit, Proto-Chinese, …)<br>↪ The first letter where sent by the Persian Queen Atossa (500 B.C.)<br>↪ Guttenberg created the printing press (1450), enabling the exchange of written information on a massive scale.<br>↪ Technology evolved in leaps and bounds and new communication devices were invented: the landline telephone (1876), the radio (1920), television and fax (1926), the internet with modern computers (1983) and finally mobile phones (1984)</p><p>Until finally reaching the present day, when we produce <strong>328.77 million terabytes</strong> of information per day [<a href="https://explodingtopics.com/blog/data-generated-per-day">source</a>] in many different ways (messages, pictures, blogs, …)</p><h4>Why +6900 languages?</h4><p>Back to Homo Erectus, there is fossil evidence from 1.7 million years ago of the <strong>migratory journeys</strong> of this species. Homo Erectus dispersed across Europe and Asia at a rapid pace and with them the languages they used adapted to geography, culture, and religion.<br>Furthermore, language was also used as a security barrier and as the identifying mark of a community; only those who belonged to the group knew the language and could communicate.</p><h4>Language in our Brains</h4><p>In his talk, <a href="https://www.youtube.com/watch?v=nd5cklw6d6Q">The Origins and Evolution of Language</a>, Michael Corballis explains the fundamental difference between human and animal language:</p><ul><li><strong>Human language</strong>: Infinite capacity to say something different and to create meanings.</li><li><strong>Animal language:</strong> Repetitive, automatic, and emotional with no option to create new meanings.</li></ul><p>But at some point in history, this difference wasn’t so, as human language had not yet been invented. <br>Most theories point out that the root of human language lies in primate mirror mechanisms. Looking at the human brain, neuroscientists detected that those regions dedicated to language in humans are the evolution of the <strong>primate’s mirror mechanisms</strong> <a href="https://pubmed.ncbi.nlm.nih.gov/19342089/">[1]</a><a href="https://www.cambridge.org/core/journals/nordic-journal-of-linguistics/article/abs/michael-a-arbib-how-the-brain-got-language-the-mirror-system-hypothesis-new-york-oxford-university-press-2012-pp-xvii-413/E424E4FB5BA8EBE12670417128A0B5B8">[2]</a><a href="https://journals.sagepub.com/doi/10.1111/j.1467-8721.2007.00491.x">[3]</a>. Furthermore, many studies have shown that apes and humans can communicate by gestures and signs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/0*KsvsTlR3-8iGaZx8.png" /><figcaption><strong>Figure 1.</strong> (Left) Location of brain regions dedicated to language in the human brain. (Right) Mirror mechanisms in apes’ brain. Source <a href="http://location of brain regions dedicated to language in the human brain">NTU</a>.</figcaption></figure><p>In simple words, <strong>we learn language by imitation.*</strong></p><p>*This statement will be more relevant in future chapters and sections</p><h3>Natural Language Processing (NLP)</h3><blockquote>Natural language processing (NLP) is the discipline of building machines that can manipulate human language — or data that resembles human language — in the way that it is written, spoken, and organized. — <a href="https://www.deeplearning.ai/resources/natural-language-processing/">DeepLearning AI</a></blockquote><h4>Why NLP?</h4><p>One of computer science’s fundamental goals is to create machines that can communicate with humans seamlessly and intelligently.</p><p>As Alan Turing, father of computation and artificial intelligence, stated in his well-known <a href="https://watermark.silverchair.com/lix-236-433.pdf?token=AQECAHi208BE49Ooan9kkhW_Ercy7Dm3ZL_9Cf3qfKAc485ysgAAA1wwggNYBgkqhkiG9w0BBwagggNJMIIDRQIBADCCAz4GCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMt3MwCj5mjrxf9qYBAgEQgIIDD4M9LBVa5Q0KHs-ucVcHGo1l5y--JmL_Hmy0e39xD83nkYKVnMADl_E1ssgrw-AwN6KTs4cLaLZce2aVzVzIWCQNON-v9V5A4INX2GnOp9H4LPjCcWbRLKEMtG7T0MaTtslEDeLnH9k9IPXB2D-O_zA3G6BIJDkV25oUlgLup4_yz4krwJHpsoIwctLsMyMk86mL4gu9imm8aGbKbG4LYn5oXaWGRYtbquGZIzNx1fHu0nHay7-3jEEnVXMtm3pmsJJ6LoEx0dwuaHOY8wtGPB7kYqV20GrbFeJtpWO2Mq8uSIXrd79iQ-iMJqn5llyHk5SSHBTgrGK6VBS5oRoTOt-EnKJ0xrBdwBkB4oLyJzOwSv75MHhKvTexU9QJ1ty7b5dU6bPFGLbGTKPng0al3ZItszmPO27TmX7Fk7XaSiEusY219ib8gLjC_aZOnSC2MShCkCwrnwTrznbR6B10oi2PfxwIctTRioy9OpHuxqbNQVJCLJYK30ruiBXfBslzfsb46V1yj0odxkLejFRI5pyoqFrlIGRL_sRmzFxJ8hvTSaJ5pcpClQcj03fNemnffEqq5CrqLbkZSYWdII4iP22ooGEHbsz8NM9dMoSsVJM9hGFFtJPlacCNW6JNcirlEnD5zdoIRnuHnMhI_gMdViqOb3B2q1Xh8qj_AwIfa1gsZivoeNZNols95j-20yLo8A5gsjM8jtZsg65gSkeel2G88dSzO6FooDqDlIe-YTKTMEl_o5mIuSp3h1wicivSDTxin0-Mn4uwOs7hHXfBkdOB8jKDnVpZyuGJSJlknQU7QSe30wUrAUKkMdJqsigr5JdjZfsLhyoY0_chL80kZhQD5qG0Y9nWOrWtqtJBIsK4thhMq5xI6x3bw6ombdFxInCGGnTndS0CGOUV1sKPnHSy62ZvtWACHPDb8aVxgxzr-XKFMaLAwvTBKLEcuNo8qAjuRl8izA0R1uWbfJlg1BwFe9amHsN3DpTZ_wfVx1w_y1ymev0YeJlo4-gWKo1if9bhHfE2gv4IZwCO50rgRg"><strong>Turing test (1950)</strong></a>: Intelligent behavior can be attributed to a machine when, during a dialogue with a human evaluator who lacks knowledge about the identity of their conversational partner (whether it be another human, an entity, or a machine), said evaluator is unable to discern whether they are conversing with a machine or a fellow human being.</p><p>So begins the race to find computers capable of dialoguing coherently with humans.</p><h4>A Historical Journey through NLP</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iF37CdoH2tZ7T5-At-GYfQ.png" /><figcaption><strong>Figure 2</strong>. NLP timeline. Image by the author.</figcaption></figure><p><strong>1950–1970:</strong> <em>Symbolic NLP<br></em>Scientists begin to explore the possibilities of teaching languages by abstracting a set of predefined rules (rule-based systems).</p><ul><li>The <strong>Georgetown experiment</strong> successfully translates 64 short sentences from Russian into English (1954)</li><li><strong>Chomsky </strong>tries to encompass all language in a formal structure and builds its hierarchy of grammar (1956)</li><li>Computer programs like ELIZA (1965) or SHRDLU (1970) show promising results as the <strong>first chatbots</strong>.</li></ul><p><strong>1970–1980: </strong><em>AI winter<br></em>Due to a lack of funding and a loss of interest from researchers, AI research went through a difficult period in which the rate of progress was drastically reduced.</p><p><strong>1980–2010:</strong> <em>Statistical and Machine Learning NLP<br></em>Statistical methods start gaining impulse among researchers and new innovative approaches are used:</p><ul><li>IBM Research achieves great results in machine translation with the <a href="https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1162/handouts/Collins_annotated.pdf"><strong>IBM alignment models</strong></a> (1990)</li><li>Hidden Markov Models (<strong>HMM</strong>)show potential when processing short sentences <a href="https://adiwijaya.staff.telkomuniversity.ac.id/files/2014/02/HMMs_speech_recognition.pdf">B. H. Juang &amp; L. R. Rabiner</a> (1991)</li><li>Previous work during the 70s led to the <a href="https://patents.google.com/patent/US5761689A/en"><strong>Autocorrect algorithm by Microsoft</strong></a> (1994), the precursor of the automatic proofreading algorithms we use every day.</li><li>A couple of years later, Microsoft announces <a href="https://patents.google.com/patent/US6377965B1/en"><strong>Autocomplete algorithm</strong></a> (2002) present in every smartphone and search engine nowadays.</li><li>Google launches <a href="https://blog.google/products/translate/ten-years-of-google-translate/"><strong>Google Translate</strong></a> in 2006, the statistical machine translation service that revolutionized the world.</li></ul><p><strong>2000-today:</strong> <em>Neural NLP</em></p><ul><li>In 2003 <a href="https://dl.acm.org/doi/10.5555/944919.944966"><strong>Bengio et. al</strong></a> outperform the statistical approaches using a multi-layer perceptron (a dense NN)</li><li>Years later, <a href="https://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf">Mikolov et. al (2010)</a> present the <strong>Recurrent Neural Network</strong> architecture becoming the state of the art language model.</li><li>New language models begin to emerge: <a href="https://browse.arxiv.org/pdf/1301.3781.pdf">Word2Vec (2013)</a>, <a href="https://nlp.stanford.edu/pubs/glove.pdf">Glove (2014)</a>, <a href="https://browse.arxiv.org/pdf/1409.0473.pdf">Bahdanau et al. (2015)</a> …</li><li>In 2017 Vaswani et. al publish <a href="https://browse.arxiv.org/pdf/1706.03762.pdf"><strong>Attention Is All You Need</strong></a>, the article that introduced the Transformer models and changed the entire AI landscape.</li><li><strong>Pre-trained Language Models</strong> like <a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf">GPT (2018)</a>, <a href="https://browse.arxiv.org/pdf/1810.04805.pdf">BERT (2018)</a>, <a href="https://browse.arxiv.org/pdf/1907.11692.pdf">RoBERTa (2019)</a>, <a href="https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf">GPT-2 (2019)</a> … enter the scene, and with them the wave of Large Language Models (LLM).</li></ul><h4>Challenges and Applications of NLP</h4><p>As a field in constant growth and evolution, NLP […]. Some of these applications are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uLOHlMAFcnD2vw0Yu5SNZw.png" /><figcaption><strong>Figure 3.</strong> NLP applications and challenges (some of them). Image by the author</figcaption></figure><ul><li><strong>Sentiment analysis<br></strong>Classify whether a text has positive, negative or neutral meaning</li><li><strong>Machine Translation<br></strong>Translate texts from one language A to another language B</li><li><strong>Autocorrect<br></strong>Automatically correct misspelled words</li><li><strong>Part Of Speech Tagging<br></strong>Classify words in a text into part-of-speech classes (nouns, verbs, adjectives, …)</li><li><strong>Autocomplete<br></strong>Automatically complete words/parts of a text</li><li><strong>Question Answering<br></strong>Given a question respond with the correct answer</li><li><strong>Text generation</strong><br>Generate text in the same way a human will do</li></ul><p>However, because text is sequential data (i.e. a sequence of sentences, words, subwords, characters…) most of the advances in NLP also have repercussions in other disciplines such as biology and medicine (gene sequencing), chemistry (molecular structures) or economics (time series forecasting).</p><h3>Summary</h3><p>A summary of the main takeaways of this article is presented:</p><ul><li>Human Language started <strong>100,000 years ago</strong> and it evolved from gestures to verbal communication to many other forms like art or text using the technology available at the moment.</li><li>The human brain regions in charge of Language, are the evolution of primate’s mirror mechanisms. <strong>We learn language by imitation!</strong></li><li>Natural language processing (NLP) is the discipline of building machines that can manipulate human language — or data that resembles human language — in the way that it is written, spoken, and organized. — DeepLearning AI</li><li>NLP was born from the <strong>desire of scientists to build intelligent machines capable of communicating with humans</strong> and its history up to the present day goes through 4 phases:<br>- 1950–1970: <em>Symbolic NLP<br>- </em>1970–1980:<strong> </strong><em>AI winter<br>- </em>1980–2010: <em>Statistical and Machine Learning NLP<br>- </em>2000-today: <em>Neural NLP</em></li><li>NLP applications range from text processing and linguistics to many other different areas of science. Because text data can be seen as sequential data, <strong>NLP advancements can be also applied in Biology, Chemistry, or Economics</strong> among others.</li></ul><p><em>Hi, I’m Gabriel Furnieles, a Mathematical Engineering student specializing in Artificial Intelligence. I write about AI and Data Science topics. I hope you enjoyed the article and found it helpful, if so, please consider following me </em><a href="https://medium.com/u/e77c10fd9715"><em>Gabriel Furnieles</em></a><em> and subscribing to my newsletter so stories will be sent directly to you 👇</em></p><p><a href="https://gabrielfurnieles.medium.com/subscribe">Get an email whenever Gabriel Furnieles publishes.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef13384923f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformers in depth - Part  1. Introduction to Transformer models in 5 minutes]]></title>
            <link>https://medium.com/data-science/transformers-in-depth-part-1-introduction-to-transformer-models-in-5-minutes-ad25da6d3cca?source=rss-e77c10fd9715------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad25da6d3cca</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Gabriel Furnieles]]></dc:creator>
            <pubDate>Mon, 27 Mar 2023 19:48:52 GMT</pubDate>
            <atom:updated>2023-04-03T10:07:53.939Z</atom:updated>
            <content:encoded><![CDATA[<h3>Transformers in depth – Part 1. Introduction to Transformer models in 5 minutes</h3><h4>Understanding Transformer architecture and its key insights in 5 minutes</h4><p>This is the first part of the article’s extended version, soon you will find its continuation <a href="https://medium.com/@gabrielfurnieles">here</a>.</p><blockquote><strong>Author’s note.</strong> For this very first part, I’ve decided to introduce the notions and concepts necessary to get a better understanding of Transformer models and to make it easier to follow the next chapter. If you are already familiar with Transformers, you can check the last section to get a summary of this article and feel free to jump to the second part, where more mathematics and complex notions are presented. Nevertheless, I hope you find some value in the explanations of this text too. Thanks for reading!</blockquote><p>Since the release of the latest Large Language Models (LLaM), like the GPT series of <a href="https://openai.com/blog/chatgpt">OpenAI</a>, the open source model <a href="https://bigscience.huggingface.co/blog/bloom">Bloom</a> or Google’s announcements about <a href="https://blog.google/technology/ai/lamda/">LaMDA </a>among others, Transformers have demonstrated their huge potential and have become the cutting-edge architecture for Deep Learning models.</p><p>Although several articles have been written on transformers and the mathematics under their hood <a href="https://arxiv.org/pdf/2207.09238.pdf">[2]</a> <a href="https://jalammar.github.io/illustrated-transformer/">[3]</a> <a href="https://kazemnejad.com/blog/transformer_architecture_positional_encoding/">[4]</a>, in this series of articles I’d like to present a complete overview combining what I’ve considered the best approaches, with my own point of view and personal experience working with Transformer models.</p><blockquote>This article attempts to provide a deep mathematical overview of Transformer models, showing the source of their power<strong> </strong>and explaining the reason behind each of its modules.</blockquote><blockquote><strong>Note.</strong> The article follows the original transformer model from the paper <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf">Vaswani, Ashish, et al. 2017.</a></blockquote><h4>Setting the environment. A brief introduction to Natural Language Processing (NLP)</h4><p>Before getting started with the Transformer model, it is necessary to understand the task for which they have been created, to process text.</p><p>Since neural networks work with numbers, in order to feed text to a neural network we must first transform it into a numerical representation. The act of transforming text (or any other object) into a numerical form is called <strong>embedding</strong>. Ideally, the embedded representation is able to reproduce characteristics of the text such as the relationships between words or the sentiments of the text.<br>There are several ways to perform the embedding and it is not the purpose of this article to explain them (more information can be found in NLP Deep Learning), but rather we should understand their general mechanisms and the outputs they produce. If you are not familiar with embeddings, just think of them as another layer in the model’s architecture that transforms text into numbers.</p><p>The most commonly used embeddings work on the words of the text, transforming each word into a vector of a really high dimension (the elements into which the text is divided to apply the embedding are called <strong>tokens</strong>). In the original paper [1] the <strong>embedding dimension</strong> for each token/word was <em>512</em>. It is important to note that the vector modulus is also <strong>normalized </strong>so the neural network is able to learn correctly and avoid exploding gradients.</p><p>An important element of embedding is the <strong>vocabulary</strong>. This corresponds to the set of all tokens (words) that can be used to feed the Transformer model. The vocabulary isn’t necessarily to be just the words used in the sentences, but rather any other word related to its topic. For instance, if the Transformer will be used to analyze legal documents, every word related to the bureaucratic jargon must be included in the vocabulary. Note that the larger the vocabulary (if it is related to the Transformer task), the better the embedding will be able to find relationships between tokens.</p><p>Apart from the words, there are some other <strong>special tokens</strong> added to the vocabulary and the text sequence. These tokens mark special parts of the text like the beginning &lt;START&gt;, the end &lt;END&gt;, or padding &lt;PAD&gt; (the padding is added so that <strong>all sequences have the same length</strong>). The special tokens are also embedded as vectors.</p><p>In mathematics, the embedding space constitutes a <strong>normalized vector space</strong> in which each vector corresponds to a specific token. The basis of the vector space is determined by the relationships the embedding layer has been able to find among the tokens. For example, one dimension might correspond to the verbs that end in <em>-ing</em>, another one could be the adjectives with positive meaning, etc. Moreover, the angle between vectors determines the similarity between tokens, forming clusters of tokens that have a semantic relationship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*bZOITiXQ1d7oFFbCyrlXJg.png" /><figcaption>Text embedding illustration. On the left, words/tokens are embedded as vectors where d_model represents the maximum embedding dimensions. On the right, the first 3 dimensions have been represented. Although the example is exaggerated, note how similar words form groups (clusters) representing the similarity between them. Image by the author.</figcaption></figure><blockquote><strong>Note 1.</strong> Although only the task of text processing has been mentioned, Transformers are in fact designed to process any type of sequential data.</blockquote><h3>Transformer’s workflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CU0QlBx15Rd3TWUwc2M3YA.png" /><figcaption>Transformer diagram. Image from the paper <a href="https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf">Vaswani, Ashish, et al. 2017.</a></figcaption></figure><p>Above, is one of the most replicated diagrams in the last years of Deep Learning research. It summarizes the complete workflow of Transformers, representing each of the parts/modules involved in the process.</p><p>A higher perspective view divides Transformers into <strong>Encoder</strong> (left blue block in the diagram)<strong> </strong>and <strong>Decoder </strong>(right blue block).</p><p><strong>To illustrate how Transformers work I will use the example task of text translation from English to Spanish.</strong></p><ul><li>Encoder’s objective is to find the relationships between the tokens of the input sequence i.e. the sentence to translate into Spanish. It gets the English sentence as input (after applying the embedding) and outputs the same sequence weighted by the attention mechanism. Mathematically, <a href="http://a/">the Encoder performs a transformation in the embedding space of English tokens, weighting the vectors according to their importance in the sentence meaning</a>.</li></ul><blockquote><strong>Note 2.</strong> I haven’t defined yet what attention is, but essentially think of it as a function that returns some coefficients that define the importance of each word in the sentence with respect to the others.</blockquote><ul><li>On the other hand, Decoder takes as first input the translated sentence in Spanish (<em>Outputs </em>in the original diagram), applies attention, and then combines the result with the encoder’s output in another attention mechanism. Intuitively, <a href="http://a/">the Decoder learns to relate the target embedding space (Spanish) with respect to the input embedding space (English) so that it finds a basis transformation between both vector spaces.</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2WFGK6c02AnEpqjO1BscBQ.png" /><figcaption>Transformer’s high perspective overview. The Encoder and Decoder layers have been grouped into single blocks for simplicity, and the outputs of each block have been annotated. Image by the author.</figcaption></figure><blockquote><strong>Note 3.</strong> For clarity, I will use the notation source input to refer to the Encoder’s input (sentence in English) and target input to the expected output introduced in the Decoder (sentence in Spanish). This notation will remain consistent for the rest of the article.</blockquote><p>Now let’s have a closer look at the Inputs (source and target) and the Output of the Transformer:</p><p>As we’ve seen, Transformer input text is embedded into a high dimensional vector space, so instead of a sentence, a sequence of vectors is entered. However, there exists a better mathematical structure to represent sequences of vectors, the matrices! And even further, when training a neural network, we don’t train it sample by sample, we rather use batches in which several samples are packed. <strong>The resulting input is a tensor of shape <em>[N, L, E] </em>where <em>N</em> is the batch size, <em>L</em> the sequence length, and <em>E</em> the embedding dimension</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kNdPQhYhaMk-A3Dl283uWA.png" /><figcaption>Illustration of the <strong>Source input tensor</strong>. On the left, the 2D tensor represents the first <strong>embedded sequence</strong> of the batch. On the right, the full batch tensor containing all the sequences of a single batch that is fed into the Transformer Encoder. Similarly, the Target tensor has the same shape but contains the true output. Image by the author.</figcaption></figure><p>As for the output of the Transformer, a Linear + Softmax layer is applied which produces some Output Probabilities (recall that the Softmax layer outputs a probability distribution over the defined classes). The output of the Transformer isn’t the translated sentence but a probability distribution over the target vocabulary that determines the words with the highest probability. Note that for each position in the sequence length, a probability distribution is generated to select the next token with a higher probability. <strong>Since during training the Transformer processes all the sentences at once, we get as output a 3D tensor that represents the probability distributions over the vocabulary tokens with shape <em>[N, L, V] </em>where N is the batch size, L the sequence length, and V the vocabulary length</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fstmsw_KdY9HDZbBXtfqNA.png" /><figcaption>Illustration of the<strong> Output Probabilities tensor</strong>. On the left, the probability distribution prediction over the vocabulary for a single sequence. Each column represents a word in the target vocabulary space, and the rows correspond to the sequence tokens. On the right, the full predicted tensor for a whole batch. Image by the author.</figcaption></figure><p>Finally,<strong> the predicted tokens are the ones with the highest probability</strong>.</p><blockquote><strong>Note 4.</strong> As explained in the Introduction to NLP section, all the sequences after the embedding have the same length, which corresponds to the longest possible sequence that can be introduced/produced in/by the Transformer.</blockquote><h3>Training vs Predicting</h3><p>For the final section of the article’s Part 1, I’d like to make a point about the Training vs Predicting phases of Transformers.</p><p>As explained in the previous section, Transformers take two inputs (source and target). During Training, the Transformer is able to process all the inputs at once, meaning the input tensors are only passed one time through the model. The output is effectively the 3-dimensional probability tensor presented in the previous figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iSNi6lpr6FgIKNzszstfjQ.png" /><figcaption>Transformer training phase. First, the source input (English sentence) is embedded and attention is applied in the Encoder. Then, the target input (Spanish) is entered into the Decoder, where source attention and target attention are combined in the Decider’s 2nd layer. Finally, the probability distribution over the vocabulary tokens is generated and the highest tokens are selected to form the translated output sentence. <strong>Note that dimensions (L,E) do not change between layers</strong>. Image by the author.</figcaption></figure><p>On the contrary, in the Prediction phase, there is no target input sequence to feed the Transformer (we wouldn’t need a Deep Learning model for text translation if we already know the translated sentence). So, what do we enter as target input?<br>It is at this point that the <strong>auto-regressive behavior</strong> of Transformers comes to light. The Transformer can process the source input sequence at once in the Encoder, but for the Decoder’s module, it enters a loop where at each iteration it just produces the next token in the sequence (a row probability vector over the vocabulary tokens). The selected tokens with higher probability are then entered as the target input again so the Transformer always predicts the next token based on its previous predictions (hence the auto-regressive meaning). But what should be the first token entered at the very first iteration?<br>Remember the special tokens from the Introduction to NLP section? The first element introduced as target input is the beginning token &lt;START&gt; that marks the opening of the sentence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uwuO4A2ENjxIUrUw0ruBTA.png" /><figcaption>Transformer predicting phase. The Encoder part remains unchanged, whereas Decoder’s tensors are row vectors. The input text for the first (0) iteration is empty since there is only the &lt;START&gt; token. <strong>Note the arrow connecting the predicted token back to the target input representing the auto-regressive behavior</strong>. In the next iterations, the target input will grow as the transformer predicts new tokens from the sequence. However, only the last row in the output probability tensor is taken into account, which means that the past predicted tokens cannot change. Image by the author.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W4ACHImZGzzpGrLUuFJQ5A.png" /><figcaption>Table comparing Decoder dimensions during predicting phase (iteration 0 is depicted in the above figure). Note the increase in dimensions from 1 to L-1. Once the Transformer has predicted the second last token in the sequence, it adds the &lt;END&gt; special token finishing the prediction. Image by the author.</figcaption></figure><h3>Summary</h3><p>This Part has been an Introduction to the first notions and concepts necessary to get a better understanding of Transformers models. In the next Part, I will delve into each of the modules of the Transformers architecture where most of the mathematics resides.</p><p>The main ideas and concepts behind this article are:</p><ul><li>Transformers work in a <strong>normalized vector space</strong> defined by the embedding system and where each dimension represents a characteristic between tokens.</li><li>Transformers <strong>inputs are tensors of shape [N, L, E]</strong> where N denotes the batch size, L is the sequence length (constant for every sequence thanks to the padding) and E represents the embedding dimension.</li><li>While the <strong>Encoder </strong>finds relationships between tokens in the source embedding space, the <strong>Decoder’s </strong>task is to learn the projection from the source space into the target space.</li><li>Transformer’s output is a line vector whose length is equal to the vocabulary’s size and where <strong>each coefficient represents the probability</strong> of the corresponding indexed token being placed next in the sequence.</li><li>During training, Transformer processes all its inputs at once, outputting a [N, L, V]<strong><em> </em></strong>tensor (V is the vocabulary length). But during predicting, <strong>Transformers are auto-regressive</strong>, predicting token by token always based on their previous predictions.</li></ul><p>Soon the next article’s part will be available <a href="https://medium.com/@gabrielfurnieles"><strong>here</strong></a></p><p>Thanks for reading! If you’ve found this article helpful or inspiring, please consider following me for more content on AI and Deep Learning.</p><p><a href="https://medium.com/@gabrielfurnieles">Gabriel Furnieles - Medium</a></p><p>You might also be interested in:</p><p><a href="https://towardsdatascience.com/the-5-most-promising-ai-models-for-image-translation-de2677e526e6">The 5 most promising AI models for Image translation</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad25da6d3cca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/transformers-in-depth-part-1-introduction-to-transformer-models-in-5-minutes-ad25da6d3cca">Transformers in depth - Part  1. Introduction to Transformer models in 5 minutes</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why do We use Cross-entropy in Deep Learning — Part 2]]></title>
            <link>https://medium.com/data-science/why-do-we-use-cross-entropy-in-deep-learning-part-2-943c915db115?source=rss-e77c10fd9715------2</link>
            <guid isPermaLink="false">https://medium.com/p/943c915db115</guid>
            <category><![CDATA[categorical-cross-entropy]]></category>
            <category><![CDATA[binary-cross-entropy]]></category>
            <category><![CDATA[cross-entropy-loss]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-neural-network]]></category>
            <dc:creator><![CDATA[Gabriel Furnieles]]></dc:creator>
            <pubDate>Mon, 05 Dec 2022 19:20:59 GMT</pubDate>
            <atom:updated>2022-12-05T19:20:59.493Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why do We use Cross-entropy in Deep Learning — Part 2</h3><h4>Explanation of one of the most widely used loss functions in Artificial Neural Networks</h4><blockquote><strong>Note</strong>. This is the continuation of the first part: <a href="https://medium.com/towards-data-science/why-do-we-use-cross-entropy-in-deep-learning-part-1-e787f885c1c">Why do We use Cross-entropy in Deep Learning — Part 1</a></blockquote><p><em>Entropy</em>, <em>Cross-entropy</em>, <em>Binary Cross-entropy, </em>and C<em>ategorical Cross-entropy </em>are crucial concepts in Deep Learning and one of the main loss functions used to build Neural Networks. All of them derive from the same concept: <strong>Entropy</strong>, which may be familiar to you from physics and chemistry. However, not many courses or articles explain the terms in-depth, since it requires some time and mathematics to do it correctly.</p><p>In the first post, I presented three different but related conceptions of entropy and where its formula derives from. However, there is still one key concept to address, since Deep Learning does not use Entropy but a close relative of it called <strong>Cross-entropy</strong>. What this term means, how to interpret it and on which Deep Learning problems it is applied is the<strong> </strong>focus of this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HRuKKhhHkz0gsYQ2GywyNw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@dissii?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">mahdis mousavi</a> on <a href="https://unsplash.com/s/photos/deep-learning?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Cross-Entropy</h3><p>Recalling the first post (<a href="https://medium.com/towards-data-science/why-do-we-use-cross-entropy-in-deep-learning-part-1-e787f885c1c">link</a>), the entropy formula can be interpreted as <strong><em>the optimal average length of an encoded message</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/0*4x8h9_Y146wZqlaM.png" /><figcaption>Entropy formula for discrete random variable X using binary codification (log base is 2). Image by the author.</figcaption></figure><blockquote><strong>Note1</strong>. Recall from the previous post that the base of the logarithm isn’t really important, since changing a logarithm base is equivalent to multiplying by a constant. This only applies to optimization and comparative methods, where constants are ignored.</blockquote><p>Alternative to entropy, <strong>Cross-entropy</strong> is defined as the entropy of a random variable <em>X</em>, encoded using a probability distribution <em>P(x),</em> but using a different probability distribution <em>Q(x) </em>to compute the average.</p><p>In other words, <strong>what would be the optimal average message length of a random variable <em>X</em> distributed as <em>Q(x)</em> but using the encoding system of the distribution <em>P(x)?</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/0*FblBHh07FtQoJUX5.png" /><figcaption>Cross-entropy formula using binary codification (log_2). Image by the author.</figcaption></figure><blockquote><strong>Note2</strong>. Another common notation to define Cross-entropy is H(Q,P). However, I consider using sub-indices instead in order to keep the notation consistent with the Entropy formula.</blockquote><p>To clear things up, let’s use an example. I am going to take the same example as in the first post, but remember that it works the same with any discrete random variable.</p><p>In the example, we were studying the optimal average length of encoding our laboratory experiments which could result in 6 different outcomes (<strong><em>microstates</em></strong>): Microstate 1 (M1), M2, …, M6.<br>Let’s suppose now that we have some parameters <strong><em>K</em></strong><em> </em>that we can modify in our experiment so that when we use different combinations of <em>K</em> we obtain different probability distributions for the results (the microstates)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Yw6F2OiWYC27QN5JvZsiA.png" /><figcaption>Probability distributions of the experiment results for Variant K1 and Variant K2. The Y axis (Probability) has been scaled to improve visualization. Image by the author.</figcaption></figure><p>As we have seen, entropy is a function of a probability distribution, i.e. the entropy of a distribution <em>P(x)</em> and <em>Q(x)</em> does not have to be the same. The reason is that to calculate entropy we have set the respective probabilities of each result as the cost value (see section Entropy Formula of the first part for a detailed explanation) In other words, <strong>the encoding for P(x) isn’t necessarily the same as the encoding of Q(x)</strong>. Although <em>X</em> is the same random variable for both cases, what changes is its distribution.</p><p>Now the following dilemma emerges: <strong>If we could only use one encoding system to send the results of experiment variants K1 and K2, which encoding would be more optimal, and therefore should we use? </strong>Encoding K1 or K2?</p><p>The solution to this problem goes through computing the Entropy and Cross-entropy for each variant K1 and K2 (with respect to the other) and averaging the results to obtain the global average length using encoding systems K1 and K2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*10yL5L_G1LzZD4yjWpuAQA.png" /><figcaption>Which encoding system is better K1 or K2? From top line to bottom entropy and cross-entropy are calculated for both encodings. Note that encoding refers to the cost log_2(1/P(xi)) assigned to a certain result xi. Image by the author</figcaption></figure><p>As calculated, the optimal coding system for sending messages of the K1 and K2 variants is <strong>the one from K1</strong>, which produces the shortest length (2.622).</p><p>One might think that the difference between the two encodings isn’t so significant, as the results have to be rounded up when working with bits. However, when the sequences of words included in the message are sufficiently high, this difference is palpable.<br>Suppose your message it’s a tweet with 280 bits maximum. With encoding K1 you will be able to send 280/2.622 ~ 106 words on average, whereas using encoding K2 enables you to communicate 103 words (Who hasn’t ever suffered from not being able to write 3 more words in a tweet?) And the gap continues to grow as you increase the number of words per message.</p><blockquote><strong>Note3</strong>. Furthermore, there is a concept called <a href="https://colah.github.io/posts/2015-09-Visual-Information/#fractional-bits">fractional bits</a>, which is a method of maximizing the optimal length of words when communicating more than one word whose optimal length is decimal.</blockquote><p>An <strong>important observation</strong> is that Cross-entropy isn’t symmetric, so that</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/141/0*r3J5IhJ2FzbUV0o0.png" /><figcaption>Cross-entropy isn’t symmetric. Image by the author</figcaption></figure><h3>Cross-entropy in Deep Learning</h3><p>We have already seen an example of how we can apply Cross-entropy to a problem, but since Deep Learning is not very interested in (just) encoding messages, why, when, and how did this function come into the scene?</p><p>The answer, of course, lies in its application, so let’s introduce the problem Cross-entropy solves in Deep Learning.</p><h4>Deep Learning for classification tasks</h4><p>Imagine you want to classify electrocardiogram signals (ECM) into a set of different patterns that represent cardiovascular diseases. Actually, this is a very difficult task even for the most experimented cardiologists since patterns of cardiac abnormalities can be very subtle and the difference between a healthy person and a person with cardiac disease is minimal.</p><blockquote><strong>Note4</strong>. If you are interested in how the classification of ECM is tackled nowadays using Deep Learning refer to <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8802771/">this article</a>.</blockquote><p>Forcing the model to just <strong>predict an integer from <em>1</em> to <em>K</em></strong><em>, </em>where K is the number of classes (ECM patterns), <strong>isn’t a good tactic</strong>. Using this system, the error of our model would be “absolute”, in the sense that we would be able to tell if the model has performed well or badly, but not how bad or how good it has been in its prediction. <br>Suppose the model has predicted class 2 but the correct result is class 4, then how can we compute the error? Calculating the distance (or the MSE) between correct and predicted class (|4–2| = 2) can’t be used as an error measurement, because then we are stating that predicting class 2 is better than predicting class 9 (|4–2| &lt; |4–9|), that is, we are assuming that class 2 is closer to class 4 than is class 9 based on our human criteria. Moreover, this method also assumes that class 3 is as close to class 4 as it is to class 2, which in practice is not possible to evaluate: How similar are a crocodile, a cat, and a dog? Perhaps a cat is 1 unit similar to a dog, but then how many units<em> cat vs dog</em> are the crocodile similar to the cat? We can’t simply do it because we cannot compare a crocodile vs a cat using a cat vs dog measurement unit!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-BQ_sJLkVlyXo17ePbljHw.png" /><figcaption>We cannot compare a crocodile with a cat using a cat vs dog measurement. Photos by <a href="https://unsplash.com/ja/@gaetanocessati?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Gaetano Cessati</a> on <a href="https://unsplash.com/s/photos/crocodile?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a> (crocodile), <a href="https://unsplash.com/@lloydhenneman?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Lloyd Henneman</a> on <a href="https://unsplash.com/s/photos/cat?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a> (cat), <a href="https://unsplash.com/@marliesebrandsma?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Marliese Streefland</a> on <a href="https://unsplash.com/s/photos/dog?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a> (dog). Edited by the author.</figcaption></figure><p>Another way of thinking of the problem is <strong>probabilistic</strong>, where instead of outputting <em>1</em>,<em>2</em> or <em>K,</em> <strong>the model will estimate the probability of the ECM presenting each pattern</strong>.</p><blockquote>The output of our network will be then a probability distribution over all the <em>K</em> classes</blockquote><p>Thus, the ECM pattern with a probability greater than &gt;<em>0.5</em> will be the pattern that the network has classified in the input ECM signal.</p><blockquote><strong>Note5</strong>. This simple and intuitive assumption of assigning a data sample to its most likely class where <em>P(class k | x) </em>is the greater (reads probability of belonging to class k given the data sample x) comes from the notion of <a href="https://en.wikipedia.org/wiki/Naive_Bayes_classifier">Naive Bayes Classifier</a> which in an ideal world where we know the true underlying conditioned probabilities is the best possible classifier.</blockquote><p>To compute now the error we must also define a probability distribution for the expected output (i.e. the true class) Since we want to define probability 1 to the correct class and probability 0 to the rest, it is as simple as using <em>one-hot encoding</em>. Therefore, class 1 will be <em>[1 0 0 … 0]</em>, class 2 <em>[0 1 0 … 0],</em> and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vq8mx3rVWNHFJ4_M_3sWdw.png" /></figure><h4>Categorical Cross-Entropy</h4><p>At this point, We have defined the problem approach and the form of the model’s output, but we haven’t established any loss function yet.</p><p>We need to find a way of measuring the difference between the model’s output and the true class vector i.e. we need to define <strong>a measure of the distance between two probability distributions</strong>.</p><p>Not surprisingly mathematicians had already thought about a tool capable of solving this need. The solution lies in the field of <a href="https://en.wikipedia.org/wiki/Information_geometry">Information Geometry</a> under the concept of <strong>Divergence</strong></p><p><em>`In information geometry, a </em><strong><em>divergence </em></strong><em>is a kind of statistical distance: a binary function which establishes the separation from one probability distribution to another on a statistical manifold (a space where each point represents a probability distribution) (</em><a href="https://en.wikipedia.org/wiki/Divergence_(statistics)"><em>Wikipedia</em></a><em>)</em></p><p>There are currently different divergence measurements (<a href="https://en.wikipedia.org/wiki/Statistical_distance#Divergences">see</a>), however, one of them is very special because it significantly simplifies the calculations for us: The <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence"><strong>Kullback–Leibler divergence</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/0*tI_7q-bQbVaCZh8l.png" /><figcaption>Mathematical definition of Kullback-Leibler divergence. Image by the author.</figcaption></figure><p>It is important to note that the Kullback-Leibler divergence isn’t symmetric:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/217/0*CPBo3UaArP__bayg.png" /><figcaption>Kullback-Leibler divergence isn’t symmetric. Image by the author.</figcaption></figure><p>Since we are using the the Kullback-Leibler divergence as an error measurement (the more the distance between the probab. distributions the worse the model’s prediction), we want to <strong>minimize </strong>it.</p><blockquote><strong>Note6</strong>. The following derivation is taken from <a href="https://www.youtube.com/watch?v=Pwgpl9mKars&amp;t=210s">this</a> video.</blockquote><p>Defining :<br>P* → The True Class distribution<br>P → The predicted Model distribution<br>θ → The Model parameters<br>P*(y|x) → The true probability of belonging to <em>y</em> class given the input data <em>x</em><br>P(y|x;θ) → The predicted probability by the model of belonging to <em>y</em> class given the input data <em>x</em> and the parameters <em>θ</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*5blSOijcGhlInu-B.png" /><figcaption>Kullback-Leibler divergence rewritten for the classification problem. <strong>Note that the summation is over all the possible classes</strong> Y (in our ECM problem class 1–15), which in practice is operating alongside the the output probability vector. Image by the author.</figcaption></figure><p>And operating the terms:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kAajxf7TgnY-U5K-OymqnA.png" /><figcaption>Since the 1st term doesn’t depend on the parameters of the model (θ) we can treat it as a constant value, and therefore ignore it in the optimization problem. Image by the author.</figcaption></figure><p>Finally, we can show that:</p><blockquote>M<strong>inimizing the Kullback-Leibler divergence is equivalent to minimizing the Cross-entropy between the True class probability distribution (P*) and the predicted (P)</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/0*UFfALXTBQqD8H0Je.png" /><figcaption>Minimizing the Kullback-Leibler divergence is the same as minimizing the Cross-entropy between the True class prob. distribution (P*) and the predicted one (P). Image by the author.</figcaption></figure><p>That is, <strong>in optimization problems Cross-entropy can be seen as a measure of the distance between two probability distributions</strong>.</p><blockquote><strong>Note7</strong>. We can also think about computing the KL Divergence D_{KL}(P||P^*) since as we know KL is not symmetric. However, doing that leads to a mathematical indeterminacy. After operating the terms and removing the ones not dependent on θ, we get the formula described below. The indeterminacy occurs when P*(y|x) = 0 (remember P*(y|x) is a one-hot encoding vector) and log(0) is not defined.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/0*EztZl1iFKptBJbEb.png" /><figcaption>Computing the KL Divergence D_{KL}(P||P^*) isn’t possible due to the indeterminacy log(0). Image by the author.</figcaption></figure><p>Computing Cross-entropy alongside multiple classes receives the name of <strong>Categorical Cross-entropy</strong> to differentiate from Binary Cross-entropy. In Deep Learning libraries like Tensorflow there already exists a pre-built function to compute the Categorical Cross-entropy:</p><p>tf.keras.losses.CategoricalCrossentropy()</p><h4>Binary Cross-Entropy</h4><p>In the previous problem we had to classify a data point x into multiple classes, but what about binary classification problems where we just have 2 classes?</p><p>Again we can think of the problem using probabilities. The only difference is that now the output of our model isn’t a probability vector rather than a single value representing the probability of the input data point x of belonging to the first class, since we can compute the probability of belonging to the second class by applying P(class 2) = 1-P(class 1)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lm7OjZkPKDqabhRL4B4gsQ.png" /><figcaption>BInary classification model. <strong>Recall that the sum of all the probabilities over the possible classes must be equal to 1</strong>. Image by the author.</figcaption></figure><p>Once again we can apply the Cross-entropy formula. Using now the fact that we have just two classes we can expand the summation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zvGrCwae45IiA1xg.png" /><figcaption>Derivation of Binary Cross-entropy. Since there are only 2 classes then it is satisfied P(class 2) = 1-P(class 1) where P(class 1) = P(y1|x). Image by the author.</figcaption></figure><p>Nevertheless, a fact that I haven’t mentioned before is that usually loss functions are computed on a batch of data points, so the average loss is taken over those points. Then, you will probably see Binary Cross-entropy (BCE) as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/0*mtxum1kB0qSSXzMu.png" /><figcaption>BCE formula for a batch of N data points. Image by the author.</figcaption></figure><p>Likewise Categorical Cross-entropy, most Deep Learning libraries have a pre-built function for Binary Cross-entropy. In the case of Tensorflow:</p><p>tf.keras.losses.BinaryCrossentropy()</p><h3>Conclusions</h3><p>So far during this series of two posts I’ve presented the intuition and formal derivation of <em>Entropy</em>, <em>Cross-entropy</em>, <em>Categorical Cross-entropy</em> and <em>Binary Cross-entropy</em>, as well as their application in classification problems. Summarizing all the content seen in a few lines:</p><ul><li><strong>(Shannon) Entropy</strong> = Optimal average length of an encoded message, where the base of logarithm in the formula defines the possible characters for encoding (2 for binary code)</li><li><strong>Cross-entropy</strong> = The optimal average message length of a random variable <em>X</em> distributed as <em>Q(x)</em> but using the encoding system of the distribution <em>P(x) → </em><a href="#">In optimization problems can be seen as a measure of the distance between two probability distributions</a></li><li><strong>Categorical Cross-entropy</strong> = Cross-entropy applied as a loss function in multiclassification problems (when there are more than 2 classes)</li><li><strong>Binary Cross-entropy</strong> = Cross-entropy applied as a loss function in binary classification problems (when there are just 2 classes)</li></ul><p>Finally, if you want to read more on how classification is performed using Neural Network models, I recommend reading my post <a href="https://medium.com/towards-data-science/sigmoid-and-softmax-functions-in-5-minutes-f516c80ea1f9?source=user_profile---------3----------------------------">Sigmoid and SoftMax Functions in 5 minutes</a>, in which I explain the two activation functions used for outputting the probability distributions in both multiclass and binary problems.</p><p><a href="https://towardsdatascience.com/sigmoid-and-softmax-functions-in-5-minutes-f516c80ea1f9">Sigmoid and SoftMax Functions in 5 minutes</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=943c915db115" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-do-we-use-cross-entropy-in-deep-learning-part-2-943c915db115">Why do We use Cross-entropy in Deep Learning — Part 2</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why do We use Cross-entropy in Deep Learning — Part 1]]></title>
            <link>https://medium.com/data-science/why-do-we-use-cross-entropy-in-deep-learning-part-1-e787f885c1c?source=rss-e77c10fd9715------2</link>
            <guid isPermaLink="false">https://medium.com/p/e787f885c1c</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[cross-entropy]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[mathematics]]></category>
            <dc:creator><![CDATA[Gabriel Furnieles]]></dc:creator>
            <pubDate>Thu, 10 Nov 2022 13:55:16 GMT</pubDate>
            <atom:updated>2023-07-05T17:59:02.312Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why do We use Cross-entropy in Deep Learning — Part 1</h3><h4>Explanation of one of the most widely used loss functions in Artificial Neural Networks</h4><p><strong>The second part is available </strong><a href="https://medium.com/towards-data-science/why-do-we-use-cross-entropy-in-deep-learning-part-2-943c915db115"><strong>here</strong></a><strong>!</strong></p><p><a href="https://towardsdatascience.com/why-do-we-use-cross-entropy-in-deep-learning-part-2-943c915db115">Why do We use Cross-entropy in Deep Learning — Part 2</a></p><p>If you’ve just started in the field of Deep Learning and have read some specialized articles, I am very sure that you have come across any of the following terms: <em>entropy</em>, <em>cross-entropy</em>, <em>binary cross-entropy, </em>or <em>categorical cross-entropy</em>.</p><p>All of them derive from the same concept: <strong>Entropy</strong>, which may be familiar to you from physics and chemistry. But, What does physics have to do with Artificial Intelligence? Where do the formulas come from? How do we interpret them? and, On which problems do we apply them?</p><p>Entropy, Cross-entropy and Categorical Cross-entropy are crucial concepts in the field of AI. However, not many courses or articles explain the terms in-depth, since it requires some time and mathematics to do it correctly. In this series of two posts, I’ll try to give a clear explanation using the resources and intuitions I gathered when I was looking for an article like this one. Furthermore, I’ll provide a formal mathematical description of the formulas and where they derive from.</p><blockquote><strong>Note</strong>. During my bibliographic research, I found <a href="https://colah.github.io/posts/2015-09-Visual-Information/"><strong>this</strong></a> incredible article by Christopher Olah, which inspired this article. If you have the time, I highly recommend reading it as well as checking Christopher’s website.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HRuKKhhHkz0gsYQ2GywyNw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@dissii?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">mahdis mousavi</a> on <a href="https://unsplash.com/s/photos/deep-learning?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Entropy definition</h3><p>The term <strong>Entropy</strong> might be familiar to you from your thermodynamic lessons in elementary physics and chemistry, but it has also two other conceptions from different branches of mathematics and computer science:</p><h4>Entropy in physics</h4><p>According to classical thermodynamics, <em>“</em><strong><em>Entropy is a measure of the amount of disorder in a system</em></strong><em>”</em>.</p><p>A physical system can be seen as a set of particles, each with an energy associated. Disorder refers to the number of possible configurations the system can adopt (also called <em>microstates</em>), the more configurations a system have, the greater the disorder and the greater the entropy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xHKMCppN2Xz_y7ltK-UA2A.png" /><figcaption>All possible microstates of a 3-particle system with overall energy 5E. The bars represent the energy associated with each particle, such that for Microstate 1: Particle 1 has 2E, Particle 2 2E and Particle 3 1E. Thus the total system energy is the sum 2E + 2E + 1E = 5E. Image by the author.</figcaption></figure><blockquote><strong>Note1</strong>. If you’d like to learn more about entropy in physics watch <a href="https://youtu.be/mg0hueOyoAw">this video</a></blockquote><h4>Entropy in statistics</h4><p>Deriving from the thermodynamics definition of entropy, we can also think of entropy as: <strong><em>“Entropy is a measure of the uncertainty of a random variable”</em></strong>.</p><p>Let’s put it this way: our random variable is the microstates (possible energy configurations) of a system, where each configuration is a value our random variable can take. Then, assuming that each microstate has the same probability of happening, the more configurations our system can adopt the more uncertainty we have about what microstate the system is actually in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_7vcO0_OcdnsQDXihhcOEw.png" /><figcaption>Probability distributions for two systems: System 1 with two Microstates (M1, M2) and System 2 with six (M1, …, M6). Image by the author.</figcaption></figure><p>Another possible understanding of entropy in statistics could be: <strong><em>“Entropy is the amount of information of a random variable”</em></strong>. Because the more uncertain we are about the outcome of a variable, the more information that variable is providing us. If we were sure or practically sure of the result of a variable, it would be worthless to measure it, as it wouldn’t be providing us with relevant information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qzaGj3FlgvtXhMLDKVJJgA.png" /><figcaption>A 3rd System similar to System 2 but with lower entropy. Recall that all probabilities must ass up to 1. Image by the author.</figcaption></figure><p>Therefore, the higher the uncertainty, the higher the information, and the higher the entropy.</p><h4>Entropy in information theory</h4><p>Information theory uses the concept of entropy to explain a very different concept than physics, <strong><em>“Entropy is the optimal average length for an encoded message”</em>.</strong></p><p>Continuing with the microstates example, suppose we would like to send the results of our experiment from our laboratory located in Madrid (Spain) across the world to the sunny beaches of Sydney (Australia). <br>First, we have to decide which code system are we going to use for writing our message (binary, hexadecimal, ASCII, …) Suppose we choose binary code for the simplicity of the example.<br>Next, we should define how we are going to represent each experiment result. Since we have 6 microstates, we could represent them as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RxELmBZze0ZaAd9gY019rQ.png" /><figcaption>Binary representation of the 6 microstates. Image by the author</figcaption></figure><p>Another possible codification could be <em>one-hot encoding</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rXt_x5OtGiAlCPINIvJ_MA.png" /><figcaption>Another possible encoding. Image by the author.</figcaption></figure><p>And there are many other possible solutions… <strong>But which one is the optimal one?</strong> <br>Imagine now that each of the message’s bits has a price, let’s say 10€ per bit, now our first codification has an average price of 30€ per experiment, whereas one-hot encoding is 60€/experiment. Therefore, we naturally want to optimize our message so that the overall price is as minimum as possible.</p><blockquote><strong>Note2</strong>. Note that I am using the word “price” instead of “cost”. This is because I will later use the term cost to refer to another concept and in this way, I avoid generating ambiguity.</blockquote><p>The optimal length to represent each microstate resulting from our experiments is what the entropy represents. <strong>We cannot get a shorter codification for our message under the entropy barrier</strong>.</p><h3>Entropy Formula</h3><p>If you look up the internet for the entropy formula, you might get a different result depending on the field you are searching for, and the reason is that many entropy formulas exist (watch this).</p><p>However, in Deep Learning we are interested in the <strong>Shannon Entropy</strong>, which comes from the Information Theory field (Shannon Entropy is also used in the statistical understanding of entropy) <br><strong><em>Why do we use it?</em></strong><em> </em>The reason will come in the next post under the section <a href="http://a/">Cross-Entropy in Deep Learning</a>, but first, let’s see where the formula derives from.</p><blockquote><strong>Note3</strong>. The reasoning to get to the entropy formula is based on Christopher Olah’s article I mentioned at the beginning of this post <a href="https://colah.github.io/posts/2015-09-Visual-Information/">[1]</a>. To get a deeper understanding I recommend you visit his website.</blockquote><h4>The Space of Codewords</h4><p>The Space of Codewords refers to the set of all possible encoded messages you can create with <em>n</em> bits of information.</p><p>So for just 1 bit, you can only represent two different <strong><em>words </em></strong>(experiment results in our example): <em>0</em> and <em>1</em>. With 2 bits, you can express four words: <em>00</em>, <em>01</em>, <em>10</em> and <em>11</em>. And so on (see table).</p><p>Then, for the binary code, we can show that <strong>with <em>n</em> representation bits we can express <em>2^n</em> words</strong> (The demonstration is trivial, just think of the number of bits as gaps to fill, for the first gap you have 2 options (<em>0 </em>or<em> 1</em>), for the second two more options again (<em>0 </em>or<em> 1</em>), and so on for the rest of the bits. Now, to calculate all the options you have, you multiply <em>2 · 2 · 2 · … · 2 = 2^n </em>options (binary strings) to form with <em>n</em> bits)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AsQC-qwODW1VDKH1f-FUrQ.png" /><figcaption>Space of Codewords for 3 bits. The last column shows the number of possible combinations (8), starting from the top: 000, 001, … Image by the author.</figcaption></figure><p>If we operate and clear n (number of bits), we can see that to represent 6 words in binary we need to use 3 bits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/387/0*Y8Z9bocb_uHMp5PB.png" /><figcaption>Clearing n and computing the minimum number of bits needed to express the 6 experiments (we cannot have 2.585 bits so we must round up). Image by the author.</figcaption></figure><p>But, this only holds if we use the same number of bits to represent every word. What would happen then if we use a <strong>variable length code</strong>?</p><p>Recalling our encoded experiment results, we still have to figure out the cheapest way to send the messages. So instead of having: <em>000</em>, <em>001</em>, <em>010</em>, <em>011</em>, <em>100, 101; </em>we might try with the representation <em>00</em>, <em>01</em>, <em>100</em>, <em>101</em>, <em>110, 111</em>.</p><p>Now the average price of the new encoding system has decreased to ~26.67€/experiment instead of 30€.</p><p>And even more important, what if each of experiments 100, 101, 110 and 111 occurs only 5% of the time? Then the average message price would be lower since our messages will usually be formed by 2-bit words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/0*aO5qloEAscvBfIFT.png" /><figcaption>Average Price of variable length encoding system. 80% of the time we’ll use 20€ words whereas only 20% the 30€ words. Image by the author.</figcaption></figure><h4>The Cost Function</h4><p>In order to achieve an optimal solution, <strong>we need an objective function to minimize</strong>.</p><p>(Remember when I said that I used the word price instead of cost because I wanted to introduce another different term and not create ambiguity? Now it’s the time) We define the <strong><em>cost </em></strong>as the portion of the Space of Codewords that we sacrifice when coding a word with n bits. When I state that the cost of using 2 bits to represent a word is 1/4, that means that I am losing 1/4 of my Space of Codewords.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ms3hS55KqSmU9L34SkXJA.png" /><figcaption>Using 00 to represent a word prevents the use of all strings beginning with 00 (grey cells). This translates into a cost of 1/4. Image by the author</figcaption></figure><p>The objective function (from now on I will refer to it as the<em> </em><strong><em>Cost function</em></strong>) must express the cost of using a code with a length of X bits to express a result (i.e. the function takes the length in bits of a code and outputs the cost of using that code in terms of the Space of Codewords)</p><p>Looking again at the table representing the Space of Codewords and the cost of using variable length codes, you might be able to get the following cost function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/0*aicBlKpEzd668B8o.png" /><figcaption>Cost function for using x bits to encode a word. The result is expressed as a portion of the Space of Codewords. Image by the author</figcaption></figure><p>Analyzing the properties of the cost function <em>C(l):</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/0*sFc7LeZ_zzOSLNeY.png" /><figcaption>The Cost Function is bounded between 0 and 1. A code of length 0 sacrifices the whole Space of Codewords since we are sending an empty message which can just represent one result. Image by the author</figcaption></figure><h4>Achieving the minimum</h4><p>Finally, we must decide which is the optimal length to represent a word depending on the cost. We have the whole Space of Codewords to work for us, thus our budget is 1, and we have to find the shortest possible codes such that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/0*I7fR3dmWyPMoigCc.png" /><figcaption>The function that defines our problem. Image by the author</figcaption></figure><p>Recall from previously that we are interested in using a <strong>variable length code depending on the probabilities of each result to occur</strong> (thus representing l<em>1, …, l6</em> as different variables and not just l)</p><p>Intuitively, we want to assign a <strong>shorter </strong>code (and therefore a higher cost in the Space of Codewords) to the most frequently occurring words, and a <strong>longer </strong>code (lower cost) to words that are less likely to appear. <strong>But how do we quantify <em>shorter </em>and <em>longer</em>?</strong> Is it 3 bits short or long?</p><blockquote>The answer is Probability</blockquote><p>Think of it this way: We have a Cost function that must add up to 1 (otherwise we are wasting Space of Codewords) and we want to assign to each of the terms a cost proportional to their probability that, <strong>add up to one! </strong>Same as the cost function, the sum of the probabilities of the experiments is 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/0*WaINlsC2WwLuNIzd.png" /><figcaption>The lowest cost of a word of length L is the probability of occurrence of the word itself P(x) Image by the author</figcaption></figure><blockquote><strong>Note4</strong>. Although the statement I’ve just made needs mathematical proof, I don’t want to overload this post. If you are interested in reading it, <a href="https://colah.github.io/posts/2015-09-Visual-Information/#optimal-encodingshttps://colah.github.io/posts/2015-09-Visual-Information/#optimal-encodings">here </a>is a link to a very intuitive and visual explanation.</blockquote><p>Finally, we just have to operate the terms and write the length <em>l </em>as a function of the probability <em>P(x)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/0*48_zpdcbl-ru9pQ1.png" /><figcaption>Length of a coded word as a function of its probability. Image by the author.</figcaption></figure><blockquote><strong>Note5</strong>. When you substitute the probability value (continuous in [0,1]) into the formula above, you will see that the result of length “l “ is a decimal value. But that can’t be right! How can we construct a code with 1.6145 bits? Well, I’ll leave that explanation to <a href="https://colah.github.io/posts/2015-09-Visual-Information/#fractional-bits">Christopher</a> but the short answer is, we can’t, we have to round up, although there is a way to do it better…</blockquote><h4>And the entropy</h4><p>We are able to compute the optimal (shortest) length of a binary encoded word using its probability of occurring. However, there is one last step to compute entropy since <em>“Entropy is the </em><strong><em>AVERAGE </em></strong><em>optimal length of an encoded message”, </em>and that step is to compute the average over all word lengths of our message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/0*Mm2SJBYAGw0gzrRz.png" /><figcaption>Entropy formula for discrete random variable X. Image by the author.</figcaption></figure><p>Developing the log term:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/0*wiLuBhFJGBn_0pBs.png" /><figcaption>Most common representation of the entropy formula. Image by the author</figcaption></figure><blockquote><strong>Note6</strong>. Usually, you will see the entropy formula using log base 10. In most cases, this doesn’t matter since log_10(x) = log_2(x) / log_2(10) which essentially is multiplying by a constant 1/log_2(10). This change of base only holds for optimization or comparative methods where the constants are ignored, but it isn’t the same computing the optimal average length of a message using base 2 (each bit of information only takes 2 possible values) as using base 10. Read <a href="https://www.khanacademy.org/math/algebra2/x2ec2f6f830c9fb89:logs/x2ec2f6f830c9fb89:change-of-base/a/logarithm-change-of-base-rule-intro">this</a> for more information.</blockquote><h3>The final result</h3><p>So far I’ve presented 3 different (but related) conceptions of entropy, and derived its expression from scratch using mathematics and visual representations.</p><p>However, there is one remaining task. WE HAVE TO SOLVE OUR PROBLEM OF OPTIMALLY ENCODING THE EXPERIMENT RESULTS FOR OUR LABORATORY!</p><p>So we just have to plug our experiment’s probability distribution into the entropy formula and compute the terms:</p><blockquote><strong>Note7</strong>. I’m going to use the last probability distribution plotted in the image with the title System 3, but remember that this works the same for any discrete random variable.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*6uyZrfACjqEBFLZ1.png" /><figcaption>Computing the optimal average length for the encoding system of System 3.</figcaption></figure><p>Remember that this is an average; thus, some of the experiments might be expressed using 1 bit, others using 2 bits or perhaps 3… Entropy just tells us the average minimum, but it doesn’t reflect how each message should be encoded. <strong>Note that I am using log base 2 to compute entropy since we are using binary encoding</strong> (using log base 10 yields a different result that is weighted by a constant 1/log_2(10) and doesn’t reflect the real result)</p><p>I hope this post has helped you to understand better Entropy. In the next post, I will talk about Cross-entropy and why we use it in Artificial Intelligence models. Finally, I will explain in which problems we apply it and how it can be interpreted.</p><h3>References</h3><p><strong>[1]</strong> Visual Information Theory. Christopher Olah. 2015. <a href="https://colah.github.io/posts/2015-09-Visual-Information/">https://colah.github.io/posts/2015-09-Visual-Information/</a></p><p><strong>[2]</strong> Is ENTROPY Really a “Measure of Disorder”? Physics of Entropy EXPLAINED and MADE EASY. Parth G. 2020. <a href="https://www.youtube.com/watch?v=mg0hueOyoAw">https://www.youtube.com/watch?v=mg0hueOyoAw</a></p><p><strong>[3]</strong> A better description of entropy. Steve Mould. 2016. <a href="https://youtu.be/w2iTCm0xpDc">https://youtu.be/w2iTCm0xpDc</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e787f885c1c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-do-we-use-cross-entropy-in-deep-learning-part-1-e787f885c1c">Why do We use Cross-entropy in Deep Learning — Part 1</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Saving Multiple Images in Tensorboard with tf.summary.image]]></title>
            <link>https://medium.com/data-science/saving-multiple-images-in-tensorboard-with-tf-summary-image-15df9c11d6c9?source=rss-e77c10fd9715------2</link>
            <guid isPermaLink="false">https://medium.com/p/15df9c11d6c9</guid>
            <category><![CDATA[tensorboard]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Gabriel Furnieles]]></dc:creator>
            <pubDate>Wed, 19 Oct 2022 17:26:39 GMT</pubDate>
            <atom:updated>2022-10-19T17:26:39.068Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to store multiple images within Tensorboard to see the evolution of your Neural Network. Case application to the GAN model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4httPbpz-KxRcGCAmC8wgg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@sannisahil?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Sanni Sahil</a> on <a href="https://unsplash.com/collections/10458041/dashboard?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a>. Edited by author.</figcaption></figure><p>Recently I’ve been developing an image-generating model and have had the need to save several images in <strong>Tensorboard</strong> to see how the model evolved in each epoch.</p><p>After a long search and several documentation files, I finally came up with the result that you can see in the image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*et6Y1i4ZUdyV7Kr6tVIPww.png" /><figcaption>Screenshot taken from Tensorboard project. Top image is the input given to the Neural Network, middle image is the output of the network (the translated image) and bottom is the ground truth (what the model was supposed to output). The progress bar that is on top of the images represents the epoch when the image was generated. Image by the author</figcaption></figure><p>In the example shown above the model attempts to “translate” a given image into an art painting by Monet, using the Pix2Pix model [<a href="https://arxiv.org/pdf/1611.07004">link to paper</a>]. You can find the dataset you’re looking at in the Kaggle repository [<a href="https://www.kaggle.com/datasets/shcsteven/paired-landscape-and-monetstylised-image">link</a>]</p><p><strong>Note1</strong>. The number of epochs with which the model has been trained is very low since I just wanted to show the dashboard, so the results look really poor.</p><h3>Why use Tensorboard</h3><p><strong>Tensorboard </strong>is a visualization tool that provides the framework to analyze the performance of a neural network. It can be integrated using the <em>Tensorflow </em>and <em>Pytorch </em>libraries, generating an interface on a local port of your machine where to project metrics such as the evolution of the loss function or to track customized ones, such as the generated images as in this case.</p><p>These visualizations are <strong>essential </strong>for the development stage of a neural network model, allowing us to detect key problems such as <strong>overfitting/underfitting</strong>, measure and <strong>compare multiple models</strong>, or provide insights on what changes might improve the overall performance<strong> (hyperparameter tunning)</strong><br>For instance, analyzing the generated images on each epoch can help us to detect weaknesses in the generator such as the generation of grainy patterns or stagnation by mode collapse, but those are topics for another article.</p><p>For more information about Tensorboard, you can check the documentation <a href="https://www.tensorflow.org/tensorboard/get_started">here</a>.</p><h3>Code</h3><p>I don’t want to waste your time so here is the code to save the images:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8067b8f97953c12dc53608d17c53e5a0/href">https://medium.com/media/8067b8f97953c12dc53608d17c53e5a0/href</a></iframe><p>Next, to launch a training using the custom callback:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/33c68cd282c267d83118525e391dff45/href">https://medium.com/media/33c68cd282c267d83118525e391dff45/href</a></iframe><h4>Code explanation</h4><p>For those who just don’t copy and paste the code, let me explain what happens when you run the above code.</p><p><strong>1st</strong> We need to create a keras-callbacks subclass where we can define our custom callback.<br>Callbacks are functions that are executed at a specified frequency during training [<a href="https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback">see doc</a>]. To tell Tensorflow to use Callbacks when training we just have to pass them as arguments in the form of a list object (see step 5th)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*JofHt_L7FArQuX2NZFUFEw.png" /><figcaption>Screenshot taken from the code. Image by the author</figcaption></figure><p>In our case, the callback class receives as arguments a batch of images from both training and validation sets, the model’s generator (so we can generate images), and the path (logdir) where to store the images to display after in Tensorboard.</p><p>We also define the instance variable self.writer, which creates a summary file writer for the given log directory (where we are going to store the information for later display on Tensorboard)</p><p><strong>2nd </strong>We define de class method on_epoch_end which, as its name indicates, will be executed after every epoch.</p><p><strong>Note2</strong>. The arguments that receive this method are prefixed by Tensorflow, so don’t try to change them.</p><p>Within the on_epoch_end method, we must also define a function to generate the images (this way the code looks cleaner and better organized).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*Iu9sev_TvPMOVakR_KdQxw.png" /><figcaption>Screenshot taken from the code. Image by the author</figcaption></figure><p>The function generate_imgs takes a set of images (a TakeDataset element from Tensorflow.data) and the generator (<em>g</em>), and returns a list of 3-display images concatenating vertically the input image x , the translated image by the model out , and the ground truth y. If we don’t concatenate the images these will be displayed on different cards on Tensorboard.</p><p><strong>Note3</strong>. Before concatenating we have to remove the batch dimensions using the function tf.squeeze() to prevent exceptions.</p><p><strong>3rd</strong> Next, we save the images using tf.summary.image()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*uEZk3ZDqP28-Nl3n1EKX1g.png" /><figcaption>Screenshot taken from the code. Image by the author</figcaption></figure><p>The first line self.writer.as_default() tells Tensorflow to store the next operations in the same graph (the <em>self.writer</em> graph) so all the images generated after each epoch by the callback will be recorded in the same file [check <a href="https://www.tensorflow.org/api_docs/python/tf/Graph#as_default">doc</a>, <a href="https://stackoverflow.com/questions/56421878/why-do-i-need-as-default-to-write-instructions-in-a-tensorflow-graph">link</a>]<br>Next, the tf.name_scope() function adds to the name of every image the prefix “<em>train/</em>” or “<em>val/</em>”, so the train and validation generated images are saved in different folders in your working directory (<strong>within Tensorboard this is reflected as different sections</strong>, but in reality both files are named the same and belong to the same summary file)</p><p><strong>Note4</strong>. In this case, I’ve defined the name of the images as the scope again, so they are going to be named “<em>train/train/</em>” or “<em>val/val</em>”, but for further projects, I recommend changing it.</p><p><strong>4th</strong> Initialize the class. <br>Back to our train.py file where we’ve created our model, we initialize the class by calling it (line 9). For the project, I decided to take 4 images from the training set and other 4 images from the validation, each image from a different batch. Next, I specify the generator (pix2pix is a <em>keras.model</em> so I can call the generator as a method) and the logdir where to save the summary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*R0fqiu5-7WudpigIrRd6TA.png" /><figcaption>Screenshot taken from the code. Image by the author</figcaption></figure><p><strong>Note5</strong>. Someone might think that if we are passing the generator as an argument, then during training their weights are not going to be updated and the results wouldn’t be correct. However, Tensorflow manages to do exactly that, since we are passing a class instance pix2pix.g that points to the created class pix2pix i.e. when pix2pix updates its weights, it is applied to all its instances.<br>I’ve personally checked this (and you can do it too if you don’t believe me) by adding the line within the custom callback:</p><p>print(self.g.get_weights()[0][0][0][0])</p><p>The result is a print of the first neuron weights every epoch, so you can notice the modifications.</p><p><strong>5th</strong> Train the model using the custom callback</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*5smgyPyWy97LYm8T1LWYdA.png" /><figcaption>Screenshot taken from the code. Image by the author</figcaption></figure><h3>Conclusions</h3><p>Being able to visualize the evolution of your model during the training phase is crucial for good development and provides key insights on what direction to take to increase the accuracy and performance of the model.</p><p>However, it’s also important to <strong>take into account the costs</strong> of evaluating our custom metrics, since time is an important fact when it comes to training models. In this project example, unpacking the images and testing the generator can take a few extra minutes depending on your hardware and the image size, considerably slowing the training.</p><p>A good solution to this problem is establishing a <em>frequency variable</em> that will be compared with the epoch number within the custom callback class so that if epoch % frequency == 0 then we test the generator and save the results.</p><p>I hope this post has been useful and helpful, for more articles on Artificial Intelligence and Data Science you can check my other posts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15df9c11d6c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/saving-multiple-images-in-tensorboard-with-tf-summary-image-15df9c11d6c9">Saving Multiple Images in Tensorboard with tf.summary.image</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>