<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[AI In practice - Medium]]></title>
        <description><![CDATA[AI related articles covering everyhing from model development and experimentation to deployment to Production environment - Medium]]></description>
        <link>https://medium.com/ai-in-practice?source=rss----dfcc9a03fc8d---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>AI In practice - Medium</title>
            <link>https://medium.com/ai-in-practice?source=rss----dfcc9a03fc8d---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 00:32:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/ai-in-practice" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[OpenAI or Open-Source? Choosing the Right Embedding Model for Polish Text]]></title>
            <link>https://medium.com/ai-in-practice/openai-or-open-source-choosing-the-right-embedding-model-for-polish-text-7840c2a16b46?source=rss----dfcc9a03fc8d---4</link>
            <guid isPermaLink="false">https://medium.com/p/7840c2a16b46</guid>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[polish]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[hugging-face]]></category>
            <dc:creator><![CDATA[Andrey Pigarev]]></dc:creator>
            <pubDate>Sat, 18 Jan 2025 17:10:03 GMT</pubDate>
            <atom:updated>2024-12-18T20:29:23.838Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*-t2VYu1-CCgZLju1IOI8gw.png" /></figure><h3>Introduction</h3><p>In the last few years, OpenAI has seen tremendous growth in its usage. Today, it offers not only access to large language models (LLMs) like GPT-4 for chat completions but also embedding models. While these models work exceptionally well for English texts, their performance on other languages — such as Polish — is less clear.</p><p>In this article, I aim to evaluate the effectiveness of OpenAI’s embedding models compared to open-source alternatives from Hugging Face in the task of embedding Polish text.</p><p>The article will cover the following:</p><ul><li>A brief introduction to embeddings, the Retrieval-Augmented Generation (RAG) approach, and the role of embeddings within it.</li><li><strong>Data Curation:</strong> The process of gathering and cleaning a dataset for the evaluation.</li><li><strong>Data Augmentation:</strong> Generating questions to enhance the dataset.</li><li><strong>Model Selection:</strong> OpenAI and Hugging Face embedding models used for evaluation</li><li><strong>Vector Database:</strong> A database to efficiently store and search embeddings.</li><li><strong>Performance Evaluation: </strong>Comparing the performance of OpenAI and Hugging Face models for embedding Polish text.</li><li><strong>Other considerations:<br>- Speed:</strong> Evaluating the time required for embedding generation.<br>- <strong>Cost:</strong> Measuring the cost of generating embeddings.</li><li><strong>Conclusions:</strong> Key findings and insights from the evaluation.</li><li><strong>Caveats / Next steps</strong></li></ul><p>All code is available in my <a href="https://github.com/AndreyProfDev/articles_materials">GitHub repository</a></p><p>Analysis can also be run on <a href="https://colab.research.google.com/drive/1odBFSu7XftsYlDvLZaXjY4nkaAfAupye">Google Colab</a></p><h3>Brief Introduction to Embeddings and RAG</h3><h4>What is an Embedding?</h4><p>Embedding is a method of converting free text into a machine-understandable numerical representation. Essentially, it is a unique set of numbers that represents:</p><ul><li>A single word (e.g., in word-level embeddings like GloVe or Word2Vec).</li><li>An entire text (e.g., in sentence or document embeddings from encoder models like BERT or RoBERTa).</li></ul><p>A key property of embeddings is that they represent semantic similarity — similar words or sentences have embeddings that are close to each other in the vector space.</p><p>To illustrate this, imagine an embedding represented by just two numbers [x, y] (real embeddings typically have hundreds of dimensions). Suppose we embed the words “dog,” “cat,” and “brick.” If we plot these embeddings as points on a 2D graph, the points for “dog” and “cat” would be much closer to each other than to “brick.” This closeness reflects the semantic similarity between “dog” and “cat” compared to “brick.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*ohB9559R0XucF0KBGXl2Yg.png" /></figure><h4>What is RAG?</h4><p>The similarity property of embeddings allows them to be used for finding similar words, sentences, or articles within a pre-populated knowledge base. This capability is at the core of Retrieval-Augmented Generation (RAG).</p><p>Let’s break it down with an example:</p><p>Imagine you have a knowledge base (in this case, a vector database) populated with pages from Wikipedia. For each page, you calculate its corresponding embedding vector. Now, suppose you want to answer the question, “What year was the airplaneinvented?” The retrieval module in your system will:</p><ul><li><em>Transform your question into an embedding vector </em>using the same embedding model.</li><li><em>Compare this vector to the vectors in the database</em> to find the most similar one. In practice, search algorithms, such as approximate nearest neighbor (ANN) methods, often reduce the number of comparisons needed.</li><li><em>Retrieve the closest match</em> and fetch the corresponding text from the database (<em>The Wright brothers’ invention of the airplane, starting with the 1903 Wright Flyer, truly changed the world</em>).</li><li><em>Pass the retrieved text along with the original question to a large language model</em> (LLM), which generates the final answer.</li></ul><h3>Data Curation</h3><p>For the input dataset, I selected freely available articles from the Polish-language Wikipedia. Wikipedia provides a service that allows users to export pages belonging to specific categories. These export tools are available in both:</p><ul><li><a href="https://en.wikipedia.org/wiki/Special:Export">English Wikipedia (Export pages)</a></li><li><a href="https://pl.wikipedia.org/wiki/Specjalna:Eksport">Polish Wikipedia (Eksport stron)</a>.</li></ul><p>The output is provided as an XML document in MediaWiki format. To make the data usable, I wrote a custom parser to:</p><ol><li>Convert the XML data into plain text.</li><li>Split the content into individual articles.</li><li>Further divide each article into sections for better granularity.</li></ol><p>I intentionally avoided using pre-cleaned datasets, such as <em>ipipan/polqa</em> from Hugging Face, because my goal was to go through the entire process of model evaluation — starting with raw data preparation.</p><h3>Data Augmentation</h3><p>From the previous step, I obtained 1,653 unique sections extracted from Polish Wikipedia. However, at this point, I didn’t have any questions to pair with the text for embedding evaluation. To address this, I used OpenAI GPT-4 to generate artificial questions, due to its good performance for this task based on the experience.</p><p>This approach might give some advantage to OpenAI-based embedding models since the questions are generated by the model from the same model family where some evaluated embedding models reside. However, I decided to proceed with this method for simplicity — we’ll see how this potential bias impacts the results.</p><p>The prompt used to generate questions is presented below (English version, please see my GitHub repository for Polish version):</p><pre>You are helpful assistant who generates questions for a given text.<br>Based on the provided text you generate five questions.<br>It should be possible to answer the questions based on the text.<br>Don&#39;t generate direct questions.<br>Question are self-contained and general, without explicitly mentioning a context, system, course, or extract.<br>You provide your response in JSON format with the following structure:<br>{{<br>questions&quot;: [<br>&quot;Question 1&quot;,<br>&quot;Question 2&quot;,<br>&quot;Question 3&quot;,<br>]<br>}}</pre><p>The generated questions were then paired with their respective sections to prepare the dataset for model assessment.</p><h3>Model Selection</h3><p>For the evaluation I used Open AI models and opens source Hugging Face models. Below is an overview of the selected models:</p><h4>OpenAI Models</h4><p>OpenAI provides embedding models through its API (see <a href="https://platform.openai.com/docs/guides/embeddings/embedding-models">Vector embeddings — OpenAI API</a> for details). They vary in embedding dimension and associated cost.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7474bcd0d18ad3509bcd68cd87efde7f/href">https://medium.com/media/7474bcd0d18ad3509bcd68cd87efde7f/href</a></iframe><h4>Hugging Face Models</h4><p>I’ve identified several Hugging Face open-source models fine-tuned for Polish text (see <a href="https://huggingface.co/models?pipeline_tag=sentence-similarity&amp;sort=downloads&amp;search=polish">Models — Hugging Face</a> for details). The selected models include:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6da03647a7aae8beb365fe45d2c39146/href">https://medium.com/media/6da03647a7aae8beb365fe45d2c39146/href</a></iframe><h3>Vector Database</h3><p>To efficiently store and search embeddings, a special type of storage is required, often called a vector database. For this exercise, I selected Faiss from Meta because it is fast and operates as a standalone database.</p><p>Once populated with embeddings, Faiss retrieves the closest match using the following operation:</p><p><em>j = argmini||x — xi||</em></p><p>Where:</p><ul><li><em>||</em>*<em>||</em> represents the Euclidean distance (L2).</li><li><em>x</em> is the query embedding.</li><li><em>xi </em>are the stored embeddings in the database.</li><li><em>j</em> is the index of the closest embedding.</li></ul><h3>Performance Evaluation</h3><p>Before diving into the results, let’s summarize the approach used to evaluate the performance of the models. For each section from Wikipedia:</p><ol><li>Five questions are generated for each section using the data augmentation process described earlier.</li><li>Each question is transformed into its corresponding embedding using the selected models.</li><li>The generated embeddings are searched within the vector database (Faiss) to retrieve the closest matches, considering the top-1, top-5, and top-10 results.</li><li>For each retrieved match, we check whether the text corresponding to the match is the original section from Wikipedia.</li></ol><p>Using the results generated from this process, we evaluate the performance of the models from different perspectives. Let’s dive in.</p><h3>Average number of questions correctly answered</h3><p>This metric calculates the average number of questions for which the correct section is retrieved. For example, let’s assume:</p><ul><li>There are only three Wikipedia sections in the dataset.</li><li>For the first section, <strong>2 out of 5</strong> questions matched the section.</li><li>For the second section, <strong>5 out of 5</strong> questions matched the section.</li><li>For the third section, <strong>3 out of 5</strong> questions matched the section.</li></ul><p>The average number of correctly answered questions is calculated as:</p><p>Average = (2+5+3)/3 = 3.33</p><p>Below are the results for our selected models:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*eKzTVDlxCBmrL5aEoOT0Sg.png" /></figure><p>From open-source models <em>KartonBERT-USE-base-v1</em> demonstrates very impressive results. It is on par with <em>Open AI Large </em>embedding model.</p><h3>Total number of questions correctly answered</h3><p>This metric calculates the total number of questions for which the correct section is retrieved across all Wikipedia sections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*qo1om08vwWD9zuim_YG8tw.png" /></figure><p>The results show that the best open-source model, <em>KartonBERT-USE-base-v1</em>, performs on par with OpenAI’s Large model.</p><h3>Other considerations</h3><h4>Embedding generation cost</h4><p>This metric calculates the total cost required to generate embeddings for all questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*OYXEUWrgaHdkcMFYgmK9Ug.png" /></figure><p>It’s important to note that embeddings generated using Hugging Face models were run on a desktop PC, resulting in zero cost. As such, they are excluded from this evaluation.</p><h4>Embedding generation speed</h4><p>This metric measures the total time required to generate embeddings for all sections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*rv2j94JOzecQK0dpKQ__Zw.png" /></figure><p>As shown in the results, the time required to generate embeddings with OpenAI models is more than 50 times higher than with open-source models.</p><h3>Conclusions</h3><p>OpenAI models demonstrate strong performance when compared to open-source alternatives. They represent a reasonable option if security concerns are not an issue and if there is no access to a dedicated GPU-powered machine (or if embedding generation speeds on CPU-only is acceptable).</p><p>At the same time, it is possible to achieve comparable performance using open-source models, such as <em>KartonBERT-USE-base-v1</em>, which offers a cost-effective and efficient alternative.</p><h3>Caveats / Next steps</h3><p>The following are potential caveats that might introduce biases into the analysis. I plan to explore and address these in future articles:</p><ul><li><strong>Potential Dataset Bias</strong><br> All conclusions about performance are based on the selected dataset (1,683 sections from Polish Wikipedia). Using a different dataset could yield different results.<br> In particular for some models good performance might be due to text from Wikipedia being part of model fine-tuning dataset.</li><li><strong>Potential Question Generation Bias:<br> </strong>As mentioned earlier, the questions were generated automatically using OpenAI’s GPT-4 model, which may introduce biases into the evaluation. To improve the precision of the analysis, it would be better to use questions and answers prepared by humans, such as the <em>ipipan/polqa</em> dataset.</li><li><strong>Sequence Length Limitation:</strong><br> The maximum sequence length for open-source models is significantly lower than that of OpenAI models. By default, Hugging Face models truncate text that exceeds this threshold, which might contribute to their lower performance. This issue is a subject to further investigation</li><li><strong>Text category specific performance<br> </strong>The performance charts above reflect results across all data. However, the performance could vary significantly across different categories of text (e.g., medicine, religion). A more detailed category-based analysis would provide additional insights.</li><li><strong>Batch Processing for Embeddings:</strong><br> Conclusions about embedding generation speed are based on generating embeddings one by one. Both Hugging Face and OpenAI libraries offer batch processing capabilities, which could significantly reduce generation time, particularly for OpenAI models where the current approach includes network latency for each generation request.</li><li><strong>Other Hugging Face models<br> </strong>There are many more open-source retrieval models trained specifically for polish language. As an example, see models from <a href="https://huggingface.co/spaces/sdadas/pirb">Polish Information Retrieval Benchmark</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7840c2a16b46" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-in-practice/openai-or-open-source-choosing-the-right-embedding-model-for-polish-text-7840c2a16b46">OpenAI or Open-Source? Choosing the Right Embedding Model for Polish Text</a> was originally published in <a href="https://medium.com/ai-in-practice">AI In practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>