<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lyon NLP on Medium]]></title>
        <description><![CDATA[Stories by Lyon NLP on Medium]]></description>
        <link>https://medium.com/@lyon-nlp?source=rss-2fff37de73cd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-mMcqxSlMQBmXcHwGSy32g.png</url>
            <title>Stories by Lyon NLP on Medium</title>
            <link>https://medium.com/@lyon-nlp?source=rss-2fff37de73cd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:08:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lyon-nlp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MTEB Leaderboard : User guide and best practices]]></title>
            <link>https://medium.com/@lyon-nlp/mteb-leaderboard-user-guide-and-best-practices-32270073024b?source=rss-2fff37de73cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/32270073024b</guid>
            <category><![CDATA[benchmark]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[french]]></category>
            <category><![CDATA[embedding]]></category>
            <dc:creator><![CDATA[Lyon NLP]]></dc:creator>
            <pubDate>Wed, 13 Mar 2024 10:21:50 GMT</pubDate>
            <atom:updated>2024-03-13T10:21:50.727Z</atom:updated>
            <content:encoded><![CDATA[<h3>MTEB Leaderboard : User guide and best practices</h3><p><a href="https://github.com/MathieuCiancone">Mathieu Ciancone</a>, <a href="https://github.com/imenelydiaker">Imene Kerboua</a>, <a href="https://github.com/schmarion">Marion Schaeffer</a>, <a href="https://github.com/GabrielSequeira">Gabriel Sequeira</a>, <a href="https://github.com/wissam-sib">Wissam Siblini</a></p><p>MTEB [1] is a multi-task and multi-language comparison of embedding models. It comes in the form of a <a href="https://huggingface.co/spaces/mteb/leaderboard">leaderboard</a>, based on multiple scores, and only one model stands at the top! Does it make it easy to choose the right model for your application? You wish!<br>This guide is an attempt to provide tips on how to make clever use of MTEB. As our team worked on making the French benchmark available [2], the examples will rely on the French MTEB. Nonetheless, those tips apply to the entire benchmark.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yPPuKTe1moxWWTZtrwZIAw.png" /><figcaption>French MTEB leaderboard</figcaption></figure><h3>Don’t let all those scores abuse you</h3><p><em>MTEB is a leaderboard. It shows you scores. What it doesn’t show you? Significance.</em></p><p>While being a great resource for discovering and comparing models, MTEB might not be as straightforward as one might expect. As of today (1st of March 2024), many SOTA models have been tested, and most of them display close average scores. For the French MTEB, these average scores are computed on 26 different tasks (and 56 for English MTEB!) and no standard deviation comes with it. Even though the top model looks better than the others, the score difference with a model that comes after it might not be significant. One can directly get the <a href="https://huggingface.co/datasets/mteb/results/tree/main">raw result</a>s to compute statistical metrics. As an example, we performed critical difference tests and found that, with a p-value of 0.05, the current 9 top models in the French MTEB leaderboard are statistically equivalent. It would require even more datasets to perceive statistical significance.</p><h3>Dive into data</h3><p><em>Do not just look at the average scores of models on the task you are interested in. Instead, look at the individual scores on the datasets that best represent your use case.</em></p><p>Let’s say you are interested in doing retrieval for an application in the legal domain. The legal language is very specific, and models that have been trained on general-purpose data might perform poorly in your application. Consequently, instead of just taking the best model of MTEB for the retrieval task, look specifically at the tasks that involve datasets containing legal-related content. Indeed, the scores on datasets such as <a href="https://huggingface.co/datasets/lyon-nlp/mteb-fr-retrieval-syntec-s2p"><em>Syntec</em></a> or <a href="https://huggingface.co/datasets/maastrichtlawtech/bsard"><em>BSARD</em></a> [3] might be more representative of the models’ performance on your data. No need for a deep dive into the dataset’s data, most of the time the dataset’s card and a short preview will tell you everything you need.</p><h3>Consider the model’s characteristics</h3><p><em>Using the model displaying the best average score for your application might be tempting. However, a model comes with its characteristics, leading to its usage constraints. Make sure those constraints match yours.</em></p><p>Before going for the top-of-the-list model, check the model’s characteristics: How many tokens does its context window handle? What is the size of the model? Has it been trained on multilingual data? Indeed, those features may conflict with the constraints of your application. For example, here are a few things you might want to consider:</p><ul><li><strong>compute capacity</strong>: If you need to run experiments on a single-GPU laptop, do not select 7B parameters models, smaller models might be more adapted.</li><li><strong>input token sequence length</strong>: If you need to perform sentence similarity on small sentences, no need for a context window of 32k tokens, the 128 token window of <a href="https://huggingface.co/dangvantuan/sentence-camembert-large"><em>sentence-camembert-large</em></a> might be enough.</li><li><strong>storage capacity/latency</strong>: If you hesitate between the base and large flavours of <a href="https://huggingface.co/intfloat/multilingual-e5-large"><em>multilingual-e5-large</em></a> [4], ask yourself if the 1 point difference on retrieval average score is worth going from a 1.1 GB to a 2.2 GB model and a 728 to a 1024 output embedding size.</li></ul><h3>Do not forget MTEB is a leaderboard…</h3><p><em>And as leaderboards sometimes do, it encourages to compete without following the rules.</em></p><p>Indeed, keep in mind that many providers want to see their models on top of that list and, being based on public datasets, some malpractices such as data leakage or overfitting on data could bias MTEB evaluation. So if you find a model that seems good to you, look further into the model’s development and training setup: many contributors make the effort to provide a detailed model card specifying the model’s training data, purpose or licence. Look into those details, pick the model you are interested in as well as 2 or 3 secondary choices, and run some tests on your data to see if it still fits.</p><h3>How to evaluate your model?</h3><p><em>You might want to know how your custom model performs against other models in the leaderboard.</em></p><p>It’s easy! Just follow the steps described in <a href="https://huggingface.co/blog/mteb#benchmark-your-model">this blog</a>. By running this, you should see you model performance. And if you want, feel free to add the results to your model card in order to add them to the leaderboard.</p><p>Thank you for reading and happy MTEB browsing! 🤗</p><h3>Bibliography</h3><p><strong>[1]</strong> Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive Text Embedding Benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics.</p><p><strong>[2]</strong> [<a href="https://huggingface.co/blog/lyon-nlp-group/french-mteb-datasets](https://huggingface.co/blog/lyon-nlp-group/french-mteb-datasets)">https://huggingface.co/blog/lyon-nlp-group/french-mteb-datasets](https://huggingface.co/blog/lyon-nlp-group/french-mteb-datasets)</a></p><p><strong>[3]</strong> Antoine Louis and Gerasimos Spanakis. 2022. A Statutory Article Retrieval Dataset in French. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6789–6803, Dublin, Ireland. Association for Computational Linguistics.</p><p><strong>[4]</strong> Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder and Furu Wei. Text Embeddings by Weakly-Supervised Contrastive Pre-training. ArXiv abs/2212.03533 (2022): n. pag.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32270073024b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extending the Massive Text Embedding Benchmark to French: the datasets]]></title>
            <link>https://medium.com/@lyon-nlp/extending-the-massive-text-embedding-benchmark-to-french-the-datasets-ff9dcb700d9e?source=rss-2fff37de73cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff9dcb700d9e</guid>
            <category><![CDATA[benchmark]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[french-language]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Lyon NLP]]></dc:creator>
            <pubDate>Fri, 12 Jan 2024 12:31:50 GMT</pubDate>
            <atom:updated>2024-03-13T10:19:37.564Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://github.com/MathieuCiancone">Mathieu Ciancone</a>, <a href="https://github.com/imenelydiaker">Imene Kerboua</a>, <a href="https://github.com/schmarion">Marion Schaeffer</a>, <a href="https://github.com/GabrielSequeira">Gabriel Sequeira</a>, <a href="https://github.com/wissam-sib">Wissam Siblini</a></p><h3>Introduction</h3><p>With the recent boom of natural language applications, the ability to select a method that generates high-quality text representations has become crucial. To help with this, the Massive Text Embedding Benchmark (MTEB) [1] was introduced. It allows the evaluation and comparison of text embeddings methods on various NLP tasks and datasets. <br>An embedding is a dense vector representation that captures the semantic meaning of a text and can be used for downstream NLP tasks such as text classification, information retrieval, machine translation, etc.<br>MTEB originally compared 33 different models on 8 different tasks : bitext mining, classification, pair classification, retrieval, reranking, clustering, summarization and semantic textual similarity. Overall, it gathered 58 datasets across tasks, most of them in English.</p><p>We extend this work to the French language. The project is available here 👉 <a href="https://github.com/Lyon-NLP/mteb-french](https://github.com/Lyon-NLP/mteb-french).">https://github.com/Lyon-NLP/mteb-french.</a> In order to compare embeddings obtained from texts in French, we identified 14 relevant datasets and created 3 new ones targeting the set of tasks used in MTEB. Of course, these datasets can also be used for a wide range of other applications.</p><p>This first article of a series focuses on presenting these datasets, their characteristics and the goal behind each task. By bringing all the information together in one place, we hope to make it easier to search for French datasets for NLP and encourage the evaluation of French embeddings.</p><p>So if you are building an NLP model or application targeting the French language, this article is for you! 😉</p><h3>Datasets</h3><p>In MTEB, the evaluation is divided into the 8 distinct tasks mentioned above. We will present the datasets according to these 8 tasks, but keep in mind that a given dataset could potentially be used in other ways.</p><figure><img alt="Datasets for French MTEB" src="https://cdn-images-1.medium.com/max/953/1*7nK1GpHvNDn88YwwuOp7JA.png" /><figcaption>Overview of datasets for embedding evaluation on French. Monolingual datasets are in blue and multilingual ones in purple.</figcaption></figure><h4>Bitext Mining</h4><p>Given two sets of sentences, this task aims to find the best match for each sentence from the first set in the second set. Generally, the second set contains translations of the sentences from the first set. For MTEB evaluation, models are used to embed each sentence and then the closest pairs are found using cosine similarity. <br>The main metric computed for this evaluation is the F1 score.</p><p><strong>Diabla:</strong> We use the DiaBLa dataset, already available on <a href="https://huggingface.co/">HuggingFace</a>. This dataset contains a set of informal written dialogues for evaluating English-French machine translation on informal texts [2]. The dataset contains over 5700 text pairs extracted from 144 dialogues.<br>Link to dataset: <a href="https://huggingface.co/datasets/rbawden/DiaBLa](https://huggingface.co/datasets/rbawden/DiaBLa)">https://huggingface.co/datasets/rbawden/DiaBLa</a></p><p><strong>Flores:</strong> Flores is a benchmark dataset for machine translation between English and low-resource languages [3,4,5]. We use the French subset of this dataset, in other words, the English-to-French translated texts. It contains approximately 997 samples.<br>Link to dataset: <a href="https://huggingface.co/datasets/facebook/flores](https://huggingface.co/datasets/facebook/flores)">https://huggingface.co/datasets/facebook/flores</a></p><h4>Classification</h4><p>For the classification task, we evaluate which embedding models are best suited for the task of identifying to which class a sentence belongs based on its vector representation. To that end, a model is used to embed a train and a test set. Then, a logistic regression classifier is trained on the train set and evaluated on the test set with the Accuracy metric.</p><p><strong>Amazon Review:</strong> We use the French subset of Amazon Review. The dataset contains 200k samples in the train set, and 5K samples both in the validation and test sets. The text of the product reviews is classified according to the associated rating between 0 and 4.<br>Link to dataset: <a href="https://huggingface.co/datasets/mteb/amazon_reviews_multi/viewer/fr](https://huggingface.co/datasets/mteb/amazon_reviews_multi/viewer/fr)">https://huggingface.co/datasets/mteb/amazon_reviews_multi/viewer/fr</a><br> <br><strong>MasakhaNEWS:</strong> MasakhaNEWS is a public dataset for news topic classification, containing news over 16 languages spoken in Africa [6]. We use the French subset of this dataset, it contains over 1480 samples in the train set, 211 in the validation set, and 422 samples in the test set. The dataset provides each text with its label (among 5 distinct classes: sports, business, etc.). Samples are equally distributed over 4 classes (23% of the samples each), but the last class is under-represented (5% of the samples).<br>Link to dataset: <a href="https://huggingface.co/datasets/masakhane/masakhanews/viewer/fra](https://huggingface.co/datasets/masakhane/masakhanews/viewer/fra)">https://huggingface.co/datasets/masakhane/masakhanews/viewer/fra</a><br> <br><strong>Massive Intent:</strong> We use the French subset of the Amazon Massive Intent dataset [7,8]. The dataset is about detecting a user’s intent from a given sentence. The data was collected from virtual assistants usage such as Alexa. It contains over 11,500 samples in the train set, 2030 samples in the validation set, and 2970 samples in the test set.<br>Link to dataset: <a href="https://huggingface.co/datasets/mteb/amazon_massive_intent/viewer/fr]()">https://huggingface.co/datasets/mteb/amazon_massive_intent/viewer/fr</a></p><p><strong>MTOP:</strong> As above, we selected a subset of the dataset Multilingual Task-Oriented Semantic Parsing [9]. It contains over 11,800 samples in the train set, 1580 samples in the validation set, and 3190 samples in the test set. The dataset is duplicated, one copy with 10 labels (<em>mtob_domain</em>) and the other with 111 labels (<em>mtob_intent</em>).<br>Links to datasets: <a href="https://huggingface.co/datasets/mteb/mtop_intent](https://huggingface.co/datasets/mteb/mtop_intent),">https://huggingface.co/datasets/mteb/mtop_intent,</a> <a href="https://huggingface.co/datasets/mteb/mtop_domain](https://huggingface.co/datasets/mteb/mtop_domain)">https://huggingface.co/datasets/mteb/mtop_domain</a></p><h4>Pair classification</h4><p>In this task, a pair of sentences is given with an additional label denoting if the pair is a duplicate or a paraphrase. Both sentences are embedded using a model and the distance between them is computed using several distance metrics such as cosine similarity, euclidean distance, etc. <br>The evaluation metric for this task is average precision based on the cosine similarity.</p><p><strong>Opusparcus:</strong> This dataset is a paraphrase corpus for six languages, where the paraphrases are subtitles from movies and TV shows [10]. We evaluate embeddings on the <em>test.full</em> and <em>validation.full</em> splits of the French dataset French subset. Each part contains respectively 1670 samples and 1630 samples.<br>Link to dataset: <a href="https://huggingface.co/datasets/GEM/opusparcus/viewer/fr.100](https://huggingface.co/datasets/GEM/opusparcus/viewer/fr.100)">https://huggingface.co/datasets/GEM/opusparcus/viewer/fr.100</a></p><h4>Retrieval</h4><p>Considering a query, the retrieval task aims to find the most relevant documents (often paragraphs) among a corpus of documents, using cosine-similarity between vectors. The benchmarking of models regarding this task is particularly interesting for their further implication in Retrieval Augmented Generation (RAG) pipelines.<br>Several metrics are used to evaluate this task, the main one being Normalized Discounted Cumulative Gain (<em>NDCG@10</em>).</p><p><strong>AlloProf:</strong> This question-answering dataset is collected from Alloprof, a Quebec-based primary and high-school help website. It contains almost 30k question-answer pairs, spread over a variety of school subjects [11]. More than half of the answers also contain a link to a reference page addressing the question’s topic. The dataset has been cleaned and formatted for the retrieval task: only French questions mentioning a reference page were kept, and all reference pages have been consolidated as a corpus dataset.<br>Links to datasets : <a href="https://huggingface.co/datasets/lyon-nlp/alloprof](https://huggingface.co/datasets/lyon-nlp/alloprof)">https://huggingface.co/datasets/lyon-nlp/alloprof</a> (formatted), <a href="https://huggingface.co/datasets/antoinelb7/alloprof](https://huggingface.co/datasets/antoinelb7/alloprof)">https://huggingface.co/datasets/antoinelb7/alloprof</a> (original)</p><p><strong>Syntec:</strong> This dataset has been built from the Syntec collective bargaining agreement, with the initial purpose of being used for the retrieval task. This rather small dataset is split into 2 subsets: 100 manually created questions mapped to the article which contains the answer, and the 90 articles from the collective bargaining agreement.<br>Link to dataset: <a href="https://huggingface.co/datasets/lyon-nlp/mteb-fr-retrieval-syntec-s2p](https://huggingface.co/datasets/lyon-nlp/mteb-fr-retrieval-syntec-s2p)">https://huggingface.co/datasets/lyon-nlp/mteb-fr-retrieval-syntec-s2p</a></p><p><strong>BSARD:</strong> The Belgian Statutory Article Retrieval Dataset (BSARD) is a French native dataset initially built for information retrieval in the legal domain [12]. It consists of 1,100 legal questions posed by Belgian citizens and labeled by experienced jurists with relevant articles among more than 22,600 statutory articles from Belgian law. It is a particularly difficult dataset for models that are not specialized in the legal domain.<br>Link to dataset: <a href="https://huggingface.co/datasets/maastrichtlawtech/bsard](https://huggingface.co/datasets/maastrichtlawtech/bsard)">https://huggingface.co/datasets/maastrichtlawtech/bsard</a></p><h4>Reranking</h4><p>The goal of the reranking task is to sort a small set of documents in terms of relevance regarding a given query. The reranking task is often used in recommender systems, or as a complement to the retrieval task. In the context of MTEB, the aim is to evaluate the models’ ability to produce embeddings that yield a cosine similarity correlated with the document’s relevance to the question.</p><p>To evaluate this task, each dataset of the original MTEB benchmark is composed of a query, paired with a few positive (i.e. relevant) documents along with negative (i.e. irrelevant) documents. Despite our efforts, we didn’t find any relevant French dataset structured like so. Consequently, we decided to build our own using the AlloProf and the Syntec retrieval datasets. These already have queries and positive documents so we applied the following process to generate the negatives. The corpus of documents and queries has been embedded using an embedding model. Then, we computed the cosine similarity between every document and query. Documents that are not in the top 10 similarity for a query were labeled as its negative documents.</p><p><strong>AlloProf:</strong> We adapted the Alloprof dataset with the technique presented above to fit the reranking task. For more information about this dataset, please refer to its description in the <em>Retrieval</em> section.<br>Link to dataset: <a href="https://huggingface.co/datasets/lyon-nlp/mteb-fr-reranking-alloprof-s2p](https://huggingface.co/datasets/lyon-nlp/mteb-fr-reranking-alloprof-s2p)">https://huggingface.co/datasets/lyon-nlp/mteb-fr-reranking-alloprof-s2p</a></p><p><strong>Syntec:</strong> As described above, this dataset’s structure has been modified to fit the reranking task. For more information about this dataset, please refer to its description in the <em>Retrieval</em> section.<br>Link to dataset : <a href="https://huggingface.co/datasets/lyon-nlp/mteb-fr-reranking-syntec-s2p](https://huggingface.co/datasets/lyon-nlp/mteb-fr-reranking-syntec-s2p)">https://huggingface.co/datasets/lyon-nlp/mteb-fr-reranking-syntec-s2p</a></p><h4>Clustering</h4><p>This task tries to group sentences or paragraphs into meaningful clusters. To do that, the texts are embedded and a k-means model is applied with the known number of clusters.<br>The metric used to score the model is the v-measure, which does not depend on the cluster label.</p><p><strong>AlloProf:</strong> For this task, documents can be clustered into different topics based on their textual description (fields <em>text</em> and <em>title</em>). For more information about this dataset, please refer to its description in the <em>Retrieval</em> section.<br>Link to dataset: <a href="https://huggingface.co/datasets/lyon-nlp/alloprof](https://huggingface.co/datasets/lyon-nlp/alloprof)">https://huggingface.co/datasets/lyon-nlp/alloprof</a></p><p><strong>HAL: </strong>This dataset was built by scrapping <a href="https://hal.science/](https://hal.science/)">https://hal.science/</a> where scientific publications in various fields are published. We only kept publications in French and extracted their <em>id</em>, <em>title</em> and <em>domain</em>. In total, 85,000 publications can be clustered according to their subject area (field <em>domain</em>) by their <em>title</em>.<br>Link to dataset: <a href="https://huggingface.co/datasets/lyon-nlp/clustering-hal-s2s](https://huggingface.co/datasets/lyon-nlp/clustering-hal-s2s)">https://huggingface.co/datasets/lyon-nlp/clustering-hal-s2s</a></p><p><strong>MasakhaNEWS:</strong> We reuse this multilingual news topic classification dataset for the clustering. We filtered the dataset to only keep the French subset of the test set. In total, 1500 news can be clustered according to their topic (field <em>label</em>) by their textual description (fields <em>text</em> and <em>headline</em>). For more information about this dataset, please refer to its description in the <em>Classification</em> section.<br>Link to dataset: <a href="https://huggingface.co/datasets/masakhane/masakhanews/viewer/fra](https://huggingface.co/datasets/masakhane/masakhanews/viewer/fra)">https://huggingface.co/datasets/masakhane/masakhanews/viewer/fra</a></p><p><strong>MLSUM:</strong> We use the Multilingual Summarization Corpus (MLSUM) [12] of online newspapers for the clustering task. We filtered the dataset to only keep the French subset of the test set. In total, 15,800 online newspapers can be clustered according to their topic by their textual description (fields <em>text</em> and <em>title</em>).<br>Link to dataset: <a href="https://huggingface.co/datasets/mlsum/viewer/fr/test](https://huggingface.co/datasets/mlsum/viewer/fr/test)">https://huggingface.co/datasets/mlsum/viewer/fr/test</a></p><h4>Summarization</h4><p>This task aims to score a machine-generated summary based on its similarity with human-written summaries. To do that, all summaries are embedded and distances between the machine-generated and human-written summaries are computed. The highest cosine similarity score is kept as the machine-generated summary score. Pearson and Spearman correlations with ground truth human assessments are used to evaluate the score computed. <br>This task is close to the STS one.</p><p><strong>SummEval:</strong> This dataset consists of 100 news articles from the CNN/DailyMail dataset [13]. Each of these news articles comes with 10 human-written summaries and 16 machine-generated summaries annotated by 8 persons across coherence, consistency, fluency, and relevance. As this dataset is only available in English, we translated it into French using <a href="https://www.deepl.com/">DeepL</a>. Human-written and machine-generated summaries are embedded and compared with cosine similarity and the average relevance of expert annotations is used as ground truth assessment.<br>Link to dataset: <a href="https://huggingface.co/datasets/lyon-nlp/summarization-summeval-fr-p2p](https://huggingface.co/datasets/lyon-nlp/summarization-summeval-fr-p2p)">https://huggingface.co/datasets/lyon-nlp/summarization-summeval-fr-p2p</a></p><h4>Semantic Textual Similarity (STS)</h4><p>This task aims to compute the similarity between two sentences and give a continuous score. Here, pairs of sentences are labeled with a score between 1 and 5 (lower is low similarity and higher is high similarity). The score generated from embeddings is compared with the labeled score using Pearson and Spearman correlations.</p><p><strong>STS Benchmark Multilingual:</strong> This dataset consists of a mix between several English datasets used in the STS tasks organized in the context of SemEval between 2012 and 2017. It includes text from image captions, news headlines and user forums as pairs of sentences and a score of their similarity. The English version was translated in French with <a href="https://www.deepl.com/">DeepL</a>. We use the 1379 samples of the test set.<br>Link to the dataset: <a href="https://huggingface.co/datasets/stsb_multi_mt](https://huggingface.co/datasets/stsb_multi_mt)">https://huggingface.co/datasets/stsb_multi_mt</a></p><p><strong>STS22 Crosslingual:</strong> This dataset consists of pairs of news articles, with their similarity labeled with a score between 0 and 5. It comprises 10 languages. For french evaluation of models, we only use the french subset which is composed of 104 article pairs.<br>Link to the dataset: <a href="https://huggingface.co/datasets/mteb/sts22-crosslingual-sts/viewer/fr](https://huggingface.co/datasets/mteb/sts22-crosslingual-sts/viewer/fr)">https://huggingface.co/datasets/mteb/sts22-crosslingual-sts/viewer/fr</a></p><p><strong>SICK-FR:</strong> The Sentences Involving Compositional Knowldedge (SICK) dataset consists of about 10,000 English sentence pairs that include many examples of the lexical, syntactic and semantic phenomena. Each pair is labeled with both a “relatedness in meaning” score (with a 5-point rating scale) and an entailment relation (with three possible gold labels: entailment, contradiction, and neutral). For the purpose of this benchmark, we use SICK-FR : a french translation of SICK, along with the “relatedness in meaning” score.<br>Link to the dataset: <a href="https://huggingface.co/datasets/Lajavaness/SICK-fr](https://huggingface.co/datasets/Lajavaness/SICK-fr)">https://huggingface.co/datasets/Lajavaness/SICK-fr</a></p><h3>Conclusion</h3><p>We started this initiative after realizing that it was often difficult to select the right NLP method for French applications. Of course, there are many good multilingual models out there. But when looking closely at their training process, it appears that most of the training data is actually in English. And since the benchmarks evaluating these models are also in English, their performance in French is hard to assess.</p><p>One of the reasons for this might be the lack of good-quality French datasets. Indeed many datasets in French are either too specialized in a specific domain to be used in a benchmark or do not come in a “ready-to-use” format and need significant work in terms of cleaning and formatting.</p><p>Identifying and preparing relevant French datasets to be used for the MTEB-French was not a trivial task, and we hope that this work can help the community accelerate the evaluation of models. The next step for MTEB-French implementation is to identify relevant models to evaluate.</p><p>The models chosen, along with justification for their selection, will be the topic of the next article. Stay tuned! 😎</p><h3>Bibliography</h3><p>[1] Muennighoff, Niklas et al. “MTEB: Massive Text Embedding Benchmark.” Conference of the European Chapter of the Association for Computational Linguistics (2022).</p><p>[2] Bawden, Rachel et al. “DiaBLa: a corpus of bilingual spontaneous written dialogues for machine translation.” Language Resources and Evaluation 55 (2019): 635–660.</p><p>[3] team, Nllb et al. “No Language Left Behind: Scaling Human-Centered Machine Translation.” ArXiv abs/2207.04672 (2022): n. pag.</p><p>[4] Goyal, Naman et al. “The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation.” Transactions of the Association for Computational Linguistics 10 (2021): 522–538.</p><p>[5] Guzmán, Francisco et al. “Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English.” ArXiv abs/1902.01382 (2019): n. pag.</p><p>[6] Adelani, David Ifeoluwa et al. “MasakhaNEWS: News Topic Classification for African languages.” ArXiv abs/2304.09972 (2023): n. pag.</p><p>[7] FitzGerald, Jack G. M. et al. “MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages.” Annual Meeting of the Association for Computational Linguistics (2023).</p><p>[8] Bastianelli, Emanuele et al. “SLURP: A Spoken Language Understanding Resource Package.” Conference on Empirical Methods in Natural Language Processing (EMNLP 2020)</p><p>[9] Xia, Menglin and Emilio Monti. “Multilingual Neural Semantic Parsing for Low-Resourced Languages.” Conference on Lexical and Computational Semantics (2021).</p><p>[10] Creutz, Mathias. “Open Subtitles Paraphrase Corpus for Six Languages.” Conference on Language Resources and Evaluation (LREC 2018).</p><p>[11] Lefebvre-Brossard, Antoine et al. “Alloprof: a new French question-answer education dataset and its use in an information retrieval case study.” ArXiv abs/2302.07738 (2023): n. pag.</p><p>[12] Louis, Antoine et al. “A Statutory Article Retrieval Dataset in French.” Annual Meeting of the Association for Computational Linguistics (2021).</p><p>[13] Scialom, Thomas et al. “MLSUM: The Multilingual Summarization Corpus.” Conference on Empirical Methods in Natural Language Processing (2020).</p><p>[14] Fabbri, A. R. et al. “SummEval: Re-evaluating Summarization Evaluation.” Transactions of the Association for Computational Linguistics 9 (2020): 391–409.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff9dcb700d9e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Labeling tools for NLP]]></title>
            <link>https://medium.com/@lyon-nlp/labeling-tools-for-nlp-36a8179f15d8?source=rss-2fff37de73cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/36a8179f15d8</guid>
            <category><![CDATA[annotation-tools]]></category>
            <category><![CDATA[data-labeling]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Lyon NLP]]></dc:creator>
            <pubDate>Thu, 27 Oct 2022 13:51:09 GMT</pubDate>
            <atom:updated>2022-10-27T13:51:09.585Z</atom:updated>
            <content:encoded><![CDATA[<p>Obtaining data labeled for supervised tasks is not easy. In this article, we present some barriers and tools for the annotation process.</p><h3>Annotation</h3><p>Current artificial intelligence systems are based on the data used to build them. This is particularly the case for machine learning algorithms whose objective is to learn a behavior from data. Data is therefore a central element. To design efficient algorithms, the data must be qualitative (representative of the domain, correctly formatted and containing all the necessary information, among other things) and quantitative (numerous enough to cover the entire scope with several examples for the different possible configurations). These rules obviously apply to the domain of language processing.</p><p>Annotation in language processing has different specificities. First of all, it can concern text or speech data. In the case of speech data, it is possible to cut out portions, identify speakers, transcribe in text form, assign various labels, etc. For textual data, it is also possible to assign labels (class, polarity, subject…) to sentences, to split them, translate them or to assign labels to words to indicate their meaning or function, etc.</p><p>Despite being a tedious process, there are still only few automatic annotation methods. Among them we can mention the technique of the word masking, where a word of a sentence is randomly deleted. There is also the prediction of successive sentences with the extraction of sentences which follow each other or not in a text. Besides these, annotation remains a manual process and thus obtaining labeled data is still a major concern in the scientific field.</p><h3>Labeling Tools</h3><h4>Piaf plateform</h4><p>The PIAF¹ project is led by Etalab, the department of the Interministerial Digital Directorate (DINUM), that “<em>coordinates the design and implementation of the State’s strategy in the field of data</em>”². The primary objective of this project is to build a public and open question and answer dataset in French.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Arh8CKYDuC0s0Hg4cOUO2w.png" /></figure><p>At the begining of the project, a simplified annotation tool was created. This tool allowed the community to contribute to the annotation of data in order to collect thousands of question/answer pairs. Once the annotation campaign was completed, the dataset was published and it is now available to the scientific community with the annotation tool in open access.</p><p>The annotation platform is easy to install, just follow the <a href="https://github.com/etalab/piaf">tutorial</a>. Once the installation is successful, the administration of users and data to annotate is simple. The tool is collaborative, so several people can annotate the same data and use the tool simultaneously.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*FvWYRq8k0ukkRXds4uTKWQ.png" /><figcaption>Figure 1 : PIAF Q&amp;A annotation window.</figcaption></figure><p>The use is intuitive: the person who wants to annotate documents must start by choosing a category of text to label. For each text, the user has to write a question and highlight the part of the text that answers it. It is possible to export the annotations at any time under the standard format of the Stanford SQuAD³ dataset.</p><p>This tool is particularly useful for applications to question/answer systems, but is unfortunately limited to this type of annotation. It would be very interesting to adapt the platform to other use cases such as sentiment analysis or named entity recognition.</p><h4>Label Studio</h4><p>Label Studio⁴ is a free and open-source labeling tool. It is easy to install and particularly scalable, which makes it suitable for annotating a wide variety of tasks (natural language processing, computer vision, time series, etc.).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gqqJ6MOI4w6HD6mvXlSI6A.png" /></figure><p>The labeling interface is fully configurable. As this configuration is not necessarily easy, the platform has many templates that cover the most common labeling tasks (sentiment analysis, named entities, question answering, object detection) and make the configuration possible in a few clicks. Moreover, the labeled data can be exported in many formats, including those usually ingested by learning algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*O2yx3pv-8hGDXpmyWmtlCg.png" /><figcaption>Figure 2 : Label-studio template selection when creating a project.</figcaption></figure><p>Label Studio is also suitable for users of cloud services since the tool can be deployed on a server to allow different collaborators to annotate the data simultaneously, and to make the labels accessible at any time. In this case, Label Studio can connect to various online storage services (AWS S3, Azure Blob Storage, etc.) to retrieve data and store labels.</p><h4>Doccano</h4><p>Doccano⁵ is an open-source annotation tool, free of charge. Initially created for text annotation tasks (sequence-to-sequence, text classification, named entities), it has been recently extended for image classification, object detection and speech-to-text annotations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nuRdvk0l-oNDSBCG87VYTA.png" /></figure><p>Doccano can be installed via pip and launched locally. It will use a SQLite3 database by default, but if preferred the database can be changed to PostgreSQL. The purpose of the database is to save annotations and datasets locally and not upload them to a server, which is preferable when working with confidential datasets. The tool is also available as a Docker image. Thus, it can be deployed to different cloud providers (e.g., AWS, Heroku, etc.). It can also be connected to cloud storage sources (e.g., Amazon S3, Google Cloud Storage, etc.).</p><p>Doccano offers a simple user interface which makes the user experience smoother. Different templates of labeling are available. An annotation project can be started by selecting a template and completing the required information. The tool is also made for collaborative working, so it is possible to create users and affect roles for each project. You can also write annotation guidelines for annotators to read when doing the task. Metrics are also available, for example for the text classification task it shows the distribution of labels across the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*vmUjZmLn42361E5tiPp2eg.png" /><figcaption>Figure 3 : Doccano project creation page.</figcaption></figure><p>Most importantly, datasets can be imported from files of different formats according to the project type (image or text classification, etc.). For example for text classification, a dataset can be loaded in CSV, JSON or fastText format. Same file formats are used to export texts and save their annotations.</p><p>We presented 3 annotations tools in open access among all those available on the market. We hope that this information will guide you in your annotation tasks.</p><p>As conclusion, making annotated data accessible is also a way of sharing knowledge and limiting the complexity of annotation.</p><p>¹ <a href="https://piaf.etalab.studio/">https://piaf.etalab.studio/</a><br>² <a href="https://www.etalab.gouv.fr/qui-sommes-nous/">https://www.etalab.gouv.fr/qui-sommes-nous/</a><br>³ <a href="https://rajpurkar.github.io/SQuAD-explorer/">https://rajpurkar.github.io/SQuAD-explorer/</a><br>⁴ <a href="https://labelstud.io/">https://labelstud.io/</a><br>⁵ <a href="https://github.com/doccano/doccano">https://github.com/doccano/doccano</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36a8179f15d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>