<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Magnus Sahlgren on Medium]]></title>
        <description><![CDATA[Stories by Magnus Sahlgren on Medium]]></description>
        <link>https://medium.com/@magnus.sahlgren?source=rss-55147f0c79d4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SlBmg6z6fDAl75HerSgLWA.jpeg</url>
            <title>Stories by Magnus Sahlgren on Medium</title>
            <link>https://medium.com/@magnus.sahlgren?source=rss-55147f0c79d4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:20:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@magnus.sahlgren/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Open release of GPT-SW3]]></title>
            <link>https://medium.com/@magnus.sahlgren/open-release-of-gpt-sw3-814331cb931c?source=rss-55147f0c79d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/814331cb931c</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Magnus Sahlgren]]></dc:creator>
            <pubDate>Fri, 24 Nov 2023 07:39:27 GMT</pubDate>
            <atom:updated>2023-11-24T07:39:27.514Z</atom:updated>
            <content:encoded><![CDATA[<p>We are now taking the next steps in the democratization and value creation of Large Language Models (LLMs) in Sweden and the Nordics, with the open release of the <a href="https://medium.com/ai-sweden/what-is-gpt-sw3-5ca45e65c10">GPT-SW3 </a>family of LLMs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*na3I_1ZcJ-ySPuN0" /><figcaption>Image produced by DALL·E 3.</figcaption></figure><p><a href="https://medium.com/ai-sweden/what-is-gpt-sw3-5ca45e65c10">As previously described</a>, GPT-SW3 is the first LLM for the Nordic languages. The models have been developed by the <a href="https://www.ai.se/en/projects-9/natural-language-understanding">NLU group</a> at AI Sweden with support from <a href="https://ri.se/">RISE</a>, the <a href="https://wasp-sweden.org/industrial-cooperation/research-arenas/wara-media-and-language/">WASP WARA for media and language</a>, the <a href="https://www.nsc.liu.se/">National Supercomputer Center at Linköping university</a>, and <a href="https://www.nvidia.com/">Nvidia</a>, with funding from the <a href="https://www.vinnova.se/">Swedish Innovation Agency (Vinnova)</a>. <a href="https://medium.com/ai-sweden/why-do-we-need-a-large-gpt-for-swedish-9a0409877f46">GPT-SW3 was developed</a> as a research project, with the primary goal to build and provide the first Nordic LLM as a foundational resource for Nordic NLP, but also to build competence and awareness around LLMs in Sweden, and to explore and validate potential applications of a nationally developed LLM. The open release of the GPT-SW3 models marks the end of the development phase of GPT-SW3. Our initial goal was to build and provide the first LLM for the Nordic languages, and with the open release of the models we have now fulfilled this goal.</p><h3>From restricted to open access</h3><p>In <a href="https://medium.com/ai-sweden/gpt-sw3-pre-release-evaluating-large-generative-language-models-for-swedish-8799abfac670">a previous post</a>, we described our reasons for opting for an initial restricted pre-release of the GPT-SW3 models. During the pre-release phase, we have manually granted access to the models to researchers in the Nordic region, for the primary purpose of research and validation. To recapitulate, the main reason for this cautious approach was to be able to identify any potential issues with the models’ behaviors and to minimize the risk of adversarial usage of the models. We have collected feedback from users during the pre-release phase, and based on this feedback we now consider an open release to be the natural next step.</p><p><strong>The open release will now allow users complete access and free usage of the GPT-SW3 models. This means that it will now be possible to download, modify, finetune, integrate and to build commercial and non-commercial products based on the models. This is an important step in the democratization of access to powerful AI models, and to accelerate the use of AI in Sweden and in the Nordic region.</strong></p><h3>Instruction tuning</h3><p>We are releasing our base GPT-SW3 models as well as a set of instruction-finetuned versions of the models. The -instruct versions of our models have been finetuned on a set of instruction datasets, including <a href="https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm">Dolly</a>, <a href="https://open-assistant.io/">OpenAssistant</a>, a selection of data from <a href="https://github.com/LAION-AI/Open-Instruction-Generalist">Open Instruction Generalist</a> (OIG), and a dataset compiled specifically for our models based on <a href="https://www.fass.se/">FASS</a> (The Swedish pharmaceutical formulary). The instruction data was formatted into a unified turn-based format, where an initial user query is followed by an assistant response, which in turn is (potentially) followed by a follow up user query, and so on. To accommodate the multilingual capacities of our models, we translated the OpenAssistant and Dolly datasets. The OpenAssistant data was translated from English to Swedish, Danish, Norwegian, and Icelandic, while the Dolly data set was translated from English to Swedish and Danish. The translations were done with our own GPT-SW3 base models.</p><h3>The next steps</h3><p>Our focus going forward is to take the next steps in the development of foundation models in and for Sweden, the Nordics and Europe. What this means in more detail is something we will come back to in a future post. Stay tuned!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dQWlfqyphjMGWk3T" /><figcaption>Image produced by DALL·E 3.</figcaption></figure><ul><li>Access GPT-SW3 <a href="https://huggingface.co/AI-Sweden-Models">here</a>.</li><li>Read more about the team behind GPT-SW3 and AI Sweden’s work with NLU <a href="https://www.ai.se/en/projects-9/natural-language-understanding">here</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=814331cb931c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPT-SW3 as Klara, the artificial friend]]></title>
            <link>https://medium.com/ai-sweden/gpt-sw3-as-klara-the-artificial-friend-e0cdb50aab75?source=rss-55147f0c79d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0cdb50aab75</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[gpt-3]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Magnus Sahlgren]]></dc:creator>
            <pubDate>Wed, 01 Mar 2023 14:52:54 GMT</pubDate>
            <atom:updated>2023-03-24T15:33:58.932Z</atom:updated>
            <content:encoded><![CDATA[<h4>In this post, we look back at the first public display of GPT-SW3. This occurred during an art exhibition in Stockholm, Sweden, where GPT-SW3 made a guest appearance in the form of a chatbot named Klara.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w-M1ccCWroVFqyie" /><figcaption>© Nobel Outreach Photo: Richard von Hofsten</figcaption></figure><h4><strong>The Klara chatbot and Life Eternal</strong></h4><p>Between 1 October, 2022 and 29 January, 2023, the <a href="https://nobelprizemuseum.se/">Nobel Prize Museum</a> organized an exhibition called <a href="https://evigtliv.nobelprizemuseum.se/en/">Life Eternal</a> at Liljevalchs art gallery in Stockholm. The exhibition brought together the fields of science, art and cultural history, showed different approaches to eternity, explored the crucial issues of our era and offered hope for the future. The exhibition featured 13 rooms, each with different themes, artworks, and installations with connections to the Nobel Prize categories.</p><p>One of the commissioned installations was centered around the literature laureate Kazuo Ishiguro’s novel “Klara and the Sun”, which tells the story of an android (Klara) whose purpose is to be an artificial friend to her human companion (Josie). This room, designed by Sahara Widoff and David Berner, consisted of an artistic interpretation of one of the scenes in the novel, brought to life by synchronized light, sound, animations and a masterful sculpture of Josie by <a href="https://www.odnilsson.com/">Oscar Nilsson</a>. In connection to the theme of this room, a set of computers in the main hall offered the visitors to talk to a chatbot inspired by Ishiguro’s character Klara.</p><h4><strong>Klara, GPT-SW3 and ChatGPT</strong></h4><p>The Klara chatbot was no ordinary chatbot, but a set of five different carefully engineered prompt interfaces to <a href="https://medium.com/ai-sweden/what-is-gpt-sw3-5ca45e65c10">the first large generative language model for Swedish, GPT-SW3</a>. GPT-SW3 is a large language model that has been trained on <a href="https://medium.com/ai-sweden/the-nordic-pile-a8d5aaf3db60">1.2 TB of text data in the North Germanic languages (i.e. Swedish, Norwegian, Danish, Icelandic, and also English)</a>. This meant that the Klara chatbot had the capacity to talk about anything and everything (in 5 different languages), even if she at first asked the visitors to talk about one of five themes, including the climate, the sun, space, eternal life, and what happens after death. If cleverly instructed, however, Klara could be made to discuss any topic the visitor wanted.</p><p>When the exhibition started in October 2022, large generative language models were a relatively unknown phenomenon to the general public, and only a limited number of tech-savvy visitors probably had a grasp of the underlying technology and its potential. This all changed on 30 November 2022, when <a href="https://openai.com/blog/chatgpt/">ChatGPT</a> was unleashed on the world, instantly propelling the capacity of large generative language models to the forefront of public awareness.</p><p>ChatGPT is the result of several steps of iterative improvement of the underlying GPT-3 language model developed by <a href="https://openai.com/">OpenAI</a>. By comparison, GPT-SW3 is a basic generative language model of the same type as GPT-3, but without the additional improvements of ChatGPT. As such, the Klara chatbot had limited understanding of user intents compared to ChatGPT, and was prone to the same type of repetitive behavior and creative attitude towards factual information as other large generative language models. Even so, the Klara chatbot had a general language capacity that has never before been exhibited by a Swedish chatbot, and the Life Eternal exhibition was <strong>the first time that the general Swedish population had the chance to interact with a large generative Swedish language model</strong>; a historic event for Swedish language technology.</p><h4><strong>How Klara was built</strong></h4><p>The Klara chatbot was the result of a collaboration between the NLU research group at AI Sweden and the Danish design studio <a href="https://www.yoke.dk/">YOKE</a>. YOKE designed and developed the physical installation at Liljevalchs, as well as the graphical user interface that featured wonderful futuristic animations in ASCII-art. The NLU group at AI Sweden hosted its 6.7B parameter GPT-SW3 model on an Nvidia A100 GPU provided by <a href="https://www.cgit.se/">CGit</a>, and provided access to the model via an API that was built exclusively for the exhibition.</p><p><strong>The most important work, however, was done by </strong><a href="https://aliceheiman.xyz/"><strong>Alice Heiman</strong></a>, who has been part of the NLU group since its start in the autumn of 2021, and who designed the <em>prompt templates</em> that were used to make GPT-SW3 behave in a manner similar to Ishiguro’s character Klara. The prompt templates consist of three different parts:</p><ul><li>A <em>header</em> that specifies the general theme of the discussion in terms of a number of keywords (e.g. “sun, clouds, sun beams, sundown”).</li><li>An <em>intro</em> that provides the instruction to the model. This intro specifies the roles of the participants of the discussion (the visitor is “the philosopher Josie” and the chatbot is “the robot Klara”), as well as the desired behavior of the chatbot (“the discussion is respectful and considerate, and gives wise answers…”).</li><li>A set of <em>themed questions and answers</em> that provide examples of how a conversation between Josie and Klara may look like.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VN3WPyUO4vymsaOd" /><figcaption>Klara prompt design and image by Alice Heiman.</figcaption></figure><p>The prompt template was not visible to the visitors, but was prepended to every user input each time GPT-SW3 was called through the API. This means that even if the visitors only wrote very short inputs to Klara (such as “hello”), the model’s input always consisted of the entire prompt template plus the user statement. In order for Klara to remember what was said previously in the discussion, we also fed the last 6 dialogue turns (if available) as context to the model (in the “user question” part of the prompt). This context acted as a sort of memory that enabled Klara to refer back to things that were said previously, which significantly contributed to a more coherent conversation. However, since Klara had a continuous stream of visitors, we reset the conversations and the terminals after 1 minute of idle time, so that the previous conversations would not affect the experience of new visitors.</p><h4><strong>Klara and her visitors</strong></h4><p>The exhibition had over 40,000 visitors. Quite impressively (since Klara was hosted on a lab machine), Klara had zero downtime during the exhibition. In total, <strong>Klara had 7,437 conversations with visitors</strong>, with a mean length of 10 dialogue turns per conversation. The longest conversation was 525 turns (half of which is Klara’s statement and half is from visitors), and the second longest conversation was 369 turns (half from Klara, half from visitors).</p><p>The length of the conversations are shown in the image below (number of turns is on the y-axis, and time is on the x-axis), and the image even further below shows the distribution of conversation lengths (number of turns is on the y-axis, and number of conversations is on the x-axis). As expected, there were only a few long conversations and a lot of short ones. It should be noted that during busy times at the exhibition, the terminals probably did not reset between visitors, leading to several conversations being logged as the same event with a large number of dialogue turns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*CsWO3ayZVIVPO621" /><figcaption>Length of conversations over time.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*7Hs0HjzB4FX8z5KD" /><figcaption>Distribution of conversation lengths.</figcaption></figure><h4><strong>Retrospective</strong></h4><p>The Klara chatbot was the first public display of GPT-SW3, and the first time the general public in Sweden had a chance to interact with a large generative Swedish language model. Even if Klara was only based on a relatively small and basic generative language model (i.e. our 6.7B parameter model that was only pretrained using standard causal language modeling with no instruction tuning), Klara worked for the most part remarkably well, and visitors were in general very positive to the experience of talking to Klara. This is, of course, primarily thanks to the clever prompt design by Alice Heiman, but it also shows the promising capacity and generality of GPT-SW3.</p><p>It is important to stress the difference between general GPT models such as GPT-SW3, and instruction-tuned versions such as ChatGPT, which are specifically finetuned to follow instructions and to understand user intents. The size of the investment and resources that has gone into developing ChatGPT cannot compare to the very limited funding and resources that has been available for the development of GPT-SW3. Even so, we (the NLU group at AI Sweden) are currently working on ways to augment and refine our basic GPT-SW3 models, and we believe that future chatbots based on GPT-SW3 will likely use some form of <a href="https://arxiv.org/abs/2109.01652">instruction tuning</a>, and possibly also other refinements such as various forms of <a href="https://arxiv.org/abs/2112.04426">retrieval augmentation</a> or the capacity to <a href="https://arxiv.org/abs/2302.04761">use tools</a>.</p><p>On our part, we are thoroughly impressed by the remarkable foresight and courage of the Nobel Prize Museum to include in the exhibition an unconstrained general-capacity chatbot built on a bleeding edge large generative language model. The exhibition and collaboration with the Nobel Prize Museum and YOKE was a great experience for us, and we think that the intersection between art and science is a great place to showcase the capacities of generative AI. We are proud to be able to preserve one instance of Klara at the AI Sweden office in Stockholm, so if you happen to visit us sometime in the future, you might be able to meet Klara in person.</p><p>GPT-SW3 is developed by the <a href="https://www.ai.se/en/projects-9/natural-language-understanding">NLU group at AI Sweden</a>, with contributions from <a href="http://ri.se/">RISE</a>, the <a href="https://wasp-sweden.org/research/research-arenas/wara-media/">WASP WARA for media and language</a>, <a href="https://www.nvidia.com/sv-se/">Nvidia</a>, and the <a href="https://www.nsc.liu.se/">National Supercomputer Center at Linköping university</a>. If you are interested in trying GPT-SW3, you can <a href="https://forms.gle/G5qNkHYAV1vmRecdA">sign up for access</a>, and once approved you can find our models on <a href="https://huggingface.co/AI-Sweden">Hugging Face</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0cdb50aab75" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-sweden/gpt-sw3-as-klara-the-artificial-friend-e0cdb50aab75">GPT-SW3 as Klara, the artificial friend</a> was originally published in <a href="https://medium.com/ai-sweden">AI Sweden</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Nordic Pile]]></title>
            <link>https://medium.com/ai-sweden/the-nordic-pile-a8d5aaf3db60?source=rss-55147f0c79d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8d5aaf3db60</guid>
            <category><![CDATA[gpt-3]]></category>
            <dc:creator><![CDATA[Magnus Sahlgren]]></dc:creator>
            <pubDate>Thu, 22 Sep 2022 13:30:40 GMT</pubDate>
            <atom:updated>2023-01-10T21:21:40.737Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>How we work with training data for GPT-SW3</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gS4ezUbgEUdsd9_871orzQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@lysanderyuen">Lysander Yuen</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p>This post describes the work done by the NLU research group at AI Sweden to collect and process training data for the GPT-SW3 language model. We call the resulting dataset <strong>The Nordic Pile</strong>.</p><p>As we have described in our previous posts [<a href="https://medium.com/ai-sweden/what-is-gpt-sw3-5ca45e65c10">1</a>, <a href="https://medium.com/ai-sweden/why-do-we-need-a-large-gpt-for-swedish-9a0409877f46">2</a>], we develop GPT-SW3 in order to produce a foundational resource for Swedish (and Nordic) NLP, and to investigate the practical usefulness of large generative models for solving real-world NLP tasks. Our hope is that the model will be useful across a variety of different application domains and use cases, ranging from academic research to applications in both the public and private sectors.</p><p>We know from previous research that foundation models need to be large both with respect to <a href="https://arxiv.org/abs/2001.08361">the number of parameters in the model</a>, and also with respect to <a href="https://arxiv.org/abs/2203.15556">the amount of training data that the model has seen</a>. Since our goal is to develop a model that can be as representative as possible of the Swedish-speaking population, we aim for a final model with more than 100 billion parameters, and <strong>we aim to use training data that as closely as possible reflects the dialects, sociolects, demography and interests of the Swedish citizens</strong>.</p><p>One challenge (among many) with such an ambition is that Swedish is a relatively small language with a relatively limited number of speakers. This means that there is limited amounts of text data available in Swedish, in particular when it comes to readily available datasets. For NLP in Swedish, the main source of data is <a href="https://spraakbanken.gu.se/">Språkbanken Text</a> which provides a range of NLP datasets, but most are relatively small, and some are sentence-scrambled which limits their usefulness for training language models. We can also mention the <a href="https://kb.se/">National Library of Sweden </a>that collects and stores all text published in Sweden. Unfortunately, this rich and valuable source of data is not available for external NLP research and development (we did contact the National Library with a request to collaborate on GPT-SW3, but they were not willing to share their data with this project).</p><h3>The Nordic Pile</h3><p>Since there are no readily available large-scale collections of Swedish text data, it has been necessary for us to compile our own dataset, based primarily on existing data sources such as<a href="https://huggingface.co/oscar-corpus"> OSCAR</a>,<a href="https://huggingface.co/datasets/mc4"> MC4</a>, and<a href="https://opus.nlpl.eu/"> OPUS</a>. We have also collected data from repositories such as<a href="https://www.diva-portal.org/"> DiVA</a>,<a href="https://www.fass.se/"> FASS</a>,<a href="https://data.riksdagen.se/data/"> the Swedish government’s open data portal</a>,<a href="https://www.1177.se/"> 1177</a>,<a href="https://sv.wikipedia.org/"> Wikipedia</a>,<a href="https://litteraturbanken.se/"> Litteraturbanken</a>, as well as websites of Swedish authorities, and some of the largest Swedish discussion forums such as<a href="https://www.flashback.org/"> Flashback</a>,<a href="https://www.familjeliv.se/"> Familjeliv</a>, and Swedish discussions on<a href="https://www.reddit.com/"> Reddit</a>.</p><p>In addition to this, we have also relied on the fact that Swedish is part of a relatively small family of languages (the North Germanic language group) that includes Norwegian, Danish, Icelandic, and Faroese. We therefore also include existing datasets in these languages, except for Faroese where we were not able to find any data. Examples of datasets in these languages include the before-mentioned OSCAR, MC4, OPUS, Wikipedia, and Reddit data sources, as well as the<a href="https://huggingface.co/datasets/NbAiLab/NCC"> Norwegian Colossal Corpus</a>, the<a href="https://gigaword.dk/"> Danish Gigaword Corpus</a>, and the<a href="https://clarin.is/en/resources/gigaword/"> Icelandic Gigaword Corpus</a>. We also include a sample of English data from<a href="https://pile.eleuther.ai/"> The Pile</a> as well as programming code collected from C<a href="https://huggingface.co/datasets/codeparrot/github-code">odeParrot</a>. Lastly, we also include a special mathematics dataset in both Swedish and English, since we are interested in investigating the extent to which the model will be able to perform simple math.</p><p>The resulting dataset, which we call <strong>the Nordic Pile</strong>, amounts to approximately 1.3 TB of data in total, and has the following distribution over languages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*iIncbwLKlQIDPoNl" /></figure><p>In addition to being sufficiently large for training a multi-billion parameter language model, the Nordic Pile dataset also contains a rich typological variety that we hypothesize will be useful for the model’s performance in all of the Nordic languages. However, the language distribution of the original data may be somewhat suboptimal if the goal is to build a primarily Swedish, and secondarily Nordic language model. We therefore weight the different languages to better reflect our end goal, and we also weight the various data sources to better reflect our ambition to arrive at a representative model for the Swedish population. This weighting of data sources includes down-weighting webcrawled data, which is known to be noisy and of relatively low quality, and up-weighting high-quality editorial sources such as Wikipedia. We also<a href="https://aclanthology.org/2022.acl-long.577/"> deduplicate</a> our data by using both <strong>exact deduplication</strong> (i.e. removing copies of the same content), and <strong>fuzzy deduplication</strong>, where we use Locality-Sensitive Hashing (LSH) to remove content that is very similar to other content on the dataset. The final version of the Nordic Pile that we use as training data for GPT-SW3 consists of the following distribution of languages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*8AsVydLLtjQrnr31" /></figure><h3>Ethical and legal considerations</h3><p>Our main guiding principles when collecting the Nordic Pile has been to ensure that the data is both representative and of high quality, while at the same time being of sufficient quantity to enable training of large language models. To ensure that the models can generalize over a variety of different domains, and that they are useful for a range of different applications, we have tried to collect texts that are representative of a variety of language styles and usages, knowledge domains, and social groups. The mix of editorial sources, such as newspaper articles or texts published on the homepages of governmental authorities and public sector organizations, and user-generated content such as blogs and forums assures a wide range of both styles and topics.</p><p>Since most of the datasets in the Nordic Pile are too big to allow for careful manual quality inspection, we instead rely on programmatic rules for quality assurance that follow best practices from the research literature. Decisions on which datasets to include and which to exclude were taken collectively by the GPT-SW3 development team after thorough discussions about the advantages, disadvantages, and risks of the various datasets. Advantages and disadvantages relate primarily to the quality of the text sources regarding technical processability. For example, the SOU dataset (<a href="https://www.sou.gov.se/">Statens offentliga utredningar</a>) contains text extracted from PDFs, but because of the pitfalls of PDF text extraction, this dataset was eventually dismissed. Another reason for dismissing datasets is the occurrence of computer generated texts, which was the case e.g. for a dataset containing subtitles from YouTube.</p><p>We have also mapped a number of potential risks with using the Nordic Pile dataset. Most of these relate primarily to text sources that we know contain, or that are very likely to contain, large amounts of personal information. To filter such material, we selected certain social discussion subforums and defined a list of websites that shall not be included in the training data: these include, for example,<a href="http://hitta.se/"> hitta.se</a> and<a href="http://enrio.se/"> eniro.se</a>, i.e. websites that collect information about individuals such as address, telephone number, birthday, persons living in the same household etc. from official registers and telecom operators. We also perform anonymization as a measure to support the rights of data subjects potentially present in the text data collected from online forums. For discussion forums (Flashback and Reddit) we anonymize all user names by exchanging them with randomly sampled names compiled from name lists by SCB (and other national equivalents).</p><p>Another risk that we identified relates to discriminatory language and hate speech present in online discourses and thus in the Nordic Pile. In this case, we decided not to filter our datasets. This decision comes with potential upsides and downsides. On the upside, we believe that not aggressively content filtering the training data will lead to a more versatile model that can be used as a foundational resource for a broad set of NLP tasks — including, for example, the detection and analysis of discrimination and hate speech at scale in the context of research or content moderation. The downside is that the model will be able to reproduce the problematic content and thus reinforce discrimination and hate speech present in today’s online discourses. Our decision shifts the focus to the responsible use of the model, and it increases the need for clear usage guidelines and safety checks when building applications based on the model.</p><p><strong>To tackle these issues, we are currently establishing a Subject Matter Expert Pool.</strong> This expert pool consists of researchers from the humanities and social sciences who have profound knowledge of the social and cultural contexts in which the texts of our training data have been produced. Together with them, we will address questions of representativeness, fairness and discrimination as well as “undesired” model outputs in concrete use case scenarios. We aim to iteratively re-evaluate our decisions on filtering.</p><p><strong>In our work with the Nordic Pile, we have made significant efforts to both comply with GDPR and at the same time ensure that the data contains as representative and diverse content as possible in order to arrive at a model that reflects the entire Swedish society.</strong> It is clear that we will not be entirely successful in this. The training data for GPT-SW3 is probably quite representative of Swedish internet discourse in general, and of the Swedish public sector, but we know that this data does not necessarily reflect the entire Swedish population. On the other hand, it is not clear if it would be at all possible to collect such data that would be completely representative of a population. Given the limited resources available to us in the GPT-SW3 initiative, we have done our best to make sure that our data collection and processing efforts have been as responsible and transparent as possible, while at the same time being open to reassess and restructure our plan and workflow based on new evidence and input. Our overarching goal remains to produce a language model that can be a foundational resource for Swedish and Nordic NLP.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8d5aaf3db60" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-sweden/the-nordic-pile-a8d5aaf3db60">The Nordic Pile</a> was originally published in <a href="https://medium.com/ai-sweden">AI Sweden</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why do we need a large GPT for Swedish?]]></title>
            <link>https://medium.com/ai-sweden/why-do-we-need-a-large-gpt-for-swedish-9a0409877f46?source=rss-55147f0c79d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a0409877f46</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[gpt-3]]></category>
            <dc:creator><![CDATA[Magnus Sahlgren]]></dc:creator>
            <pubDate>Thu, 14 Jul 2022 10:35:55 GMT</pubDate>
            <atom:updated>2023-01-23T09:06:46.471Z</atom:updated>
            <content:encoded><![CDATA[<h4>What are the advantages of building a large language model for Swedish, and what should we look out for?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_vptzEAMv7s3CfYNsGmFSg.png" /><figcaption>Photo by <a href="https://unsplash.com/@itfeelslikefilm">Janki Ferlic</a> on <a href="https://unsplash.com/photos/sfL_QOnmy00">Unsplash</a></figcaption></figure><h4>Generality and usability</h4><p>As we noted in our <a href="https://medium.com/ai-sweden/what-is-gpt-sw3-5ca45e65c10">first post</a>, with enhanced size of the language models (and the training data) comes improved capacity, in particular with respect to the <strong>generality</strong> of the models. This generality manifests in <strong>zero-shot</strong> and<strong> few-shot </strong>capacity, which refers to the model’s ability to solve tasks it has not been specifically trained for, relying solely on clever instruction modeling. As an example, we might ask the model to translate from Swedish to English simply by using the instruction “Translate from Swedish into English:” and then adding the term or phrase we want to translate, e.g. “vörtbröd”, together with an arrow to signal the model’s reply. This leads to the following input to the model:</p><blockquote><strong>Translate from Swedish to English:</strong></blockquote><blockquote>vörtbröd =&gt;</blockquote><p>(Prompt instruction in boldface, followed by the target term on the next line, and an arrow to signal where the model’s reply will go. This particular prompt design is copied from <a href="https://arxiv.org/abs/2005.14165">Brown et al. 2020: Language Models are Few-Shot Learners</a>.)</p><p>This would be an example of a zero-shot setting. If we also add a couple of examples of translations from Swedish to English, we provide additional context for the model, which is then referred to as a few-shot scenario:</p><blockquote><strong>Translate from Swedish to English: strömming</strong></blockquote><blockquote><strong>sill =&gt; herring</strong></blockquote><blockquote><strong>Translate from Swedish to English:</strong></blockquote><blockquote><strong>rädisa =&gt; radish</strong></blockquote><blockquote><strong>Translate from Swedish to English:</strong></blockquote><blockquote>vörtbröd =&gt;</blockquote><p>(Prompt examples and instruction in boldface, followed by the target term, and an arrow to signal the model’s reply.)</p><p>It should be noted that there are also other ways to send instructions to the model, such as <a href="https://arxiv.org/pdf/2103.10385.pdf"><strong>P-tuning</strong></a>, which is a very interesting and promising alternative to standard prompt engineering. P-tuning instead relies on an external prompt model that trains input embeddings that can act as continuous prompts to the GPT. We will write more about the potential of P-tuning, and how it might work using GPT-SW3, in a later post.</p><p>The possibility to solve tasks without specific training is both interesting and potentially extremely valuable, since it would drastically reduce the need for training data, and the need for building specialized solutions for each different application. <strong>Our hypothesis is that if we could provide API access to a Swedish language model with few-shot capacity, it would greatly simplify the use of language models for actors that have limited competence and hardware to build and deploy their own solutions.</strong></p><p>The GPT-SW3 initiative was partly borne from lessons learned during previous initiatives such as “<a href="https://www.ai.se/en/node/81535/language-models-swedish-authorities">language models for Swedish authorities</a>”. During this project, whose goal was to facilitate for actors in the public sector in Sweden to use language models in their NLP solutions, we have seen significant needs and interest from the public sector in using language models, but we have also seen significant challenges with doing so. These challenges often concern <strong>data readiness</strong>,<strong> competence</strong>,<strong> </strong>and <strong>compute</strong>.</p><p><a href="https://nlp-data-readiness.readthedocs.io/en/latest/">Data readiness</a> refers to issues related to data, and can include questions such as “do we have access to the data we want to use?”, “can we legally use the data?”, “is the data in the right format?”, “is the data noisy?”, “is this the right data for solving the problem?”. In fact, we encounter such questions so often that we have started a dedicated project for improving the data readiness in the public sector in Sweden: the <a href="https://www.ai.se/en/node/81535/data-readiness-lab">Data Readiness Lab</a>.</p><p>Access to, and retention of, competence in language modeling, NLP, or AI in general is also a challenging factor, in particular in the public sector. This affects the potential for end users to build their own solutions based on language models, even if the models are publicly available in user-friendly frameworks such as <a href="https://huggingface.co/docs/transformers/index">Huggingface Transformers</a>. We are currently exploring an <a href="https://www.ai.se/en/partner-offering/talent-programs/NLU-Talent-Program">NLU Talent Program</a> for fostering and supplying competence in NLU to organizations that are partners in <a href="https://www.ai.se/en/partner-offering">AI Sweden</a>.</p><p>Access to suitable compute resources is also challenging for many actors, especially in the public sector. Even if their data is in order, they have recruited the required competence, and suitable models are publicly available, they still need sufficient compute resources to run the models (both for training and for inference). Acquiring and integrating such resources can be challenging, both with respect to cost (and availability), but also with respect to IT-infrastructural considerations.</p><p><strong>We believe that the few-shot capacity of very large language models may provide a way to leverage the power of language models without requiring end-users to tackle these challenges by themselves.</strong></p><h4>Linguistic sovereignty and digital democracy</h4><p>Most of the current development of very large language models is driven by private companies in other countries (and even in other continents) than Sweden. These actors may not have an interest in producing such resources for a small language such as Swedish, and even if they do, they may not have the incentives, nor the capacity, to build and provide such models in a way that reflects Swedish circumstances and needs. We think <strong>it is important that a model that aims to be a foundational resource for Swedish NLP represents the entire Swedish language, and by extension, the entire Swedish population.</strong></p><p>This obviously boils down to a question about representative, fair, and transparent data, which is something we take very seriously, and devote considerable efforts to. Since the question about data is so important, we will devote our next series of posts entirely to this topic. For now, we simply note that the GPT-SW3 initiative has the ambition to build a model that as exhaustively as possible represents the Swedish language and the Swedish population. <strong>We build GPT-SW3 in Sweden, for Sweden</strong>.</p><p>This means that we also think <strong>it is important that a foundational Swedish GPT model can be made available to all sectors in Sweden that may have a need for NLP solutions.</strong> Our vision is thus to provide GPT-SW3 as a kind of basic infrastructure for Swedish NLP that everyone can access and build services and solutions on top of.</p><p>Our current plan is to provide access to the model through an API in the same way as Open AI provides access to <a href="https://openai.com/api/">GPT-3</a>. This has proven to be a working solution in the case of GPT-3, and we want to explore if the same approach will work also for GPT-SW3. However, we aim to do things a bit differently from Open AI. Even if we currently think that access to GPT-SW3 will be limited to an API (we will discuss this in more detail in a coming post), <strong>we aim to be completely open, transparent, and collaborative </strong>throughout the entire development process.</p><p>Even if building a model such as GPT-SW3 is extremely resource-demanding both with regards to data and compute, we think the providing access to the resulting model through an API may actually lead to cost savings for the end-users, who no longer have to buy their own GPU servers and to recruit expensive experts. Perhaps somewhat counter-intuitively, very large language models may in this way contribute to environmental sustainability (in particular if we make sure that the inference server runs on green electricity, which we will do).</p><h4>New to this Medium</h4><p>My, and the AI Sweden NLU team’s, aim with being on Medium is to be as open and transparent as possible with the development process, and to invite several different perspectives on this type of model — both our own as well as those who may be more critical of this development. We will cover topics ranging from details of the development process (e.g. data collection and filtering), over reasons and considerations for leading this initiative, to more philosophical questions about the limits of large language models and the potential (or not) to reach AGI.</p><p>We’ll gather everything under the <a href="https://medium.com/ai-sweden/nlu/home">NLU / GPT-SW3 tab</a> on the <a href="https://medium.com/ai-sweden">AI Sweden page</a>. Follow, share and do let us know your thoughts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a0409877f46" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-sweden/why-do-we-need-a-large-gpt-for-swedish-9a0409877f46">Why do we need a large GPT for Swedish?</a> was originally published in <a href="https://medium.com/ai-sweden">AI Sweden</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is GPT-SW3?]]></title>
            <link>https://medium.com/ai-sweden/what-is-gpt-sw3-5ca45e65c10?source=rss-55147f0c79d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ca45e65c10</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gpt-3]]></category>
            <category><![CDATA[nlu]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Magnus Sahlgren]]></dc:creator>
            <pubDate>Wed, 08 Jun 2022 12:40:10 GMT</pubDate>
            <atom:updated>2022-06-08T16:01:53.581Z</atom:updated>
            <content:encoded><![CDATA[<h4>A quick introduction to GPT, the NLU team at <a href="https://medium.com/u/6ceb8749abf1">AI Sweden</a>, and GPT-SW3.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eTudCR7cpFjknUkmat1UEQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@juvnsky?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Anton Maksimov 5642.su</a> on <a href="https://unsplash.com/collections/72364627/textures-%26-pattern-%2F-%D1%82%D0%B5%D0%BA%D1%81%D1%82%D1%83%D1%80%D1%8B-%D0%B8-%D0%BF%D0%B0%D1%82%D1%82%D0%B5%D1%80%D0%BD%D1%8B?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h4>Background</h4><p>The field of natural language processing (NLP) enjoyed its hitherto most significant breakthrough in 2017 with the introduction of the <a href="https://arxiv.org/abs/1706.03762"><strong>Transformer</strong></a><strong> </strong>neural network architecture for language modeling. This led to what can only be described as a Cambrian explosion of language models, in which we can discern three main species of models:</p><ul><li><strong>Decoders</strong> such as <a href="https://openai.com/blog/language-unsupervised/">GPT</a> from <a href="https://openai.com/">Open AI</a>, which are <em>generative</em> language models built specifically to be able to generate language (GPT stands for Generative Pretrained Transformer).</li><li><strong>Encoders</strong> such as <a href="https://arxiv.org/abs/1810.04805">BERT</a> from <a href="https://medium.com/u/be36e94a7e47">Google</a>, which are <em>representation learning</em> mechanisms that learn useful representations of language, which are often <em>finetuned</em> to solve specific tasks.</li><li><strong>Encoder-decoders</strong> such as <a href="https://arxiv.org/abs/1910.10683">T5</a>, also from Google, which can be thought of as <em>translation</em> models that can transform an input sequence into an output sequence.</li></ul><p>Language models quickly set new state-of-the-art results across the board of NLP tasks, and became the standard tool for building NLP applications. The development of new models continued with unwavering pace, and the next significant breakthrough came in the early summer of 2020 when Open AI published a paper describing the incomparably largest language model to that date: <a href="https://arxiv.org/abs/2005.14165"><strong>GPT-3</strong></a>.</p><p>GPT-3 is, as the name suggests, the third generation of GPT, with the main difference from its predecessor (GPT-2) being the sheer size of the model; 96 layers, 175 billion parameters, a batch size of 3.2 million, and trained on 570 GB of data. Apart from the impressive engineering achievement in simply scaling a model to this size, the model also displayed some interesting properties and abilities. In particular, its capacity for <strong>zero-shot</strong> and <strong>few-shot</strong> learning was reported to be unlike any other model in the field of NLP up to that point.</p><p>It soon became apparent that this was not only empty words; Open AI took the somewhat controversial decision to <em>not</em> release the model openly (as had been common practice in the field until then), and instead opted to commercialize access to the model through an API, which relied wholly on the model’s ability to do zero-shot and few-shot learning. The new guild of “prompt hacking” was born, and the commercial success of GPT-3 was a fact.</p><h4>GPT-SW3</h4><p>As the name suggest, <strong>GPT-SW3</strong> is the Swedish counterpart to GPT-3: a large-scale generative pretrained Transformer built on massive amounts of Swedish data with the explicit purpose of being able to generate Swedish text, and to thereby function in zero-shot and few-shot scenarios. The initiative to develop such a model for Swedish is led by the <a href="https://www.ai.se/en/projects-9/natural-language-understanding"><strong>NLU research group</strong></a> at <a href="https://medium.com/u/6ceb8749abf1"><strong>AI Sweden</strong></a>, and includes a number of collaborators, including <a href="https://www.ri.se/"><strong>RISE</strong></a>, the <a href="https://wasp-sweden.org/research/research-arenas/wara-media/"><strong>WASP WARA on media and language</strong></a>, and <a href="https://www.nvidia.com/"><strong>Nvidia</strong></a>. GPT-SW3 already exists in a preliminary version with 3.5 billion parameters, which was trained on a moderately sized corpus of approximately 100 GB of mostly web data. Our current goal is to train a 175 billion parameter model with close to 1 TB of data.</p><p>There is only one publicly available computer in Sweden that is sufficiently large to train a model of this size: <a href="https://www.nsc.liu.se/systems/berzelius/"><strong>Berzelius</strong></a><strong>,</strong> located at Linköping university. Berzelius is an Nvidia DGX SuperPOD equipped with 60 DGX-A100 machines connected with Nvidia Mellanox InfiniBand HDR network. The system is a donation from <a href="https://kaw.wallenberg.org/"><strong>Knut and Alice Wallenberg foundation</strong></a>, and can reach a theoretical performance of 300 petaflops. For training our preliminary 3.5B version of GPT-SW3, we used 16 nodes for 2.5 days. For training the 175B version, we estimate that we will need 30 nodes for approximately 90 days.</p><h4>New to this <a href="https://medium.com/u/504c7870fdb6">Medium</a></h4><p>My, and the AI Sweden NLU team’s, aim with being on Medium is to be as open and transparent as possible with the development process, and to invite several different perspectives on this type of model — both our own as well as those who may be more critical of this development. We will cover topics ranging from details of the development process (e.g. data collection and filtering), over reasons and considerations for leading this initiative, to more philosophical questions about the limits of large language models and the potential (or not) to reach AGI.</p><p>We’ll gather everything under the <a href="https://medium.com/ai-sweden/nlu/home">NLU / GPT-SW3 tab</a> on the <a href="https://medium.com/u/6ceb8749abf1">AI Sweden</a> page. Follow, share and do let us know your thoughts.</p><p>Best,<br><strong>Magnus</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ca45e65c10" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-sweden/what-is-gpt-sw3-5ca45e65c10">What is GPT-SW3?</a> was originally published in <a href="https://medium.com/ai-sweden">AI Sweden</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>