--- title: What is GPT-SW3? description: A quick introduction to GPT, the NLU team at AI Sweden, and GPT-SW3. image: https://insights.ai.se/hubfs/1_eTudCR7cpFjknUkmat1UEQ.webp --- Jun 8, 2022 12:27:00 PM # What is GPT-SW3? ![Picture of Magnus Sahlgren](https://insights.ai.se/hs-fs/hubfs/Magnus_S.png?width=50&name=Magnus_S.png) [ Magnus Sahlgren ](https://insights.ai.se/author/magnus-sahlgren) Share: [facebook-f icon ](http://www.facebook.com/share.php?u=https://insights.ai.se/what-is-gpt-sw3) [linkedin-in icon ](http://www.linkedin.com/shareArticle?mini=true&url=https://insights.ai.se/what-is-gpt-sw3) [Twitter icon ](https://twitter.com/intent/tweet?url=https://insights.ai.se/what-is-gpt-sw3) [envelope icon ](mailto:?body=https://insights.ai.se/what-is-gpt-sw3) ### A quick introduction to GPT, the NLU team at AI Sweden, and GPT-SW3. ### ![1_eTudCR7cpFjknUkmat1UEQ](https://insights.ai.se/hs-fs/hubfs/1_eTudCR7cpFjknUkmat1UEQ.webp?width=1400&height=778&name=1_eTudCR7cpFjknUkmat1UEQ.webp) Photo by [Anton Maksimov 5642.su](https://unsplash.com/@juvnsky?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/collections/72364627/textures-%26-pattern-%2F-%D1%82%D0%B5%D0%BA%D1%81%D1%82%D1%83%D1%80%D1%8B-%D0%B8-%D0%BF%D0%B0%D1%82%D1%82%D0%B5%D1%80%D0%BD%D1%8B?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) ### Background The field of natural language processing (NLP) enjoyed its hitherto most significant breakthrough in 2017 with the introduction of the [**Transformer**](https://arxiv.org/abs/1706.03762)** **neural network architecture for language modeling. This led to what can only be described as a Cambrian explosion of language models, in which we can discern three main species of models: - **Decoders** such as [GPT](https://openai.com/blog/language-unsupervised/) from [Open AI](https://openai.com/), are *generative* language models built specifically to be able to generate language (GPT stands for Generative Pretrained Transformer). - **Encoders** such as [BERT](https://arxiv.org/abs/1810.04805) from  [Google](https://medium.com/u/be36e94a7e47?source=post_page-----5ca45e65c10--------------------------------) , which are *representation learning* mechanisms that learn useful representations of language, which are often *fine-tuned* to solve specific tasks. - **Encoder-decoders** such as [T5](https://arxiv.org/abs/1910.10683), also from Google, can be thought of as *translation* models that can transform an input sequence into an output sequence. Language models quickly set new state-of-the-art results across the board of NLP tasks and became the standard tool for building NLP applications. The development of new models continued at an unwavering pace, and the next significant breakthrough came in the early summer of 2020 when Open AI published a paper describing the incomparably largest language model to that date: [**GPT-3**](https://arxiv.org/abs/2005.14165). GPT-3 is, as the name suggests, the third generation of GPT, with the main difference from its predecessor (GPT-2) being the sheer size of the model; 96 layers, 175 billion parameters, a batch size of 3.2 million, and trained on 570 GB of data. Apart from the impressive engineering achievement in simply scaling a model to this size, the model also displayed some interesting properties and abilities. In particular, its capacity for **zero-shot** and **few-shot** learning was reported to be unlike any other model in the field of NLP up to that point. It soon became apparent that this was not only empty words; Open AI took the somewhat controversial decision to *not* release the model openly (as had been common practice in the field until then) and instead opted to commercialize access to the model through an API, which relied wholly on the model’s ability to do zero-shot and few-shot learning. The new guild of “prompt hacking” was born, and the commercial success of GPT-3 was a fact. ### GPT-SW3 As the name suggests, **GPT-SW3** is the Swedish counterpart to GPT-3: a large-scale generative pre trained Transformer built on massive amounts of Swedish data with the explicit purpose of being able to generate Swedish text and thereby function in zero-shot and few-shot scenarios. The initiative to develop such a model for Swedish is led by the [**NLU research group**](https://www.ai.se/en/projects-9/natural-language-understanding) at [**AI Sweden, **](https://medium.com/u/6ceb8749abf1?source=post_page-----5ca45e65c10--------------------------------)and includes a number of collaborators, including [**RISE**](https://www.ri.se/), the [**WASP WARA on media and language**](https://wasp-sweden.org/research/research-arenas/wara-media/), and [**Nvidia**](https://www.nvidia.com/). GPT-SW3 already exists in a preliminary version with 3.5 billion parameters, which was trained on a moderately sized corpus of approximately 100 GB of mostly web data. Our current goal is to train a 175 billion parameter model with close to 1 TB of data. There is only one publicly available computer in Sweden that is sufficiently large to train a model of this size: [**Berzelius**](https://www.nsc.liu.se/systems/berzelius/)**,** located at Linköping University. Berzelius is an Nvidia DGX SuperPOD equipped with 60 DGX-A100 machines connected with Nvidia Mellanox InfiniBand HDR network. The system is a donation from [**Knut and Alice Wallenberg foundation**](https://kaw.wallenberg.org/), and can reach a theoretical performance of 300 petaflops. For training our preliminary 3.5B version of GPT-SW3, we used 16 nodes for 2.5 days. For training the 175B version, we estimate that we will need 30 nodes for approximately 90 days. ### New to this  My, and the AI Sweden NLU team’s, aim with this blog is to be as open and transparent as possible with the development process, and to invite several different perspectives on this type of model — both our own as well as those who may be more critical of this development. We will cover topics ranging from details of the development process (e.g. data collection and filtering), over reasons and considerations for leading this initiative, to more philosophical questions about the limits of large language models and the potential (or not) to reach AGI. Follow, share, and do let us know your thoughts. Best, **Magnus**   [NLU](https://insights.ai.se/tag/nlu), [Large Language Models](https://insights.ai.se/tag/large-language-models), [GPT-SW3](https://insights.ai.se/tag/gpt-sw3) ## Related posts [![An image of a screen](https://insights.ai.se/hs-fs/hubfs/0_w-M1ccCWroVFqyie-1.jpg?height=200&name=0_w-M1ccCWroVFqyie-1.jpg) ](https://insights.ai.se/gpt-sw3-as-klara-the-artificial-friend) [NLU](https://insights.ai.se/tag/nlu), [Chatbots](https://insights.ai.se/tag/chatbots), [Large Language Models](https://insights.ai.se/tag/large-language-models) ## [GPT-SW3 as Klara, the artificial friend](https://insights.ai.se/gpt-sw3-as-klara-the-artificial-friend) ![Picture of Magnus Sahlgren](https://insights.ai.se/hs-fs/hubfs/Magnus_S.png?width=50&name=Magnus_S.png) [Magnus Sahlgren](https://insights.ai.se/author/magnus-sahlgren) Mar 1, 2023 11:40:00 AM In this post, we look back at the first public display of GPT-SW3. This occurred during an art... [ Read more ](https://insights.ai.se/gpt-sw3-as-klara-the-artificial-friend) [![Prompt used: A high-resolution image depicting Scrabble tiles tumbling in the air, each spelling out the letters and symbols of _GPT-SW3_. There should be individual](https://insights.ai.se/hs-fs/hubfs/DALL%C2%B7E%202023-11-14%2016.18.28%20-%20A%20high-resolution%20image%20depicting%20Scrabble%20tiles%20tumbling%20in%20the%20air%2c%20each%20spelling%20out%20the%20letters%20and%20symbols%20of%20_GPT-SW3_.%20There%20should%20be%20individu.png?height=200&name=DALL%C2%B7E%202023-11-14%2016.18.28%20-%20A%20high-resolution%20image%20depicting%20Scrabble%20tiles%20tumbling%20in%20the%20air%2c%20each%20spelling%20out%20the%20letters%20and%20symbols%20of%20_GPT-SW3_.%20There%20should%20be%20individu.png) ](https://insights.ai.se/open-release-of-gpt-sw3) [NLU](https://insights.ai.se/tag/nlu), [Large Language Models](https://insights.ai.se/tag/large-language-models), [GPT-SW3](https://insights.ai.se/tag/gpt-sw3) ## [Open release of GPT-SW3](https://insights.ai.se/open-release-of-gpt-sw3) ![Picture of Magnus Sahlgren](https://insights.ai.se/hs-fs/hubfs/Magnus_S.png?width=50&name=Magnus_S.png) [Magnus Sahlgren](https://insights.ai.se/author/magnus-sahlgren) Nov 16, 2023 6:36:12 AM We are now taking the next steps in the democratization and value creation of Large Language Models... [ Read more ](https://insights.ai.se/open-release-of-gpt-sw3) [![](https://insights.ai.se/hs-fs/hubfs/1_COYm2B2zF-2Gggr0yOh2aw.webp?height=200&name=1_COYm2B2zF-2Gggr0yOh2aw.webp) ](https://insights.ai.se/gpt-sw3-pre-release) [NLU](https://insights.ai.se/tag/nlu), [Large Language Models](https://insights.ai.se/tag/large-language-models), [GPT-SW3](https://insights.ai.se/tag/gpt-sw3), [AI](https://insights.ai.se/tag/ai) ## [GPT-SW3 Pre-release](https://insights.ai.se/gpt-sw3-pre-release) ![Picture of Daniel Gillblad](https://insights.ai.se/hs-fs/hubfs/daniel-gillblad-1.webp?width=50&name=daniel-gillblad-1.webp) [Daniel Gillblad](https://insights.ai.se/author/daniel-gillblad) Jan 23, 2023 11:53:00 AM Evaluating large, generative language models for Swedish and next-generation AI applications Photo... [ Read more ](https://insights.ai.se/gpt-sw3-pre-release) ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Magnus Sahlgren", "url" : "https://insights.ai.se/author/magnus-sahlgren" }, "dateModified" : "2023-10-24T10:27:32.733Z", "datePublished" : "2022-06-08T10:27:00.000Z", "headline" : "What is GPT-SW3?", "image" : [ "https://insights.ai.se/hubfs/1_eTudCR7cpFjknUkmat1UEQ.webp" ], "mainEntityOfPage" : { "@id" : "https://insights.ai.se/what-is-gpt-sw3", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://insights.ai.se/hubfs/AI%20Sweden%20logo%20-%20yellow.svg" }, "name" : "AI Sweden" } } ```