AI Sweden Insights

Open release of GPT-SW3

Written by Magnus Sahlgren | Nov 16, 2023 5:36:12 AM

We are now taking the next steps in the democratization and value creation of Large Language Models (LLMs) in Sweden and the Nordics, with the open release of the GPT-SW3 family of LLMs.

 

Image produced by DALL·E

As previously described, GPT-SW3 is the first LLM for the Nordic languages. The models have been developed by the NLU group at AI Sweden with support from RISE, the WASP WARA for media and language, the National Supercomputer Center at Linköping University, and Nvidia. GPT-SW3 was developed as a research project, with the primary goal to build and provide the first Nordic LLM as a foundational resource for Nordic NLP, but also to build competence and awareness around LLMs in Sweden, and to explore and validate potential applications of a nationally developed LLM. The open release of the GPT-SW3 models marks the end of the development phase of GPT-SW3. Our initial goal was to build and provide the first LLM for the Nordic languages and with the open release of the models we have now fulfilled this goal. 

From restricted to open access

In a previous post, we described our reasons for opting for an initial restricted pre-release of the GPT-SW3 models. During the pre-release phase, we have manually granted access to the models to researchers in the Nordic region, for the primary purpose of research and validation. To recapitulate, the main reason for this cautious approach was to be able to identify any potential issues with the models’ behaviors and to minimize the risk of adversarial usage of the models. We have collected feedback from users during the pre-release phase, and based on this feedback we now consider an open release to be the natural next step. 

The open release will now allow users complete access and free usage of the GPT-SW3 models. This means that it will now be possible to download, modify, finetune, integrate, and to build commercial and non-commercial products based on the models. This is an important step in the democratization of access to powerful AI models, and to accelerate the use of AI in Sweden and in the Nordic region.

Instruction tuning

We are releasing our base GPT-SW3 models as well as a set of instruction-finetuned versions of the models. The -instruct versions of our models have been finetuned on a set of instruction datasets, including Dolly, OpenAssistant, a selection of data from Open Instruction Generalist (OIG), and a dataset compiled specifically for our models based on FASS (The Swedish pharmaceutical formulary). The instruction data was formatted into a unified turn-based format, where an initial user query is followed by an assistant response, which in turn is (potentially) followed by a follow-up user query, and so on. To accommodate the multilingual capacities of our models, we translated the OpenAssistant and Dolly datasets. The OpenAssistant data was translated from English to Swedish, Danish, Norwegian, and Icelandic, while the Dolly data set was translated from English to Swedish and Danish. The translations were done with our own GPT-SW3 base models.

The next steps

GPT-SW3 is now available as an open model for businesses and organizations to use in their products and services. The model provides Sweden with a crucial resource that strengthens our ability to use AI for increased competitiveness and a stronger welfare system. It can be used for a wide variety of tasks and addresses some of the challenges faced by both the private sector and the public sector concerning other available large-scale language models. AI Sweden will continue to accelerate the use of AI and large language models and we are for example already exploring how to use GPT-SW3 to build a digital assistant for the public sector together with partners. 

Going forward, the NLU team’s aim is to take the next steps in the development of foundation models in and for Sweden, the Nordics, and Europe. What this means in more detail is something we will come back to in a future post. Stay tuned!

 
View full post