What are the advantages of building a large language model for Swedish, and what should we look out...
The Swedish AI ecosystem is developing rapidly, partly beyond expectation. We see a much more mature AI landscape compared to just a couple of years ago, and we believe there is potential for Sweden to gain huge momentum with investments that align with the current global AI landscape.
Given the latest developments in AI, despite challenges, there is a very strong potential in the Swedish AI ecosystem that can be realized. Here, we will describe some of the local and global trends that we think we need to adapt to, some of them on a very short timescale.
AI awareness and AI maturity are increasing rapidly
In the wake of the release of AI models that generate results and communicate in ways that speak directly to humans, such as images and natural language, the awareness of AI as a radically changing force in society has never been larger. While the discussion in the media is perhaps not always grounded in technical realities, and a full understanding of what AI might actually mean for one’s life or business might be elusive, fewer now doubt its impact. At the same time, the AI efforts initialized some years ago are starting to bear fruit in solutions that need to be deployed in products and services, developed by organizations that are truly geared towards real AI products and not proof of concepts. Efforts for accelerating AI in Sweden must embrace these changes and focus even stronger on responsible operationalization rather than awareness.
Large foundation models and language models have changed the game
We are now at a stage where generative models can truly assist, and in some parts replace, human work and creativity. Image models can generate visual representations that can easily be taken for natural or human creation, and large language models can generate text on relatively complex subjects in several different styles. As these models grow more capable and cover more modalities such as video, music, programming, and engineering, we will have to account for new ways of working with creative disciplines and a new reality where text and representations that earlier could only be attributed to human effort could have been generated by a machine with its specific limitations and representation of the world. There is clearly a place for more actors in this space and it is, for representation and inclusion purposes, important that Sweden and Europe an active part of these developments. Driven by private actors, the development of these types of models is currently happening largely in the US. Given the importance of this type of technology, Sweden needs to engage, but without similar private actors, the model likely has to be different. Collaborative efforts to develop e.g. open large language models such as GPT-SW3 have proven that there are alternative paths to development, but there is a need for concerted efforts.
A new AI ecosystem and a new digital infrastructure is forming
The capacity and performance of AI and Machine Learning (ML) models are rapidly increasing, in large part due to the availability of data sets, benchmarks, and standard models to ML researchers. AI is to a large degree developed based on shared resources and AI Artifacts, and easy access to such artifacts can speed up research and development significantly. While relevant and high-quality data sets remain key for AI research and development, access to artifacts such as meta-data, knowledge, models, applications, algorithm implementations, and benchmarks are similarly critical components. As new applications are increasingly built on already existing models and components, including very large foundational models requiring large computational resources not only for training but also for inference, we are rapidly moving towards a new type of AI infrastructure and ecosystem with an increased focus on artifact reuse and inference services. There are very strong dependencies between all types of AI artifacts such as computational resources, data access, inference services, and meta-data such as data and model descriptions, and next-gen AI infrastructures will need to integrate these resources well. A coordinated, easily accessible next-gen AI infrastructure resource for Swedish researchers, industry, and developers would have the potential to accelerate research and development significantly.
AI is taking a clear place in science, engineering and innovation
For many research-dependent businesses, e.g. pharmaceuticals, and engineering-driven enterprises, e.g. within automotive, AI is becoming an integrated part of not only services and products but strategy, structure, and organization for research and development. The development of new materials, substances, and software will fully depend on AI methods and an organization built around these methods. While companies have huge knowledge bases around and models for the physical and economical aspects of their business, we expect these to be continually more integrated with data-driven and machine-learning solutions, opening up completely new avenues not only for product development but also spurring on fundamental AI research. This could open up new possibilities for collaboration, as the efficient search for the right solution out of an extremely large set of possibilities might not at a fundamental level differ too much between discovering new materials, proteins, or code, as well as the development of new areas of strengths for Sweden through combining world-leading research in other disciplines with AI.
High-value AI applications are gradually being realized
While self-driving is, as always, some time away, AI is gradually starting to realize its potential in e.g. telecom systems, supply chain management, drug discovery, and medical applications, often without great fanfare. Still, expectations are high, and new approaches and efforts to incorporate both state-of-the-art and tried and tested AI solutions at scale are needed. Not only a matter of developing proof of concepts and narrow applications, the top-down strategic view of businesses and public services may have to be developed — beyond predictions and diagnostics, how do we want our healthcare, energy or supply systems to work, and how do we build and structure AI solutions to achieve this? These large-scale engineering efforts offer both challenges and clear opportunities for world leadership for Sweden.
The gap between academia and industry must be made smaller
While the number of papers published in and around AI has never been higher, there is still a disconnect between research and realized value in industry and society. Huge research efforts around the world will eventually need to show societal value, and new forms of collaboration between academia and industry truly sharing people and knowledge may be necessary. At the same time, the participation of academia in the development of large-scale machine learning models is limited, mostly being the domain of well-funded industry players. Where efforts have been made outside of these private entities, developing open source alternatives at sometimes incredible speed, they are often driven by decentralized research collectives. There are huge opportunities for Sweden to embrace new collaborative research and development efforts with non-traditional goals and metrics to take a world-leading position within AI.
Curious about the impact of the work that AI Sweden and its partners collaborate on? Make sure to check out our Impact Report for 2022, and join the conversation and community on My AI.
