The holy or not so holy grail...
... of open source AI
One of my favourite podcasts is 'Leading' by former Downing Street Director of Communications Alistair Campbell and UK Cabinet Minister Rory Stewart. A couple of months ago, Meta's policy lead, Nick Clegg, appeared on the show. Wrapping up the episode, Rory notes how he has just received a "lovely email" from Nick suggesting that they sit down to have lunch and discuss "why open sourcing AI is a good idea". Rory then goes on to say that he "thinks it's pretty terrifying". In this month's newsletter, I dive into the AI open source debate and scrutinise some of the key arguments.
What even is open source AI?
The Open Source Initiative defines open source AI as a system that you can i) use for any purpose and without having to ask for permission, ii) study to understand how it and its components work, iii) modify for any purpose, including to change its output, and iv) share for others to use with or without modifications for any purpose. This definition might seem relatively obvious, and comparable to the definition of open source software. The definition of open source AI, however, is anything but.
A great paper, co-authored by the Signal message app's President Meredith Whittaker, analyses how "the terms open and open source are used in confusing and diverse ways, often constituting more aspiration or marketing than technical descriptor". A more detailed analysis across 14 dimensions of openness shows that "open source AI" is generally more closed than you would expect and that most AI systems available today squarely fail to meet the definition from the Open Source Initiative. Whilst the model's parameters - the weights - are often shared publicly, the data used to train and fine tune a given model is usually kept under wraps.
How great are the benefits?
Proponents of open source AI argue that it enables better scrutiny of how the systems work and democratises access to cutting-edge models to all countries, start-ups and anyone with fewer resources. Whilst these are appealing features, the cost-benefit analysis is more nuanced than you might think.
Closed models, such as OpenAI's o1 and Anthropic's 3.5 Sonnet, definitely face insufficient scrutiny from researchers. A fully open source release of an AI model, however, does not by itself guarantee that more scrutiny will occur and the infamous Heartbleed vulnerability demonstrates why. In this case, a vulnerability was introduced in the software securing over two-thirds of the world's websites. If exploited, Heartbleed allowed anyone to access passwords and other private information transmitted online. The vulnerability remained part of the software for over two years until it was discovered, because the open source code was running on a $2000 budget of donations and no one was paid to keep it secure. What Heartbleed shows is that open source software and AI can suffer from a "tragedy of the commons", where everyone assumes someone else will catch errors or vulnerabilities. Just because a piece of software - or an AI model - can be scrutinised, this does not guarantee that it will be.
The democratisation argument in favour of open source AI is similarly complicated. Even if an AI system can be (re)used freely, is trained using properly documented data, and can be modified, this doesn't mean that anyone can actually build this system themselves. For advanced open source AI, the amount of compute power required to train these systems is so expensive that it takes corporate resources to develop your own system. This makes open source AI a very different beast than open source software, for which anyone can in theory write the code on their laptop.
What are the outright negatives?
Sceptics of open sourcing advanced AI tend to worry about how putting untested capabilities into the open can amplify harms in irreversible ways. On the one hand, someone might (mis)use a system in a way that it wasn't intended to be used for. Earlier this month, for example, Reuters revealed how Chinese military researchers had used Meta's Llama model to build an intelligence processing tool, despite the license explicitly prohibiting military use cases. Given the open source nature of the model, however, there is no way that any license limitations could be enforced. Meta all but acknowledged this reality by quickly choosing to drop the prohibition altogether and granting permission for U.S. military use.
On the other hand, open sourcing advanced AI models could make it even harder to prevent humanity from losing control to AI systems. Just like prohibitions of certain use cases cannot be enforced by the initial provider after the system has been released, it will be impossible for the initial provider to ensure that safety measures (e.g. measures preventing autonomous replication, self-improvement, power seeking or deception) are not modified by a third party. Given that evaluators of OpenAI's latest o1 model found last week that it tried to prevent itself from being shut down in 5% of test cases, this is an increasingly urgent policy concern.
Who drives the policy conversation?
The loudest proponent of open source AI is not some alliance of small start-ups but the Meta corporation. This alone should give us some pause. If open source AI would truly level the playing field, why would Meta want to invest billions in it? In earnings calls, Meta's CEO Mark Zuckerberg has indicated that he wants to ensure that developers globally build on top of Meta's AI models, thereby ensuring that Meta controls the dominant architecture. This is a similar strategy to the one Google pursued in open sourcing its Android operating system, providing incentives to developers who then devoted their time building applications for Android phones. Google thus successfully captured the ecosystem and has effectively prevented anyone from creating a new phone operating system since.
Where do we go from here?
The vast majority of AI systems pose no severe risk of harm and can thus be open sourced without any issue. At the same time, we may want to be more careful about open sourcing the most advanced AI systems before we have some safety guarantees. Last year, the Center for the Governance of AI proposed some pragmatic ways in which many open source benefits can be preserved while still addressing the risks. For example, GovAI encouraged "staged" model release - gradually adding capabilities so that a developer can monitor the impact and cancel a full release if required. They also suggested increased access for model auditors and researchers (rather than anyone and everyone) and more democratic oversight of those who develop the most advanced AI models.
GovAI's proposals can provide some inspiration to policymakers. Long before any tentative government action, however, AI developers themselves can and should think twice about what they release. During the early stages of nuclear technology - a technology with a transformational potential comparable to AI today - scientists voluntarily censored themselves long before this practice became officially sanctioned. Until one can be certain that an AI model is safe for open source release, a similar approach seems wise.
Meme of the month
Suggested reading
If you want to read some more, here are my recommendations for this month:
Vincent Manancourt’s farewell piece as he wraps up at Politico, tracing how the UK AI Safety Institute came to be: “Inside Britain’s plan to save the world from runaway AI”.
The Information on how the recent addition of reasoning capabilities to AI models has allowed researchers to speed up nuclear fusion research.
In some good news, outgoing President Biden and President Xi agreeing that humans, rather than AI, should control nuclear weapons.
Wish you all some great holidays and, as ever, please don’t hesitate to reach out on mark [at] future of life [dot] org if you have any feedback. Also, I am hiring for a new Head of U.S. Policy, more details here.