Rethinking open source generative AI: open-washing and the EU AI Act

The EU AI Act [44] will impose on general purpose AI providers a number of potentially burdensome requirements, including going through a conformity assessment, providing human oversight, and providing technical documentation that includes detailed information on system architecture, training datasets, provenance and curation (Annex IV in [44]). This is a major improvement over the current regulatory landscape, where models have been allowed to proliferate under the murkiest of legal conditions and with little to no regulatory oversight [7, 49].

A special feature of the EU AI Act is the importance it accords to open source. In view of the notion that open models can contribute to research and innovation, the EU AI Act provides a number of exemptions for such models, meaning that at least some of the burdensome requirements mentioned above can be escaped by attaining open source status. Key passages in this regard appear in section §60 of the latest iteration:¹

• “Software and data, including models, released under a free and open-source licence that allows them to be openly shared and where users can freely access, use, modify and redistribute them or modified versions thereof, can contribute to research and innovation in the market and can provide significant growth opportunities for the Union economy.” (§60i)
  
• [models released under a free and open-source licence] “should be subject to exceptions as regards the transparency-related requirements imposed on general purpose AI models.” (§60f)
  

What exactly are these exceptions? Under the latest version of the Act, providers of AI models “under a free and open licence” are exempted from the requirement to “draw up and keep up-to-date the technical documentation of the model, including its training and testing process and the results of its evaluation, which shall contain, at a minimum, the elements set out in Annex IXa” (Article 52c:1a). Instead, they would face a much vaguer requirement to “draw up and make publicly available a sufficiently detailed summary about the content used for training of the general-purpose AI model according to a template provided by the AI Office” (Article 52c:1d).

If this exemption or one like it stays in place, it will have two important effects: (i) attaining open source status becomes highly attractive to any generative AI provider, as it provides a way to escape some of the most onerous requirements of technical documentation and the attendant scientific and legal scrutiny; (ii) an as-yet unspecified template (and the AI Office managing it) will become the focus of intense lobbying efforts from multiple stakeholders (e.g., [12]). Figuring out what constitutes a “sufficiently detailed summary” will literally become a million dollar question.

While minor adjustments may still be made, it is clear that any decisions to be made here, both in the EU AI Act and in the requirements and templates for technical documentations surrounding it, will be hugely consequential for the open technology landscape. We are not the first to note this. Legal scholar Kate Downing has described the current version of the EU AI Act as “choose your own adventure” [15] when it comes to openness. The most important gap appears to be that there is very little reference to training datasets. Indeed, sociologist and open policy advocate Alek Tarkowski has warned that the current EU AI Act “fails to set meaningful dataset transparency standards” [56].

In sum, with the enshrinement of open source in the AI Act, the term “open source” comes to carry unprecedented legal weight. Can it bear this weight? In what follows, we briefly review how the open source community has been responding to these developments.

Until recently, classifications of software as open source could simply rely on the availability of source code under appropriate licensing: if some software is released under a licence approved by the Open Source Initiative (OSI), it means that it is fully open and minimally restrictive [45]. For software that is relatively portable and user-deployable, this was long sufficient, and it afforded users the rights to make copies, to tinker and, and to make improvements. However, the rise of large language models and text-to-image generators means that a different approach is needed [47].

There are various efforts underway to update and tailor the definition of open source to current generative AI systems. One is a public consultation dubbed the “Open Source AI Deep Dive” that the OSI board may draw on in their efforts to update their definition of open source in the age of generative AI.² Another is the “Joint Statement on AI Safety and Openness” by parties including Creative Commons, Mozilla, LAION and Open Future.³ The challenge that these efforts face is to adopt the notion of open source, which used to be fairly unambiguous, to the increasingly complex world of generative AI systems [34].

The sheer amount of training data, the architectural complexity, and the many moving parts make full openness a tall order for generative AI [53]. The compute needed for training comes with costs that are within reach of none but a handful of larger corporations or government entities [1]. Human labour can be involved at multiple points, from reinforcement learning datasets to crowd-sourced ratings. Comprehensive documentation of such complex systems requires serious effort [20]. And opening up data and training pipelines might pose not only a commercial risk but also lead to legal exposure. Naturally, many actors in the field opt to err on the side of caution and only disclose elements of their system as required [4].

Against this backdrop of a rapidly-evolving technological and regulatory landscape, the very notion of open source AI is a moving target. While current efforts to arrive at a new open source definition for AI are useful, we observe they are strongly focused on the question of licensing. This makes sense: the OSI has been very successful in shepherding the notion of open source licences. However, if everything hinges on licensing, what is to stop model providers from releasing the most inscrutable portion of their system (say, model weights) under an OSI-approved licence and collect open source benefits? This stands to be a major avenue for open-washing.

There is nothing imaginary about this. For one, we are already seeing a strong trend towards selective and self-serving forms of openness, both in release strategies and in empirical surveys of model openness ([32, 53, 60], and see below). Also, other sectors offer well-known precedents of such dynamics. Take fair trade coffee. The international fair trade movement started as a grassroots effort that directly empowered local farmers and waged labourers by giving them fairer terms in global commodity markets. Within a decade, it was adopted by multinationals like Starbucks and Nestlé, who turned it into an efficient marketing tool: green-washing at work. Sociological research has established that this resulted in these multinationals effectively co-opting the notion of fair trade. This work provides an important lesson on the playbook of corporate lobbies: “co-optation... occurs primarily on the terrain of standards, in the form of weakening or dilution” [26].

We conclude that if open licensing becomes the sole deciding factor for model openness, the open source community faces the risk that community standards will be co-opted and diluted just as Starbucks and Nestlé have co-opted and diluted the notion of fair trade in coffee. Open licensing could become an empty gesture.

Companies operating in the generative AI space currently appear to be converging on a strategy known as open-washing [60]: collect brownie points for openness without disclosing critical information of training and tuning procedures, thereby largely escaping the scientific scrutiny and legal exposure that would come with full openness.

A key sign of open-washing is the growth of what we call a ‘release by blogpost’-strategy. The bulk of open generative AI models released in the past year were first made public in a blogpost or press release touting their openness. For instance, TII's Falcon 70B was introduced as the “top-ranked open-source AI model”⁴, StabilityAI's Stable Beluga as “open access”⁵, Mistral proclaims “we have the best open source models”⁶, 01.ai's Yi 34B Chat claims to be the “next generation of open-source and bilingual LLMs”⁷, and Alibaba claimed “we opensource our Qwen series”⁸. Probably the strongest claims to the label open source (if not its content) come from Meta and its Llama 2 and Llama 3 models. The corporate blogposts that introduced Llama made the following claims: ⁹

• “Today, we're introducing the availability of Llama 2, the next generation of our open source large language model.” (Llama 2)
  
• “Meta has put exploratory research, open source, and collaboration with academic and industry partners at the heart of our AI efforts for over a decade.” (Llama 2)
  
• “Today, we're introducing Meta Llama 3, the next generation of our state-of-the-art open source large language model.” (Llama 3)
  

Arguably, companies can announce and market their products however they want. And when they do, they show their hand. In this case, we can conclude that Meta and other companies in this space see a particular benefit in putting a claim on the term open source. We will see below why this may be.

Alongside claims of openness, the release-by-blogpost model typically features some nicely laid out tables that compare the model to a selection of its competitors on a selection of scientific benchmarks like MMLU, HUmanEval, TruthQA and the like. These evaluation tables, clearly modelled after NLP's coveted SOTA tables [10], allow releases to retain the veneer of scientific work while at the same time avoiding the fine-grained accounting and the scrutiny of peer review that comes with actual scientific publication. These tables also offer ample degrees of freedom for cherry-picking, enabling model providers to present their products in the best light possible. Without technical documentation and peer review, the release-by-blogpost model is little more than pseudoscience.

When generative AI follows the release-by-blogpost model, it is reaping the benefits of mimicking scientific communication —including associations of reproducibility and rigour [21, 23]— without actually doing the work. And when generative AI co-opts the term open source, it is reaping the benefits of libre culture —including associations of transparency and associated freedoms [45, 60]— without actually contributing to the commons. This is how open-washing works. There is ample evidence that as a communication strategy, open-washing is highly effective. The launches of Llama 2 and Llama 3 were greeted with considerable excitement in mainstream media outlets, almost without exception uncritically echoing the open source claim as a major selling point. For instance, a Wired headline of April 2024 claims, “Meta's Open Source Llama 3 Is Already Nipping at OpenAI's Heels”¹⁰, and Fortune wrote, “Meta releases its new Llama 3 open-source AI model. Is it enough to keep Meta at the front of the pack?”¹¹.

The consequences of open-washing are considerable and affect multiple stakeholders. Open-washing stands in the way of innovation, because if large corporations can derive benefits from the trappings of open source without doing the requisite work, this sucks up the oxygen in the open source ecosystem, making it less attractive for smaller entities to find funding for truly open projects [1]. Open-washing is also bad for research, because it means that researchers can no longer count on being able to tinker with models and architectures even if they are advertised as open source [54]. And open-washing is bad for the public understanding of AI, because it creates artefacts designed to impress without providing people with the resources to reach a deeper understanding of the technology [39].

Open-washing is related to the notion of audit-washing [19]: the risks posed by poorly designed auditing procedures. As Goodman and Tréhu note, “Audits without clear standards provide false assurance of compliance” (p. 3). As current and impending regulation increasingly puts legal weight on the notion of open source generative AI, the spectre of false assurance looms large.

Our main goal in what follows is to propose a conception of openness that can serve the EU AI Act's goal to foster research and innovation, that can help to specify what makes a sufficiently detailed summary, and that can provide key building blocks for a model openness template.

Ultimately assessment must be evidence-based, and a key question therefore is what the building blocks of openness assessment should be.

We provide a matrix of relevant dimensions of openness in generative AI, each grounded in evidence-based judgements of degrees of openness. The dozen or so dimensions we have identified here along with three levels of openness ( open, partial, closed) provide a sufficient level of detail to provide well-informed, high quality, systematic judgements of openness in generative AI.

Here we provide a quick walkthrough of all features by means of a comparison of two systems that are both billed as open source: BloomZ (Bloom for short), introduced by the BigScience Workshop team in May 2023 as an open source chat LLM [61], and Llama 2 (Llama), introduced by Meta as “the next generation of our open source large language model” as we saw above.¹³ As we will show, our method provides a way to form a nuanced and evidence-based judgement of the truth value of this claim (Figure 1).

Availability. When it comes to open code, we find that BloomZ makes available source code for training, fine-tuning and running the model, while for Llama none of the model's source code is made available, only scripts for running the model are shared. The LLM data that was used to train the base model is documented in great detail by Bloom [61], while for Llama only the vaguest details are provided in a corporate preprint: “a new mix of data from publicly available sources, which does not include data from Meta's products or services” [57]. The statement is clearly designed to minimise legal exposure. Both systems make the LLM weights of the base model available, though for Llama access is restricted through a consent form. The training data for instruction tuning (RL data) is described and documented by Bloom as consisting of xP3 (Crosslingual Public Pool of Prompts); for Llama, the corporate preprint notes that fine-tuning was done based on “a large dataset of over 1 million binary comparisons based on humans applying our specified guidelines, which we refer to as Meta reward modeling data”, and which remains undisclosed. (The same preprint mentions that for evaluation, Meta did build on several RLHF datasets openly shared by others.) Model weights for the instruction-tuned version (RL weights) are made openly available by BloomZ, while for Llama they require an access request.

Documentation. The BloomZ project code is well-documented and actively maintained, while for Llama 2 no documentation of source code is available as the source code itself is not open. The architecture is described for BloomZ in multiple scientific papers and supported by a github repository of code and recipes on HuggingFace; for Llama, the architecture is described in less detail and scattered across corporate websites and a preprint.

BloomZ's multiple preprints document data curation and fine-tuning [50, 61] in great detail; in contrast, Llama's single preprint offers fewer details and appears strategically vague on crucial details (for instance, training datasets and instruction tuning). The scientific documentation of BloomZ also includes multiple peer-reviewed papers, from a scientific description of the multitask fine-tuning procedures [41] to an estimation of the carbon footprint [36] — currently one of the very few scientifically vetted sources of data on the energy footprint of training large language models. No peer-reviewed papers providing scientific documentation or evaluation of Llama are known currently.

BloomZ also released model cards that describe the architecture and evaluation results with extensive cross-references to other documentation on training data, training approach, model architecture, fine-tuning and responsible use. In contrast, the Llama model card only provides minimum detail and none whatsoever on training data. A data sheet is only available for BloomZ. This means that for Llama, there is no documentation of training datasets whatsoever — a prime example of a strategy described by Birhane et al. as a tactical template of “(non)declaring the training dataset information” [4].

Access and licensing. Neither BloomZ nor Llama distribute models as software packages via indexed and version controlled public code repositories such as pypi. Instead, both are primarily intended for local deployment. BloomZ is available through the Petals API, while for Llama an API is only available behind a privacy-defying signup form. Finally, the models also differ in terms of licensing. BloomZ has two relevant licences. Its source code is Apache 2.0, an OSI-approved open source licence, while the model weights are released under the Responsible AI Licence (RAIL) [13]. Llama 2 is released under Meta's own Community Licence. Both licences aim to restrict harmful use cases, but there is a key difference in how they implement how model outputs are to be represented. RAIL stipulates that a user may not “generate content without expressly and intelligibly disclaiming that the text is machine-generated”, while Llama stipulates that a user may not “represent that Llama 2 outputs are human-generated” — a much lower bar, because it leaves open a wide swathe of use cases where there may not be the explicit claim of human-generated output, but merely a strong implication.

This walkthrough shows that drilling into the details of generative AI systems using the dimensions of openness of our framework makes critical differences visible. Only BloomZ can substantially claim open source status, while Meta's Llama is at best open weights, and is closed in almost all other aspects. Llama, in all currently available versions, is a prime example of a model that claims openness benefits by merely providing access to its most inscrutable element: model weights.

With a first view of the framework in hand we can extend our survey to a larger sample of generative AI systems. We focus on models that bill themselves as open, aiming to include well-known players but also small models and work by smaller teams or organisations, some of which make up for their lack in size and model performance with high standards of openness and transparency. Every single openness judgement is directly linked to publicly available evidence, and all data points are available in a versioned data repository.¹⁴ Therefore we only focus on describing the most important findings and trends.

2.2.2 Text-to-image generators: mostly closed. In the same fashion, we assess 6 text-to-image generators (Figure 3). Again, we add Open AI's DALL-E models as a closed reference. Overall, the survey yields far fewer systems in comparison to text generators. A possible reason for this might be that relatively few image datasets are available. Text-to-image generators also differ in terms of their machine learning architecture. For instance, image generators do not generally implement instruction-tuning, a key component of text generators.

Most relevant for evidence-based openness assessment are the ways in which text-to-image generators implement ways of tracking provenance of synthetic imagery and set up guardrails against creating undesired content. Some systems use watermarking to enable a form of provenance tracking. For moderation, text-to-image systems commonly rely on forms of prompt moderation, often text filtering or classification. The status of such provenance and safety measures is not always documented, and this is what the openness judgements seek to capture. (Thus, a model may implement prompt moderation but also document it; in that case, it counts as open for that dimension. In contrast, when a model may or may not watermark its output, and does not disclose this either way, it counts against openness.) The respective dimensions of the assessment framework have been adjusted accordingly.

One system stands out when it comes to openness, transparency and documentation: Stable Diffusion by Stability AI, Runway and the Computer Vision and Learning Group at Ludwig Maximilians Universitat Munich, Germany. Some of the other assessed systems build on or fine-tune the various models of Stable Diffusion. Some other systems are open-weight only. Open AI's DALL-E is completely closed.

This means that those interested in text-to-image generators face a relatively clear choice where to look for a very open alternative to proprietary and closed products. It also means that only Stable Diffusion has been open to scrutiny from scientists, regulators and the general public. This has enabled auditing of the underlying datasets, which has revealed legal challenges [49] and deeply problematic content [4, 5].

A key goal of our work is to provide flexible ways for the evidence-based assessment of openness. While the focus is on evidence-based expert judgements of degrees of openness supported by public documentation, the resulting fine-grained data must sometimes be translated into more reductive scores, labels, or classifications.

There are multiple ways to turn fine-grained openness assessments into metrics of openness that support classification or comparison (Figure 4). One is to assign weights to openness classes and to derive, based on this, a cumulative openness score (Figure 4, panel 2). This would be a gradient measure of openness. For simplicity and transparency, here we have picked weights of 1 (open), 0.5 (partial) and 0 (closed) and we have weighted all dimensions equally. Different choices are possible. For instance, in situations where it is important to know exactly what is in the training data or how the instruction-tuning was carried out, these dimensions might be weighted more heavily, penalising models that are less open.

From such a gradient measure, further classifications can be derived. One would be to divide the continuum into separate categories, comparable to the EU energy label system (Figure 4, panel 3). Here too, the simplest approach would be to divide up the space evenly, but different weightings could be used to discretise the space in ways that are more fitting to particular purposes. A third, closely related approach would be to reduce the continuum to a simple dichotomous classification of models into open versus closed (Figure 4, panel 4). The figure makes visible how this cannot be anything else than a gross oversimplification: a dense multidimensional field of openness measures is reduced to a simple binary classification.

Separate from these approaches is a fourth method, increasingly popular but even more reductive (Figure 4, panel 5). This is to single out only a single measure and base an openness classification on that. This is in effect what a focus on open weight models or on open licences accomplishes. If we are satisfied with calling such systems open, we are discarding many relevant dimensions and degrees of openness just to arrive at a single binary classification. What can also be seen is how privileging one such measure may distort the overall picture: there are many open weight models currently, and so focusing on this one dimension makes it seem as if almost half of all text generators are open. By contrast, if we were to focus on the all-important instruction-tuning, by many accounts the secret sauce of chat LLMs, there would be only a handful of models that offer the requisite openness.

We take time to discuss these ways of turning rich openness assessments into reductive metrics because it is important to be aware of the distorting effects of metrics [55]. Metrics can be gamed; indeed important parts of present-day NLP and machine learning could be described as finding clever ways of gaming metrics, whether automated or human [10]. This figure therefore offers both a manual for lobbyists and the means to counter them. It can be predicted that corporate interests will argue for the easiest-to-attain openness dimensions to weigh most heavily — indeed this is likely one of the reasons behind the ‘open weight branded as-open source’ approach of companies like Meta and Mistral, and behind lobbying attempts to favour simple licence-based decisions. At the same time, knowing this, regulatory entities can make informed decisions on which forms of openness should count.

Open source AI risks and opportunities are increasingly being studied in the generative AI landscape [16]. Corporate entities in this space have hand-waved at “AI Safety” as a reason to keep system specifications under wraps [43], but this appears mostly a thinly disguised attempt to obscure the clear and present harms such models already pose [17] while minimising the considerable legal and regulatory exposure that would come with disclosing details about training data [35, 49].

Discourse about appropriate levels of openness has been prone to equate ‘open source’ with two rhetorical extremes: (i) radical openness, which would mean literally sharing every single model component and training dataset, and (ii) homeopathic openness, which is openness diluted beyond recognition, for instance by sharing only model weights. Many corporate players, moving aside the first of these as unrealistic, propose that therefore the second, diluted sense is the only attainable. But this is not the case.

Between radical openness and homeopathic openness lies meaningful openness. Openness comes in degrees, and regulation should be designed to foster meaningful forms of openness. A composite notion of openness can also provide the building blocks that can cater to specific use cases — be it a radically open system for use in research and education, or more privacy-oriented ones with privacy-enhancing techniques in place.

There is a case to be made that open systems that disclose training and fine-tuning data are safer because of the possibility of public scrutiny and professional auditing. It is true openness of this kind, and not just open weight models, that can speed up innovation and afford inclusion and diversity [20, 61]. The EU AI Act and other future regulation should incentivise and regulate data disclosure for generative AI so that it is safer and more auditable, and so that everyone can benefit from a better understanding of how to build and how to do research on this technology.

Generalising the framework. While the framework is in principle applicable to the full range of generative AI, we did not identify systems other than text and text-to-image generators that bill themselves as open source. However, as the field of generative AI is fast growing, this is bound to change in the coming years. We did identify a range of recent hybrid systems that combine text and text-to-image generation in one system, such as DeepFloyd by Stability AI, Open AI's GPT-4 or Google's Gemini model family. As these multimodal models rise in numbers and popularity, their assessment may require additional adjustments to the dimensions of openness laid out so far. Assessing other media types using the framework may require domain-specific dimensions and decisions, just as we found for text versus text-to-image.

Importantly, the overall framework is designed so that the bulk of the features are applicable to any generative AI system. Most generally, the broad areas of availability, documentation, and access and licensing should probably feature in any well-informed assessment of openness and accountability [11]. More specifically, many of the constituent features, from datasets to scientific documentation, are of general relevance to the question of how to define open in the context of AI and machine learning. This means that the framework is flexible enough to serve as a blueprint for the implementation of living guidelines [7] or for the formulation of templates such as those to be developed by the EU AI Office.

Training dataset assessment is complex. Our survey stays relatively superficial when it comes to assessing exactly how open the training data of a system is. This is due to three factors. Even the most open models we surveyed only describe what data was used instead of directly sharing it (sometimes due to licensing restrictions). But such descriptions of training data often only provide superficial detail of preprocessing steps and how exactly the data was fed into model training pipelines. This lack of documentation detail is further complicated by the sheer size of the training data that some systems are trained on, which are often combined and edited in ways that make it hard to retrace how data was used (sometimes in the service of privacy enhancing techniques). But probably the most serious complicating factor arises from the complexities of how some larger models are trained. Models can feature a simple training pipeline, but can also employ complex training procedures such as distributed or incremental training techniques (e.g. federated learning, batch learning or online learning), sometimes even using data acquired from user interactions. In sum, such challenges make it hard to assess training data openness well: we are only scraping the surface here.

Some data requires closed-door assessment. Full openness can be harmful, and in some instances assessment should take place behind closed doors. For instance, in the case of dealing with datasets that contain CSAM material, the release of data prior to assessment in the name of openness would pose a clear risk [4, 31]. But this should not mean that organisations that release production-ready systems are relieved from the requirements of full data disclosure by hand-waving to safety concerns. Especially when the systems are advertised as open source.